Nov 03 01:48:17.214697 ip-10-0-140-140 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 03 01:48:17.214710 ip-10-0-140-140 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 03 01:48:17.214717 ip-10-0-140-140 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 03 01:48:17.214935 ip-10-0-140-140 systemd[1]: Failed to start Kubernetes Kubelet. Nov 03 01:48:27.439809 ip-10-0-140-140 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 03 01:48:27.439825 ip-10-0-140-140 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot ebcc842b08f143b3ae332284918fea4f -- Nov 03 01:50:47.892668 ip-10-0-140-140 systemd[1]: Starting Kubernetes Kubelet... Nov 03 01:50:48.260143 ip-10-0-140-140 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 01:50:48.260143 ip-10-0-140-140 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 01:50:48.260143 ip-10-0-140-140 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 01:50:48.260143 ip-10-0-140-140 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 01:50:48.263339 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263256 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 01:50:48.263339 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263338 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 01:50:48.263339 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263342 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 01:50:48.263437 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263346 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 03 01:50:48.263437 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263349 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 01:50:48.263437 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263352 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 01:50:48.263437 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263371 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 01:50:48.263437 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263375 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 01:50:48.263437 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263378 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 01:50:48.263437 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263383 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 01:50:48.263437 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263386 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 01:50:48.263437 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263388 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 01:50:48.263437 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263392 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 01:50:48.263437 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263395 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 01:50:48.263437 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263398 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 01:50:48.263437 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263400 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 01:50:48.263437 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263403 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 01:50:48.263437 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263410 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 01:50:48.263437 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263413 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 01:50:48.263437 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263415 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 01:50:48.263437 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263418 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 01:50:48.263437 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263420 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 01:50:48.263437 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263424 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 01:50:48.263920 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263426 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 01:50:48.263920 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263429 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 01:50:48.263920 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263431 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 01:50:48.263920 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263434 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 01:50:48.263920 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263436 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 01:50:48.263920 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263439 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 01:50:48.263920 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263442 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 01:50:48.263920 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263445 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 01:50:48.263920 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263447 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 01:50:48.263920 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263450 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 01:50:48.263920 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263453 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 01:50:48.263920 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263456 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 01:50:48.263920 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263458 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 01:50:48.263920 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263461 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 01:50:48.263920 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263463 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 01:50:48.263920 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263466 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 01:50:48.263920 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263469 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 01:50:48.263920 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263473 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 01:50:48.263920 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263477 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 01:50:48.264377 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263480 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 01:50:48.264377 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263483 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 01:50:48.264377 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263485 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 01:50:48.264377 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263488 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 01:50:48.264377 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263490 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 01:50:48.264377 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263493 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 01:50:48.264377 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263495 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 01:50:48.264377 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263498 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 01:50:48.264377 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263501 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 01:50:48.264377 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263504 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 01:50:48.264377 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263506 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 01:50:48.264377 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263510 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 01:50:48.264377 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263513 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 01:50:48.264377 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263515 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 01:50:48.264377 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263518 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 01:50:48.264377 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263521 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 01:50:48.264377 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263523 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 01:50:48.264377 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263526 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 01:50:48.264377 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263529 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 01:50:48.264840 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263532 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 01:50:48.264840 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263534 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 01:50:48.264840 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263537 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 01:50:48.264840 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263539 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 01:50:48.264840 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263542 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 01:50:48.264840 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263544 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 01:50:48.264840 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263547 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 01:50:48.264840 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263549 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 01:50:48.264840 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263552 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 01:50:48.264840 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263554 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 01:50:48.264840 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263557 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 01:50:48.264840 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263560 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 01:50:48.264840 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263563 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 01:50:48.264840 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263566 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 01:50:48.264840 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263569 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 01:50:48.264840 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263571 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 01:50:48.264840 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263574 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 01:50:48.264840 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263577 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 01:50:48.264840 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263579 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 01:50:48.265342 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263582 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 01:50:48.265342 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263584 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 01:50:48.265342 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263587 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 01:50:48.265342 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263590 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 01:50:48.265342 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263593 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 01:50:48.265342 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263595 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 01:50:48.265342 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263598 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 01:50:48.265342 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263600 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 01:50:48.265342 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263604 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 01:50:48.265342 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263607 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 01:50:48.265342 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263609 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 01:50:48.265342 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263612 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 01:50:48.265342 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263614 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 01:50:48.265342 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263616 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 01:50:48.265342 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263619 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 01:50:48.265342 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263621 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 01:50:48.265342 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263624 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 01:50:48.265342 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263626 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 01:50:48.265342 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263629 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 01:50:48.265342 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263631 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 01:50:48.266139 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263634 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 01:50:48.266139 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263636 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 01:50:48.266139 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263639 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 01:50:48.266139 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263641 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 01:50:48.266139 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263645 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 01:50:48.266139 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.263647 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 01:50:48.266139 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264473 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 01:50:48.266139 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264481 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 01:50:48.266139 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264484 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 01:50:48.266139 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264487 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 01:50:48.266139 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264490 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 01:50:48.266139 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264492 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 01:50:48.266139 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264495 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 01:50:48.266139 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264498 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 01:50:48.266139 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264502 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 01:50:48.266139 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264504 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 01:50:48.266139 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264507 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 01:50:48.266139 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264510 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 01:50:48.266139 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264513 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 01:50:48.266662 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264516 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 01:50:48.266662 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264519 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 01:50:48.266662 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264522 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 01:50:48.266662 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264524 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 01:50:48.266662 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264527 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 01:50:48.266662 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264530 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 01:50:48.266662 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264532 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 01:50:48.266662 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264535 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 03 01:50:48.266662 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264537 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 01:50:48.266662 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264540 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 01:50:48.266662 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264542 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 01:50:48.266662 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264545 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 01:50:48.266662 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264547 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 01:50:48.266662 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264551 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 01:50:48.266662 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264556 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 01:50:48.266662 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264559 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 01:50:48.266662 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264562 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 01:50:48.266662 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264565 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 01:50:48.266662 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264568 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 01:50:48.267179 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264571 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 01:50:48.267179 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264573 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 01:50:48.267179 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264576 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 01:50:48.267179 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264579 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 01:50:48.267179 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264581 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 01:50:48.267179 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264584 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 01:50:48.267179 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264587 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 01:50:48.267179 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264589 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 01:50:48.267179 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264592 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 01:50:48.267179 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264594 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 01:50:48.267179 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264598 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 01:50:48.267179 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264601 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 01:50:48.267179 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264604 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 01:50:48.267179 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264606 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 01:50:48.267179 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264609 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 01:50:48.267179 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264612 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 01:50:48.267179 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264615 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 01:50:48.267179 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264617 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 01:50:48.267179 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264620 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 01:50:48.267179 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264622 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 01:50:48.267675 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264625 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 01:50:48.267675 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264627 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 01:50:48.267675 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264630 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 01:50:48.267675 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264632 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 01:50:48.267675 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264635 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 01:50:48.267675 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264637 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 01:50:48.267675 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264640 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 01:50:48.267675 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264643 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 01:50:48.267675 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264645 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 01:50:48.267675 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264647 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 01:50:48.267675 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264650 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 01:50:48.267675 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264652 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 01:50:48.267675 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264655 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 01:50:48.267675 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264657 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 01:50:48.267675 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264660 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 01:50:48.267675 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264662 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 01:50:48.267675 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264665 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 01:50:48.267675 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264668 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 01:50:48.267675 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264672 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 01:50:48.267675 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264674 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 01:50:48.268179 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264677 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 01:50:48.268179 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264679 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 01:50:48.268179 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264682 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 01:50:48.268179 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264685 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 01:50:48.268179 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264688 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 01:50:48.268179 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264690 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 01:50:48.268179 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264693 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 01:50:48.268179 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264696 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 01:50:48.268179 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264699 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 01:50:48.268179 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264701 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 01:50:48.268179 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264704 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 01:50:48.268179 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264706 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 01:50:48.268179 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264709 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 01:50:48.268179 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264711 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 01:50:48.268179 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264714 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 01:50:48.268179 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264716 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 01:50:48.268179 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264719 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 01:50:48.268179 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264721 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 01:50:48.268179 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264724 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 01:50:48.268648 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264726 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 01:50:48.268648 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264729 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 01:50:48.268648 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264731 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 01:50:48.268648 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264734 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 01:50:48.268648 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264736 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 01:50:48.268648 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264740 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 01:50:48.268648 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264744 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 01:50:48.268648 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264747 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 01:50:48.268648 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264750 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 01:50:48.268648 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264753 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 01:50:48.268648 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264756 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 01:50:48.268648 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264758 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 01:50:48.268648 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264761 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 01:50:48.268648 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264763 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 01:50:48.268648 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.264766 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 01:50:48.268648 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.265683 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 01:50:48.268648 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.265700 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 01:50:48.268648 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.265705 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 01:50:48.268648 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.265718 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 01:50:48.269105 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.265722 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 01:50:48.269105 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.265727 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 01:50:48.269105 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.265733 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 01:50:48.269105 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.265740 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 01:50:48.269105 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.265744 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 01:50:48.269105 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.265749 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 01:50:48.269105 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.265753 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 01:50:48.269105 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.265757 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 01:50:48.269105 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.265761 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 01:50:48.269105 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.265765 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 01:50:48.269105 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.265768 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 01:50:48.269105 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.265777 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 01:50:48.269105 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.265781 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 01:50:48.269105 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.265786 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 01:50:48.269105 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.265790 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 01:50:48.269105 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.265794 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 01:50:48.269105 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.265798 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 01:50:48.269105 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.265803 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 01:50:48.269105 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.265807 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 01:50:48.269105 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.265811 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 01:50:48.269573 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.265815 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 01:50:48.269573 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.265820 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 01:50:48.269573 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.265824 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 01:50:48.269573 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.265828 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 01:50:48.269573 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.265836 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 01:50:48.269573 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.265840 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 01:50:48.269573 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.265845 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 01:50:48.269573 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.265849 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 01:50:48.269573 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.265854 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 01:50:48.269573 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.265859 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 01:50:48.269573 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.265863 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 01:50:48.269573 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.265868 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 01:50:48.269573 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.265872 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 01:50:48.269573 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.265876 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 01:50:48.269573 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.265880 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 01:50:48.269573 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.265885 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 01:50:48.269573 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.265893 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 01:50:48.269573 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.265897 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 01:50:48.269573 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.265901 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 01:50:48.269573 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.265905 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 01:50:48.270045 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.265909 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 01:50:48.270045 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.265913 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 03 01:50:48.270045 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.265917 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 01:50:48.270045 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.265921 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 01:50:48.270045 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.265926 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 01:50:48.270045 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.265930 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 01:50:48.270045 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.265934 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 01:50:48.270045 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.265939 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 01:50:48.270045 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.265943 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 01:50:48.270045 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.265951 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 01:50:48.270045 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.265955 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 01:50:48.270045 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.265960 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 01:50:48.270045 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.265964 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 01:50:48.270045 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.265969 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 01:50:48.270045 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.265973 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 01:50:48.270045 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.265977 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 01:50:48.270045 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.265981 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 01:50:48.270045 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.265986 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 01:50:48.270045 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.265990 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 01:50:48.270045 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.265994 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 01:50:48.270520 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.265999 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 01:50:48.270520 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.266003 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 01:50:48.270520 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.266015 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 01:50:48.270520 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.266019 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 01:50:48.270520 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.266023 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 01:50:48.270520 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.266027 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 01:50:48.270520 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.266031 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 01:50:48.270520 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.266036 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 01:50:48.270520 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.266040 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 01:50:48.270520 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.266044 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 01:50:48.270520 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.266048 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 01:50:48.270520 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.266052 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 01:50:48.270520 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.266056 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 01:50:48.270520 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.266062 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 01:50:48.270520 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.266073 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 01:50:48.270520 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.266078 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 01:50:48.270520 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.266082 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 01:50:48.270520 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.266086 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 01:50:48.270520 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.266091 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 01:50:48.270987 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.266095 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 01:50:48.270987 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.266099 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 01:50:48.270987 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.266103 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 01:50:48.270987 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.266108 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 01:50:48.270987 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.266112 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 01:50:48.270987 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.266117 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 01:50:48.270987 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.266122 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 01:50:48.270987 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.266128 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 01:50:48.270987 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.266131 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 01:50:48.270987 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.266133 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 01:50:48.270987 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.266136 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 01:50:48.270987 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.266139 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 01:50:48.270987 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.266142 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 01:50:48.270987 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.266145 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 01:50:48.270987 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.266148 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 01:50:48.270987 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.266151 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 01:50:48.270987 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.266155 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 01:50:48.270987 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.266157 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 01:50:48.270987 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.266160 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 01:50:48.271436 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.266163 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 01:50:48.271436 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.266170 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 01:50:48.271436 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.266174 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 01:50:48.271436 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.266178 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 01:50:48.271436 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267263 2579 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 01:50:48.271436 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267293 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 01:50:48.271436 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267304 2579 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 01:50:48.271436 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267311 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 01:50:48.271436 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267318 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 01:50:48.271436 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267323 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 01:50:48.271436 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267330 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 01:50:48.271436 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267337 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 01:50:48.271436 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267341 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 01:50:48.271436 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267346 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 01:50:48.271436 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267352 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 01:50:48.271436 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267372 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 01:50:48.271436 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267377 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 01:50:48.271436 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267382 2579 flags.go:64] FLAG: --cgroup-root="" Nov 03 01:50:48.271436 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267386 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 01:50:48.271436 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267391 2579 flags.go:64] FLAG: --client-ca-file="" Nov 03 01:50:48.271436 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267395 2579 flags.go:64] FLAG: --cloud-provider="external" Nov 03 01:50:48.271436 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267400 2579 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 01:50:48.271436 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267408 2579 flags.go:64] FLAG: --cluster-domain="" Nov 03 01:50:48.271436 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267413 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 01:50:48.272022 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267417 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 01:50:48.272022 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267422 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 01:50:48.272022 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267427 2579 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 01:50:48.272022 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267433 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 01:50:48.272022 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267438 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 01:50:48.272022 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267443 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 01:50:48.272022 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267447 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 01:50:48.272022 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267452 2579 flags.go:64] FLAG: --contention-profiling="false" Nov 03 01:50:48.272022 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267457 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 01:50:48.272022 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267462 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 01:50:48.272022 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267467 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 01:50:48.272022 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267472 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 01:50:48.272022 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267477 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 01:50:48.272022 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267481 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 01:50:48.272022 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267484 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 01:50:48.272022 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267487 2579 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 01:50:48.272022 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267491 2579 flags.go:64] FLAG: --enable-server="true" Nov 03 01:50:48.272022 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267494 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 01:50:48.272022 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267499 2579 flags.go:64] FLAG: --event-burst="100" Nov 03 01:50:48.272022 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267502 2579 flags.go:64] FLAG: --event-qps="50" Nov 03 01:50:48.272022 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267506 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 01:50:48.272022 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267509 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 01:50:48.272022 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267512 2579 flags.go:64] FLAG: --eviction-hard="" Nov 03 01:50:48.272022 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267516 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 01:50:48.272618 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267519 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 01:50:48.272618 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267522 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 01:50:48.272618 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267525 2579 flags.go:64] FLAG: --eviction-soft="" Nov 03 01:50:48.272618 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267529 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 01:50:48.272618 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267531 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 01:50:48.272618 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267535 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 01:50:48.272618 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267538 2579 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 01:50:48.272618 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267540 2579 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 01:50:48.272618 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267543 2579 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 01:50:48.272618 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267546 2579 flags.go:64] FLAG: --feature-gates="" Nov 03 01:50:48.272618 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267551 2579 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 01:50:48.272618 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267554 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 01:50:48.272618 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267557 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 01:50:48.272618 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267560 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 01:50:48.272618 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267564 2579 flags.go:64] FLAG: --healthz-port="10248" Nov 03 01:50:48.272618 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267567 2579 flags.go:64] FLAG: --help="false" Nov 03 01:50:48.272618 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267570 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-140-140.us-west-2.compute.internal" Nov 03 01:50:48.272618 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267573 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 01:50:48.272618 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267576 2579 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 01:50:48.272618 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267579 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 01:50:48.272618 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267583 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 01:50:48.272618 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267587 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 01:50:48.272618 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267591 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 01:50:48.273174 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267593 2579 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 01:50:48.273174 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267596 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 01:50:48.273174 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267600 2579 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 01:50:48.273174 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267603 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 01:50:48.273174 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267606 2579 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 01:50:48.273174 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267609 2579 flags.go:64] FLAG: --kube-reserved="" Nov 03 01:50:48.273174 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267612 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 01:50:48.273174 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267615 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 01:50:48.273174 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267618 2579 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 01:50:48.273174 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267620 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 01:50:48.273174 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267623 2579 flags.go:64] FLAG: --lock-file="" Nov 03 01:50:48.273174 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267626 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 01:50:48.273174 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267629 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 01:50:48.273174 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267632 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 01:50:48.273174 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267638 2579 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 01:50:48.273174 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267640 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 01:50:48.273174 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267644 2579 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 01:50:48.273174 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267647 2579 flags.go:64] FLAG: --logging-format="text" Nov 03 01:50:48.273174 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267650 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 01:50:48.273174 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267654 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 01:50:48.273174 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267656 2579 flags.go:64] FLAG: --manifest-url="" Nov 03 01:50:48.273174 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267660 2579 flags.go:64] FLAG: --manifest-url-header="" Nov 03 01:50:48.273174 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267664 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 01:50:48.273174 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267667 2579 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 01:50:48.273174 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267671 2579 flags.go:64] FLAG: --max-pods="110" Nov 03 01:50:48.273780 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267674 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 01:50:48.273780 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267684 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 01:50:48.273780 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267689 2579 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 01:50:48.273780 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267692 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 01:50:48.273780 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267695 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 01:50:48.273780 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267702 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 01:50:48.273780 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267706 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 01:50:48.273780 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267732 2579 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 01:50:48.273780 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267736 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 01:50:48.273780 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267739 2579 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 01:50:48.273780 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267742 2579 flags.go:64] FLAG: --pod-cidr="" Nov 03 01:50:48.273780 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267745 2579 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 01:50:48.273780 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267748 2579 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 01:50:48.273780 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267751 2579 flags.go:64] FLAG: --pods-per-core="0" Nov 03 01:50:48.273780 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267754 2579 flags.go:64] FLAG: --port="10250" Nov 03 01:50:48.273780 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267757 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 01:50:48.273780 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267760 2579 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0a53d2985e9b38c8e" Nov 03 01:50:48.273780 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267763 2579 flags.go:64] FLAG: --qos-reserved="" Nov 03 01:50:48.273780 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267766 2579 flags.go:64] FLAG: --read-only-port="10255" Nov 03 01:50:48.273780 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267769 2579 flags.go:64] FLAG: --register-node="true" Nov 03 01:50:48.273780 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267772 2579 flags.go:64] FLAG: --register-with-taints="" Nov 03 01:50:48.273780 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267776 2579 flags.go:64] FLAG: --registry-burst="10" Nov 03 01:50:48.273780 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267779 2579 flags.go:64] FLAG: --registry-qps="5" Nov 03 01:50:48.273780 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267782 2579 flags.go:64] FLAG: --reserved-cpus="" Nov 03 01:50:48.273780 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267785 2579 flags.go:64] FLAG: --reserved-memory="" Nov 03 01:50:48.273780 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267790 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 01:50:48.274799 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267793 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 01:50:48.274799 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267796 2579 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 01:50:48.274799 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267799 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 01:50:48.274799 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267803 2579 flags.go:64] FLAG: --runonce="false" Nov 03 01:50:48.274799 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267806 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 01:50:48.274799 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267809 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 01:50:48.274799 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267812 2579 flags.go:64] FLAG: --seccomp-default="false" Nov 03 01:50:48.274799 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267815 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 01:50:48.274799 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267818 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 01:50:48.274799 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267821 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 01:50:48.274799 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267824 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 01:50:48.274799 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267829 2579 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 01:50:48.274799 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267832 2579 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 01:50:48.274799 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267835 2579 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 01:50:48.274799 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267838 2579 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 01:50:48.274799 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267841 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 01:50:48.274799 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267844 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 01:50:48.274799 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267847 2579 flags.go:64] FLAG: --system-cgroups="" Nov 03 01:50:48.274799 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267850 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 01:50:48.274799 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267857 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 01:50:48.274799 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267860 2579 flags.go:64] FLAG: --tls-cert-file="" Nov 03 01:50:48.274799 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267863 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 01:50:48.274799 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267896 2579 flags.go:64] FLAG: --tls-min-version="" Nov 03 01:50:48.274799 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267900 2579 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 01:50:48.274799 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267904 2579 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 01:50:48.275874 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267907 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 01:50:48.275874 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267911 2579 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 01:50:48.275874 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267914 2579 flags.go:64] FLAG: --v="2" Nov 03 01:50:48.275874 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267920 2579 flags.go:64] FLAG: --version="false" Nov 03 01:50:48.275874 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267924 2579 flags.go:64] FLAG: --vmodule="" Nov 03 01:50:48.275874 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267928 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 01:50:48.275874 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.267932 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 01:50:48.275874 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268075 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 01:50:48.275874 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268079 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 03 01:50:48.275874 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268083 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 01:50:48.275874 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268086 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 01:50:48.275874 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268088 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 01:50:48.275874 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268091 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 01:50:48.275874 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268094 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 01:50:48.275874 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268096 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 01:50:48.275874 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268099 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 01:50:48.275874 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268102 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 01:50:48.275874 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268105 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 01:50:48.275874 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268109 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 01:50:48.275874 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268112 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 01:50:48.275874 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268115 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 01:50:48.276706 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268117 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 01:50:48.276706 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268120 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 01:50:48.276706 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268123 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 01:50:48.276706 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268126 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 01:50:48.276706 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268128 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 01:50:48.276706 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268131 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 01:50:48.276706 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268133 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 01:50:48.276706 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268136 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 01:50:48.276706 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268139 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 01:50:48.276706 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268142 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 01:50:48.276706 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268144 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 01:50:48.276706 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268148 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 01:50:48.276706 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268150 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 01:50:48.276706 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268155 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 01:50:48.276706 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268157 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 01:50:48.276706 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268160 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 01:50:48.276706 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268163 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 01:50:48.276706 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268166 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 01:50:48.276706 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268168 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 01:50:48.276706 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268171 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 01:50:48.277586 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268173 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 01:50:48.277586 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268176 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 01:50:48.277586 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268180 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 01:50:48.277586 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268183 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 01:50:48.277586 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268185 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 01:50:48.277586 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268188 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 01:50:48.277586 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268191 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 01:50:48.277586 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268194 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 01:50:48.277586 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268196 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 01:50:48.277586 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268201 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 01:50:48.277586 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268204 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 01:50:48.277586 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268207 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 01:50:48.277586 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268210 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 01:50:48.277586 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268212 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 01:50:48.277586 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268215 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 01:50:48.277586 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268218 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 01:50:48.277586 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268220 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 01:50:48.277586 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268223 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 01:50:48.277586 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268225 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 01:50:48.277586 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268228 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 01:50:48.278462 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268231 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 01:50:48.278462 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268233 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 01:50:48.278462 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268236 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 01:50:48.278462 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268238 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 01:50:48.278462 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268243 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 01:50:48.278462 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268248 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 01:50:48.278462 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268251 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 01:50:48.278462 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268254 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 01:50:48.278462 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268256 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 01:50:48.278462 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268259 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 01:50:48.278462 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268262 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 01:50:48.278462 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268264 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 01:50:48.278462 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268267 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 01:50:48.278462 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268269 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 01:50:48.278462 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268272 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 01:50:48.278462 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268274 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 01:50:48.278462 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268277 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 01:50:48.278462 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268280 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 01:50:48.278462 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268282 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 01:50:48.279059 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268284 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 01:50:48.279059 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268287 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 01:50:48.279059 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268291 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 01:50:48.279059 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268294 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 01:50:48.279059 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268296 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 01:50:48.279059 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268299 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 01:50:48.279059 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268302 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 01:50:48.279059 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268305 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 01:50:48.279059 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268307 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 01:50:48.279059 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268311 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 01:50:48.279059 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268315 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 01:50:48.279059 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268318 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 01:50:48.279059 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268321 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 01:50:48.279059 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268324 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 01:50:48.279059 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268326 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 01:50:48.279059 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268329 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 01:50:48.279059 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268331 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 01:50:48.279059 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268334 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 01:50:48.279059 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268338 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 01:50:48.279617 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268341 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 01:50:48.279617 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268344 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 01:50:48.279617 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268346 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 01:50:48.279617 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268349 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 01:50:48.279617 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268351 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 01:50:48.279617 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268366 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 01:50:48.279617 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268370 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 01:50:48.279617 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268372 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 01:50:48.279617 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268375 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 01:50:48.279617 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268377 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 01:50:48.279617 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268380 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 01:50:48.279617 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268383 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 01:50:48.279617 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268385 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 01:50:48.279617 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268388 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 01:50:48.279617 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.268418 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 01:50:48.279617 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268456 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 01:50:48.280297 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268461 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 01:50:48.280297 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268464 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 01:50:48.280297 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268467 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 01:50:48.280297 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268470 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 01:50:48.280297 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268473 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 01:50:48.280297 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268476 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 01:50:48.280297 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268478 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 01:50:48.280297 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268481 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 01:50:48.280297 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268483 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 01:50:48.280297 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268486 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 01:50:48.280297 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268489 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 01:50:48.280297 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268491 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 01:50:48.280297 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268494 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 01:50:48.280297 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268496 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 01:50:48.280297 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268499 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 01:50:48.280297 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268505 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 01:50:48.280297 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268508 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 01:50:48.280297 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268512 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 01:50:48.280297 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268516 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 01:50:48.281057 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268519 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 01:50:48.281057 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268522 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 01:50:48.281057 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268525 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 01:50:48.281057 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268527 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 01:50:48.281057 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268530 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 01:50:48.281057 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268532 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 01:50:48.281057 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268535 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 01:50:48.281057 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268538 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 01:50:48.281057 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268540 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 01:50:48.281057 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268543 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 01:50:48.281057 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268545 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 01:50:48.281057 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268548 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 01:50:48.281057 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268551 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 01:50:48.281057 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268554 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 01:50:48.281057 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268557 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 01:50:48.281057 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268560 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 01:50:48.281057 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268563 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 01:50:48.281057 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268565 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 01:50:48.281057 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268568 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 01:50:48.281516 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268571 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 01:50:48.281516 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268573 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 01:50:48.281516 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268576 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 01:50:48.281516 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268579 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 01:50:48.281516 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268581 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 03 01:50:48.281516 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268584 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 01:50:48.281516 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268586 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 01:50:48.281516 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268589 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 01:50:48.281516 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268592 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 01:50:48.281516 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268596 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 01:50:48.281516 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268598 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 01:50:48.281516 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268601 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 01:50:48.281516 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268604 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 01:50:48.281516 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268607 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 01:50:48.281516 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268609 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 01:50:48.281516 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268612 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 01:50:48.281516 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268614 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 01:50:48.281516 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268617 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 01:50:48.281516 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268620 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 01:50:48.281516 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268623 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 01:50:48.282122 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268626 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 01:50:48.282122 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268628 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 01:50:48.282122 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268631 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 01:50:48.282122 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268633 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 01:50:48.282122 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268636 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 01:50:48.282122 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268638 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 01:50:48.282122 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268642 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 01:50:48.282122 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268645 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 01:50:48.282122 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268648 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 01:50:48.282122 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268650 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 01:50:48.282122 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268653 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 01:50:48.282122 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268656 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 01:50:48.282122 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268659 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 01:50:48.282122 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268662 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 01:50:48.282122 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268665 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 01:50:48.282122 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268667 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 01:50:48.282122 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268670 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 01:50:48.282122 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268672 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 01:50:48.282122 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268675 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 01:50:48.282122 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268677 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 01:50:48.282834 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268680 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 01:50:48.282834 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268684 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 01:50:48.282834 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268686 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 01:50:48.282834 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268689 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 01:50:48.282834 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268692 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 01:50:48.282834 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268695 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 01:50:48.282834 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268697 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 01:50:48.282834 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268700 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 01:50:48.282834 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268703 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 01:50:48.282834 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268705 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 01:50:48.282834 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268708 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 01:50:48.282834 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268710 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 01:50:48.282834 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268713 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 01:50:48.282834 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268715 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 01:50:48.282834 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268718 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 01:50:48.282834 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268720 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 01:50:48.282834 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268723 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 01:50:48.282834 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268725 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 01:50:48.282834 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268729 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 01:50:48.283267 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268732 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 01:50:48.283267 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268735 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 01:50:48.283267 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268739 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 01:50:48.283267 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268742 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 01:50:48.283267 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268745 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 01:50:48.283267 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268749 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 01:50:48.283267 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268752 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 01:50:48.283267 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.268754 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 01:50:48.283267 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.276446 2579 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 01:50:48.283267 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.276638 2579 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 01:50:48.283267 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.276663 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 01:50:48.283267 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.276672 2579 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 01:50:48.283267 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.276758 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 01:50:48.283267 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.276767 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 01:50:48.283267 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.276772 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 01:50:48.283267 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.276777 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 01:50:48.283267 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.276781 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 01:50:48.283267 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.276786 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 01:50:48.283267 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.276791 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 01:50:48.283807 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.276796 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 01:50:48.283807 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.276800 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 01:50:48.283807 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.276804 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 01:50:48.283807 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.276808 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 01:50:48.283807 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.276813 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 01:50:48.283807 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.276817 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 01:50:48.283807 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.276822 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 01:50:48.283807 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.276826 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 01:50:48.283807 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.276830 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 01:50:48.283807 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.276835 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 01:50:48.283807 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.276839 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 01:50:48.283807 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.276844 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 01:50:48.283807 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.276848 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 01:50:48.283807 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.276852 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 01:50:48.283807 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.276857 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 01:50:48.283807 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.276861 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 01:50:48.283807 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.276865 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 01:50:48.283807 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.276870 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 01:50:48.283807 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.276874 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 01:50:48.283807 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.276878 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 01:50:48.284334 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.276884 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 01:50:48.284334 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.276889 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 01:50:48.284334 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.276897 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 01:50:48.284334 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.276902 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 01:50:48.284334 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.276907 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 01:50:48.284334 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.276911 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 01:50:48.284334 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.276915 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 01:50:48.284334 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.276919 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 01:50:48.284334 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.276924 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 01:50:48.284334 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.276929 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 01:50:48.284334 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.276945 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 01:50:48.284334 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.276949 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 01:50:48.284334 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.276954 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 01:50:48.284334 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.276958 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 01:50:48.284334 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.276962 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 01:50:48.284334 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.276966 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 01:50:48.284334 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.276970 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 01:50:48.284334 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.276975 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 01:50:48.284334 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.276979 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 01:50:48.284334 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.276984 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 01:50:48.285137 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.276988 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 01:50:48.285137 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.276995 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 01:50:48.285137 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277001 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 01:50:48.285137 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277006 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 01:50:48.285137 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277010 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 01:50:48.285137 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277014 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 01:50:48.285137 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277018 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 01:50:48.285137 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277023 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 01:50:48.285137 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277027 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 01:50:48.285137 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277031 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 01:50:48.285137 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277035 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 01:50:48.285137 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277039 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 01:50:48.285137 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277043 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 01:50:48.285137 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277047 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 01:50:48.285137 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277053 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 01:50:48.285137 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277059 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 01:50:48.285137 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277063 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 01:50:48.285137 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277070 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 01:50:48.285137 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277075 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 01:50:48.285895 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277080 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 01:50:48.285895 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277085 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 01:50:48.285895 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277090 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 01:50:48.285895 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277094 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 01:50:48.285895 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277098 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 01:50:48.285895 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277102 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 01:50:48.285895 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277106 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 01:50:48.285895 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277111 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 01:50:48.285895 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277115 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 01:50:48.285895 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277119 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 01:50:48.285895 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277123 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 01:50:48.285895 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277127 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 01:50:48.285895 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277131 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 01:50:48.285895 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277135 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 01:50:48.285895 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277140 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 01:50:48.285895 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277144 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 01:50:48.285895 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277148 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 01:50:48.285895 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277152 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 01:50:48.285895 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277156 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 01:50:48.286740 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277161 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 01:50:48.286740 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277165 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 01:50:48.286740 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277169 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 01:50:48.286740 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277173 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 01:50:48.286740 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277177 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 01:50:48.286740 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277181 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 01:50:48.286740 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277185 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 01:50:48.286740 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277189 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 01:50:48.286740 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277195 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 01:50:48.286740 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277199 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 01:50:48.286740 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277205 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 01:50:48.286740 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277209 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 01:50:48.286740 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277213 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 01:50:48.286740 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277217 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 01:50:48.286740 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277221 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 01:50:48.286740 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277225 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 01:50:48.286740 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277230 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 01:50:48.286740 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277234 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 01:50:48.286740 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277238 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 01:50:48.287423 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277242 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 01:50:48.287423 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277246 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 03 01:50:48.287423 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.277287 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 01:50:48.287423 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277558 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 01:50:48.287423 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277567 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 01:50:48.287423 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277572 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 01:50:48.287423 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277578 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 01:50:48.287423 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277582 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 01:50:48.287423 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277586 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 01:50:48.287423 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277590 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 01:50:48.287423 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277594 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 01:50:48.287423 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277598 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 01:50:48.287423 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277601 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 01:50:48.287423 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277605 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 01:50:48.287423 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277609 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 01:50:48.287423 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277613 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 01:50:48.288351 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277617 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 01:50:48.288351 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277622 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 01:50:48.288351 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277626 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 01:50:48.288351 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277630 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 01:50:48.288351 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277635 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 01:50:48.288351 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277639 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 01:50:48.288351 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277646 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 01:50:48.288351 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277652 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 01:50:48.288351 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277659 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 03 01:50:48.288351 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277664 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 01:50:48.288351 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277669 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 01:50:48.288351 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277673 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 01:50:48.288351 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277678 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 01:50:48.288351 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277682 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 01:50:48.288351 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277686 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 01:50:48.288351 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277690 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 01:50:48.288351 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277694 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 01:50:48.288351 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277698 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 01:50:48.288351 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277702 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 01:50:48.288885 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277707 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 01:50:48.288885 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277711 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 01:50:48.288885 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277715 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 01:50:48.288885 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277720 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 01:50:48.288885 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277724 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 01:50:48.288885 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277728 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 01:50:48.288885 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277732 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 01:50:48.288885 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277736 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 01:50:48.288885 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277740 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 01:50:48.288885 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277744 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 01:50:48.288885 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277748 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 01:50:48.288885 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277753 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 01:50:48.288885 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277757 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 01:50:48.288885 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277761 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 01:50:48.288885 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277765 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 01:50:48.288885 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277769 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 01:50:48.288885 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277773 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 01:50:48.288885 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277778 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 01:50:48.288885 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277782 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 01:50:48.288885 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277787 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 01:50:48.289371 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277791 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 01:50:48.289371 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277795 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 01:50:48.289371 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277801 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 01:50:48.289371 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277805 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 01:50:48.289371 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277809 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 01:50:48.289371 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277813 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 01:50:48.289371 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277817 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 01:50:48.289371 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277822 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 01:50:48.289371 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277825 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 01:50:48.289371 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277829 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 01:50:48.289371 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277834 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 01:50:48.289371 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277837 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 01:50:48.289371 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277842 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 01:50:48.289371 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277845 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 01:50:48.289371 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277850 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 01:50:48.289371 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277854 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 01:50:48.289371 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277859 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 01:50:48.289371 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277863 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 01:50:48.289371 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277867 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 01:50:48.289371 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277871 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 01:50:48.289881 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277875 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 01:50:48.289881 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277879 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 01:50:48.289881 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277884 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 01:50:48.289881 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277888 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 01:50:48.289881 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277892 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 01:50:48.289881 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277896 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 01:50:48.289881 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277900 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 01:50:48.289881 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277904 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 01:50:48.289881 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277909 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 01:50:48.289881 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277913 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 01:50:48.289881 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277918 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 01:50:48.289881 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277921 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 01:50:48.289881 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277926 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 01:50:48.289881 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277930 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 01:50:48.289881 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277934 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 01:50:48.289881 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277943 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 01:50:48.289881 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277948 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 01:50:48.289881 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277952 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 01:50:48.289881 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277956 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 01:50:48.290518 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277960 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 01:50:48.290518 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277964 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 01:50:48.290518 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277969 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 01:50:48.290518 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277973 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 01:50:48.290518 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277977 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 01:50:48.290518 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277981 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 01:50:48.290518 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277985 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 01:50:48.290518 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277989 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 01:50:48.290518 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277994 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 01:50:48.290518 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.277998 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 01:50:48.290518 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278002 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 01:50:48.290518 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278006 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 01:50:48.290518 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278013 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 01:50:48.290518 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278019 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 01:50:48.290518 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278024 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 01:50:48.290518 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.278066 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 01:50:48.290889 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278122 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 01:50:48.290889 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278129 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 01:50:48.290889 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278133 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 01:50:48.290889 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278138 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 01:50:48.290889 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278142 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 01:50:48.290889 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278146 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 01:50:48.290889 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278151 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 01:50:48.290889 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278156 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 01:50:48.290889 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278160 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 01:50:48.290889 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278164 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 01:50:48.290889 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278169 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 01:50:48.290889 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278173 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 01:50:48.290889 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278177 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 01:50:48.290889 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278182 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 01:50:48.290889 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278186 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 01:50:48.290889 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278190 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 01:50:48.290889 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278194 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 01:50:48.290889 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278198 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 01:50:48.290889 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278203 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 01:50:48.291556 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278207 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 01:50:48.291556 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278211 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 01:50:48.291556 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278215 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 01:50:48.291556 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278220 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 01:50:48.291556 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278224 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 01:50:48.291556 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278228 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 01:50:48.291556 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278232 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 01:50:48.291556 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278237 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 01:50:48.291556 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278241 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 01:50:48.291556 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278245 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 01:50:48.291556 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278249 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 01:50:48.291556 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278254 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 01:50:48.291556 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278258 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 01:50:48.291556 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278262 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 01:50:48.291556 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278266 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 01:50:48.291556 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278270 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 01:50:48.291556 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278275 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 01:50:48.291556 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278282 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 01:50:48.291556 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278287 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 01:50:48.291556 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278291 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 01:50:48.292075 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278296 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 01:50:48.292075 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278300 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 01:50:48.292075 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278304 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 01:50:48.292075 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278308 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 01:50:48.292075 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278312 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 01:50:48.292075 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278316 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 01:50:48.292075 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278320 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 01:50:48.292075 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278326 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 01:50:48.292075 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278330 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 01:50:48.292075 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278334 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 01:50:48.292075 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278338 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 01:50:48.292075 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278342 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 01:50:48.292075 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278346 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 01:50:48.292075 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278350 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 01:50:48.292075 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278372 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 01:50:48.292075 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278377 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 01:50:48.292075 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278381 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 01:50:48.292075 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278385 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 01:50:48.292075 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278389 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 01:50:48.292552 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278395 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 01:50:48.292552 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278401 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 01:50:48.292552 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278405 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 03 01:50:48.292552 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278410 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 01:50:48.292552 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278414 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 01:50:48.292552 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278418 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 01:50:48.292552 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278422 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 01:50:48.292552 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278426 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 01:50:48.292552 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278430 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 01:50:48.292552 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278434 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 01:50:48.292552 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278438 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 01:50:48.292552 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278443 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 01:50:48.292552 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278447 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 01:50:48.292552 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278451 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 01:50:48.292552 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278455 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 01:50:48.292552 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278459 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 01:50:48.292552 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278464 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 01:50:48.292552 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278468 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 01:50:48.292552 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278472 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 01:50:48.293092 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278475 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 01:50:48.293092 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278479 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 01:50:48.293092 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278484 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 01:50:48.293092 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278488 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 01:50:48.293092 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278493 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 01:50:48.293092 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278497 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 01:50:48.293092 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278501 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 01:50:48.293092 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278506 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 01:50:48.293092 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278513 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 01:50:48.293092 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278517 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 01:50:48.293092 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278521 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 01:50:48.293092 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278525 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 01:50:48.293092 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278529 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 01:50:48.293092 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278534 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 01:50:48.293092 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278538 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 01:50:48.293092 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278542 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 01:50:48.293092 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278546 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 01:50:48.293092 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278550 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 01:50:48.293092 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278555 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 01:50:48.293092 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278559 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 01:50:48.293579 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278563 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 01:50:48.293579 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278567 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 01:50:48.293579 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278571 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 01:50:48.293579 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278575 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 01:50:48.293579 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278580 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 01:50:48.293579 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278584 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 01:50:48.293579 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278588 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 01:50:48.293579 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278592 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 01:50:48.293579 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:48.278596 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 01:50:48.293579 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.278689 2579 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 01:50:48.293579 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.281613 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 01:50:48.293579 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.282464 2579 server.go:1014] "Starting client certificate rotation" Nov 03 01:50:48.293579 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.282560 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 01:50:48.293579 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.282597 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 01:50:48.293579 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.284866 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 01:50:48.293579 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.286688 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 01:50:48.293954 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.293912 2579 log.go:25] "Validated CRI v1 runtime API" Nov 03 01:50:48.297568 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.297554 2579 log.go:25] "Validated CRI v1 image API" Nov 03 01:50:48.298425 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.298380 2579 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 01:50:48.302968 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.302942 2579 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 9a63c23e-b746-4088-9b39-6cf1e227dc8d:/dev/nvme0n1p3 fcac2184-e10a-43b1-82d5-dd214366aa77:/dev/nvme0n1p4] Nov 03 01:50:48.303067 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.302964 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 01:50:48.308062 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.307938 2579 manager.go:217] Machine: {Timestamp:2026-11-03 01:50:48.306724793 +0000 UTC m=+0.310172246 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3158275 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec241a7425d58df988e7452d3ec16766 SystemUUID:ec241a74-25d5-8df9-88e7-452d3ec16766 BootID:ebcc842b-08f1-43b3-ae33-2284918fea4f Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:e1:74:bd:49 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:e1:74:bd:49 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:92:41:f2:20:eb:eb Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 01:50:48.308766 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.308753 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 01:50:48.308923 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.308899 2579 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 01:50:48.311828 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.311802 2579 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 01:50:48.311967 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.311829 2579 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-140.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 01:50:48.312526 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.312515 2579 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 01:50:48.312526 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.312526 2579 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 01:50:48.312580 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.312552 2579 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 01:50:48.312580 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.312566 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 01:50:48.313587 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.313577 2579 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 01:50:48.317722 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.317708 2579 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 01:50:48.317855 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.317845 2579 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 01:50:48.318519 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.318508 2579 kubelet.go:493] "Attempting to sync node with API server" Nov 03 01:50:48.318549 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.318523 2579 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 01:50:48.318549 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.318534 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 01:50:48.318549 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.318544 2579 kubelet.go:405] "Adding apiserver pod source" Nov 03 01:50:48.318626 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.318553 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 01:50:48.318893 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.318870 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 01:50:48.321063 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.321040 2579 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 01:50:48.326850 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.326836 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 01:50:48.327702 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.327689 2579 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 01:50:48.327767 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.327739 2579 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 01:50:48.327913 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.327902 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 01:50:48.327947 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.327918 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 01:50:48.327947 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.327924 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 01:50:48.327947 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.327930 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 01:50:48.327947 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.327936 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 01:50:48.327947 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.327941 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 01:50:48.328462 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.328453 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 01:50:48.328499 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.328463 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 01:50:48.328499 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.328470 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 01:50:48.328499 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.328476 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 01:50:48.328499 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.328485 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 01:50:48.328499 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.328493 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 01:50:48.329073 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.329063 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 01:50:48.329102 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.329075 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 01:50:48.333091 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.333068 2579 server.go:1263] "Started kubelet" Nov 03 01:50:48.333611 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.333138 2579 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 01:50:48.333611 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.333195 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 01:50:48.333611 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.333392 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 01:50:48.333985 ip-10-0-140-140 systemd[1]: Started Kubernetes Kubelet. Nov 03 01:50:48.334715 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.334686 2579 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 01:50:48.336961 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.336929 2579 server.go:317] "Adding debug handlers to kubelet server" Nov 03 01:50:48.342166 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:48.342130 2579 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 01:50:48.342403 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.342380 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 01:50:48.342892 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.342878 2579 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 01:50:48.343983 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:48.343951 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-140.us-west-2.compute.internal\" not found" Nov 03 01:50:48.343983 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.343980 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 01:50:48.344152 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.343991 2579 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 01:50:48.344200 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.344154 2579 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 01:50:48.344271 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.344259 2579 reconstruct.go:97] "Volume reconstruction finished" Nov 03 01:50:48.344318 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.344273 2579 reconciler.go:29] "Reconciler: start to sync state" Nov 03 01:50:48.345652 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.345609 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 01:50:48.345652 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.345639 2579 factory.go:55] Registering systemd factory Nov 03 01:50:48.345652 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.345655 2579 factory.go:223] Registration of the systemd container factory successfully Nov 03 01:50:48.346561 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.346544 2579 factory.go:153] Registering CRI-O factory Nov 03 01:50:48.346632 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.346582 2579 factory.go:223] Registration of the crio container factory successfully Nov 03 01:50:48.346680 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.346651 2579 factory.go:103] Registering Raw factory Nov 03 01:50:48.346708 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.346680 2579 manager.go:1199] Started watching for new ooms in manager Nov 03 01:50:48.347829 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.347813 2579 manager.go:319] Starting recovery of all containers Nov 03 01:50:48.358769 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.358750 2579 manager.go:324] Recovery completed Nov 03 01:50:48.364732 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.364715 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 01:50:48.366834 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.366801 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-140.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 01:50:48.366909 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.366856 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-140.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 01:50:48.366909 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.366872 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-140.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 01:50:48.367471 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.367457 2579 cpu_manager.go:226] "Starting" policy="none" Nov 03 01:50:48.367471 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.367469 2579 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 01:50:48.367573 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.367479 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 01:50:48.367573 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.367489 2579 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 01:50:48.368260 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.368244 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-140.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 01:50:48.368318 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.368274 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-140.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 01:50:48.368318 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.368288 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-140.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 01:50:48.369744 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.369729 2579 policy_none.go:50] "Start" Nov 03 01:50:48.369804 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.369748 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 01:50:48.369804 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.369759 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 01:50:48.371830 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.371806 2579 policy_none.go:44] "Start" Nov 03 01:50:48.407192 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.407157 2579 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 01:50:48.418930 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.418914 2579 manager.go:341] "Starting Device Plugin manager" Nov 03 01:50:48.419051 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:48.418949 2579 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 01:50:48.419051 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.418957 2579 server.go:87] "Starting device plugin registration server" Nov 03 01:50:48.419282 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.419270 2579 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 01:50:48.419342 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.419283 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 01:50:48.419838 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.419512 2579 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 01:50:48.419838 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.419594 2579 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 01:50:48.419838 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.419600 2579 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 01:50:48.420153 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:48.420130 2579 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 01:50:48.420246 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:48.420199 2579 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-140.us-west-2.compute.internal\" not found" Nov 03 01:50:48.470112 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.470079 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 01:50:48.471253 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.471238 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 01:50:48.471305 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.471259 2579 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 01:50:48.471305 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.471280 2579 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 01:50:48.471404 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:48.471313 2579 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 03 01:50:48.475435 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.475415 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-140.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 03 01:50:48.475435 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:48.475424 2579 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-140.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 03 01:50:48.476274 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:48.475425 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-140.us-west-2.compute.internal.18e465f38def61c7 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-140.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-140.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 01:50:48.333033927 +0000 UTC m=+0.336481382,LastTimestamp:2026-11-03 01:50:48.333033927 +0000 UTC m=+0.336481382,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-140.us-west-2.compute.internal,}" Nov 03 01:50:48.520197 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.520098 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 01:50:48.521162 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.521143 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-140.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 01:50:48.521264 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.521173 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-140.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 01:50:48.521264 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.521186 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-140.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 01:50:48.521264 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.521212 2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-140.us-west-2.compute.internal" Nov 03 01:50:48.534946 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.534926 2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 01:50:48.535024 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.534996 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gkd2l" Nov 03 01:50:48.572257 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.572220 2579 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-140.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-140.us-west-2.compute.internal"] Nov 03 01:50:48.572340 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.572297 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 01:50:48.573367 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.573332 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-140.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 01:50:48.573438 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.573377 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-140.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 01:50:48.573438 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.573387 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-140.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 01:50:48.575664 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.575651 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 01:50:48.575860 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.575846 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-140.us-west-2.compute.internal" Nov 03 01:50:48.575906 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.575877 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 01:50:48.576509 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.576489 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-140.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 01:50:48.576594 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.576526 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-140.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 01:50:48.576594 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.576532 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-140.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 01:50:48.576594 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.576541 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-140.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 01:50:48.576594 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.576562 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-140.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 01:50:48.576594 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.576575 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-140.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 01:50:48.578706 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.578689 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-140.us-west-2.compute.internal" Nov 03 01:50:48.578801 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.578714 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 01:50:48.579380 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.579351 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-140.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 01:50:48.579468 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.579389 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-140.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 01:50:48.579468 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.579398 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-140.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 01:50:48.606956 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:48.606939 2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-140.us-west-2.compute.internal\" not found" node="ip-10-0-140-140.us-west-2.compute.internal" Nov 03 01:50:48.609496 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:48.609481 2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-140.us-west-2.compute.internal\" not found" node="ip-10-0-140-140.us-west-2.compute.internal" Nov 03 01:50:48.622263 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.622245 2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 01:50:48.623489 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.623476 2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 01:50:48.625831 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.625816 2579 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 01:50:48.646040 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.646011 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d1cefe6117be002b0e139e82f22d4917-config\") pod \"kube-apiserver-proxy-ip-10-0-140-140.us-west-2.compute.internal\" (UID: \"d1cefe6117be002b0e139e82f22d4917\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-140.us-west-2.compute.internal" Nov 03 01:50:48.646137 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.646045 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3b5ab86e21798d22090c574595774d10-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-140.us-west-2.compute.internal\" (UID: \"3b5ab86e21798d22090c574595774d10\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-140.us-west-2.compute.internal" Nov 03 01:50:48.646137 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.646061 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b5ab86e21798d22090c574595774d10-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-140.us-west-2.compute.internal\" (UID: \"3b5ab86e21798d22090c574595774d10\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-140.us-west-2.compute.internal" Nov 03 01:50:48.716813 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.716781 2579 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-140.us-west-2.compute.internal" Nov 03 01:50:48.716813 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:48.716813 2579 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-140.us-west-2.compute.internal\": node \"ip-10-0-140-140.us-west-2.compute.internal\" not found" Nov 03 01:50:48.717031 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.716812 2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 01:50:48.743707 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.743673 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-140.us-west-2.compute.internal" Nov 03 01:50:48.746350 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.746323 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3b5ab86e21798d22090c574595774d10-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-140.us-west-2.compute.internal\" (UID: \"3b5ab86e21798d22090c574595774d10\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-140.us-west-2.compute.internal" Nov 03 01:50:48.746350 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.746351 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b5ab86e21798d22090c574595774d10-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-140.us-west-2.compute.internal\" (UID: \"3b5ab86e21798d22090c574595774d10\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-140.us-west-2.compute.internal" Nov 03 01:50:48.746513 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.746380 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d1cefe6117be002b0e139e82f22d4917-config\") pod \"kube-apiserver-proxy-ip-10-0-140-140.us-west-2.compute.internal\" (UID: \"d1cefe6117be002b0e139e82f22d4917\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-140.us-west-2.compute.internal" Nov 03 01:50:48.746513 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.746445 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d1cefe6117be002b0e139e82f22d4917-config\") pod \"kube-apiserver-proxy-ip-10-0-140-140.us-west-2.compute.internal\" (UID: \"d1cefe6117be002b0e139e82f22d4917\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-140.us-west-2.compute.internal" Nov 03 01:50:48.746513 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.746442 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3b5ab86e21798d22090c574595774d10-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-140.us-west-2.compute.internal\" (UID: \"3b5ab86e21798d22090c574595774d10\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-140.us-west-2.compute.internal" Nov 03 01:50:48.746513 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.746446 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b5ab86e21798d22090c574595774d10-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-140.us-west-2.compute.internal\" (UID: \"3b5ab86e21798d22090c574595774d10\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-140.us-west-2.compute.internal" Nov 03 01:50:48.800871 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.800794 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 01:50:48.801575 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.801561 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-140.us-west-2.compute.internal" Nov 03 01:50:48.855325 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.855297 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 01:50:48.911189 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.911161 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-140.us-west-2.compute.internal" Nov 03 01:50:48.913291 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:48.913272 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-140.us-west-2.compute.internal" Nov 03 01:50:49.281972 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.281923 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 01:50:49.282573 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.282097 2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 01:50:49.282573 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.282139 2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 01:50:49.282573 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.282138 2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 01:50:49.282573 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:49.282158 2579 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://ab176cf1b8f3e4b2b9a47658d610d089-75b99088ee9799d1.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-140-140.us-west-2.compute.internal.18e465f38ff32108\": read tcp 10.0.140.140:58888->18.118.92.231:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-140-140.us-west-2.compute.internal.18e465f38ff32108 default 5346 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-140.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-140.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-140.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 01:50:48 +0000 UTC,LastTimestamp:2026-11-03 01:50:48.573348187 +0000 UTC m=+0.576795640,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-140.us-west-2.compute.internal,}" Nov 03 01:50:49.319553 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.319523 2579 apiserver.go:52] "Watching apiserver" Nov 03 01:50:49.343014 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.342969 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 01:50:49.540568 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.540475 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 01:45:49 +0000 UTC" deadline="2028-06-19 17:19:48.328497193 +0000 UTC" Nov 03 01:50:49.540568 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.540505 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14271h28m58.787995253s" Nov 03 01:50:49.619687 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.619654 2579 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 01:50:49.620036 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.620014 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-140-140.us-west-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdpjb","openshift-cluster-node-tuning-operator/tuned-qm8zc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-140.us-west-2.compute.internal","openshift-multus/multus-trr75","openshift-network-operator/iptables-alerter-vmjc5","openshift-ovn-kubernetes/ovnkube-node-x452r","kube-system/konnectivity-agent-f524q","openshift-multus/multus-additional-cni-plugins-gwlr2","openshift-dns/node-resolver-7zdvv","openshift-image-registry/node-ca-gjs9f","openshift-multus/network-metrics-daemon-vzfhc","openshift-network-diagnostics/network-check-target-f5qvs","kube-system/global-pull-secret-syncer-kkngd"] Nov 03 01:50:49.622866 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.622840 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdpjb" Nov 03 01:50:49.624998 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.624973 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:50:49.627227 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.627205 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-trr75" Nov 03 01:50:49.629518 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.629500 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vmjc5" Nov 03 01:50:49.631745 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.631718 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:50:49.634061 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.634042 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-f524q" Nov 03 01:50:49.636545 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.636527 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gwlr2" Nov 03 01:50:49.638961 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.638936 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7zdvv" Nov 03 01:50:49.641330 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.641314 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gjs9f" Nov 03 01:50:49.643728 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.643711 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vzfhc" Nov 03 01:50:49.643816 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:49.643770 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vzfhc" podUID="dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b" Nov 03 01:50:49.646161 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.646140 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5qvs" Nov 03 01:50:49.646252 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:49.646216 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5qvs" podUID="6ae1437e-8885-42a5-b092-3862cd404c60" Nov 03 01:50:49.649220 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.649203 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kkngd" Nov 03 01:50:49.649300 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:49.649266 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kkngd" podUID="28f00eb6-78c2-4d93-bde5-c1ea12601dbd" Nov 03 01:50:49.651923 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.651900 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/df76407a-ce41-4a4a-b51e-8420b817f97c-run\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:50:49.652016 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.651938 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-system-cni-dir\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:50:49.652016 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.651969 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-multus-socket-dir-parent\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:50:49.652108 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.652086 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-etc-openvswitch\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:50:49.652149 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.652120 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9152cf36-4997-470e-8902-802a96ebc2c7-cni-binary-copy\") pod \"multus-additional-cni-plugins-gwlr2\" (UID: \"9152cf36-4997-470e-8902-802a96ebc2c7\") " pod="openshift-multus/multus-additional-cni-plugins-gwlr2" Nov 03 01:50:49.652188 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.652143 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9152cf36-4997-470e-8902-802a96ebc2c7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gwlr2\" (UID: \"9152cf36-4997-470e-8902-802a96ebc2c7\") " pod="openshift-multus/multus-additional-cni-plugins-gwlr2" Nov 03 01:50:49.652188 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.652166 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/df76407a-ce41-4a4a-b51e-8420b817f97c-lib-modules\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:50:49.652287 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.652190 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-host-var-lib-kubelet\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:50:49.652287 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.652252 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ccfs\" (UniqueName: \"kubernetes.io/projected/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-kube-api-access-7ccfs\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:50:49.652349 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.652300 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/85148983-aa3f-45c2-a062-da2748dd9011-iptables-alerter-script\") pod \"iptables-alerter-vmjc5\" (UID: \"85148983-aa3f-45c2-a062-da2748dd9011\") " pod="openshift-network-operator/iptables-alerter-vmjc5" Nov 03 01:50:49.652349 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.652322 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9152cf36-4997-470e-8902-802a96ebc2c7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gwlr2\" (UID: \"9152cf36-4997-470e-8902-802a96ebc2c7\") " pod="openshift-multus/multus-additional-cni-plugins-gwlr2" Nov 03 01:50:49.652447 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.652389 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sr2jw\" (UniqueName: \"kubernetes.io/projected/6ae1437e-8885-42a5-b092-3862cd404c60-kube-api-access-sr2jw\") pod \"network-check-target-f5qvs\" (UID: \"6ae1437e-8885-42a5-b092-3862cd404c60\") " pod="openshift-network-diagnostics/network-check-target-f5qvs" Nov 03 01:50:49.652447 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.652419 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0801f13c-d14a-4b9e-b45a-a51af48d17c3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xdpjb\" (UID: \"0801f13c-d14a-4b9e-b45a-a51af48d17c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdpjb" Nov 03 01:50:49.652520 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.652466 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/df76407a-ce41-4a4a-b51e-8420b817f97c-etc-modprobe-d\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:50:49.652520 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.652495 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/df76407a-ce41-4a4a-b51e-8420b817f97c-etc-sysctl-d\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:50:49.652624 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.652526 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-host-kubelet\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:50:49.652624 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.652570 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-run-systemd\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:50:49.652624 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.652615 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-var-lib-openvswitch\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:50:49.652732 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.652643 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-host-var-lib-cni-bin\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:50:49.652732 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.652677 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-run-ovn\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:50:49.652732 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.652712 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-host-run-ovn-kubernetes\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:50:49.652732 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.652729 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-host-cni-bin\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:50:49.652850 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.652754 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/592adf6b-1edd-4f5e-88a2-6ed92895f22a-host\") pod \"node-ca-gjs9f\" (UID: \"592adf6b-1edd-4f5e-88a2-6ed92895f22a\") " pod="openshift-image-registry/node-ca-gjs9f" Nov 03 01:50:49.652850 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.652778 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qmrm\" (UniqueName: \"kubernetes.io/projected/85148983-aa3f-45c2-a062-da2748dd9011-kube-api-access-2qmrm\") pod \"iptables-alerter-vmjc5\" (UID: \"85148983-aa3f-45c2-a062-da2748dd9011\") " pod="openshift-network-operator/iptables-alerter-vmjc5" Nov 03 01:50:49.652850 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.652804 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0801f13c-d14a-4b9e-b45a-a51af48d17c3-socket-dir\") pod \"aws-ebs-csi-driver-node-xdpjb\" (UID: \"0801f13c-d14a-4b9e-b45a-a51af48d17c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdpjb" Nov 03 01:50:49.652850 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.652819 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/df76407a-ce41-4a4a-b51e-8420b817f97c-etc-sysconfig\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:50:49.652850 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.652845 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/df76407a-ce41-4a4a-b51e-8420b817f97c-etc-tuned\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:50:49.652993 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.652871 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:50:49.652993 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.652893 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/56117bce-d686-4c24-98c8-c0ff122778ac-ovnkube-script-lib\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:50:49.652993 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.652921 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffwtk\" (UniqueName: \"kubernetes.io/projected/df76407a-ce41-4a4a-b51e-8420b817f97c-kube-api-access-ffwtk\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:50:49.652993 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.652948 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-multus-cni-dir\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:50:49.652993 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.652965 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-host-run-netns\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:50:49.652993 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.652986 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-run-openvswitch\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:50:49.653176 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.653032 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgvhv\" (UniqueName: \"kubernetes.io/projected/7c3f175d-932c-42be-9660-e4fd3dceaf49-kube-api-access-pgvhv\") pod \"node-resolver-7zdvv\" (UID: \"7c3f175d-932c-42be-9660-e4fd3dceaf49\") " pod="openshift-dns/node-resolver-7zdvv" Nov 03 01:50:49.653176 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.653059 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9152cf36-4997-470e-8902-802a96ebc2c7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gwlr2\" (UID: \"9152cf36-4997-470e-8902-802a96ebc2c7\") " pod="openshift-multus/multus-additional-cni-plugins-gwlr2" Nov 03 01:50:49.653176 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.653081 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-cni-binary-copy\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:50:49.653176 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.653106 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-host-run-k8s-cni-cncf-io\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:50:49.653176 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.653124 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-host-cni-netd\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:50:49.653176 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.653151 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfgdw\" (UniqueName: \"kubernetes.io/projected/9152cf36-4997-470e-8902-802a96ebc2c7-kube-api-access-bfgdw\") pod \"multus-additional-cni-plugins-gwlr2\" (UID: \"9152cf36-4997-470e-8902-802a96ebc2c7\") " pod="openshift-multus/multus-additional-cni-plugins-gwlr2" Nov 03 01:50:49.653176 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.653173 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/85148983-aa3f-45c2-a062-da2748dd9011-host-slash\") pod \"iptables-alerter-vmjc5\" (UID: \"85148983-aa3f-45c2-a062-da2748dd9011\") " pod="openshift-network-operator/iptables-alerter-vmjc5" Nov 03 01:50:49.653375 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.653192 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7d2438ed-bfc2-4639-b87d-84689a82123e-agent-certs\") pod \"konnectivity-agent-f524q\" (UID: \"7d2438ed-bfc2-4639-b87d-84689a82123e\") " pod="kube-system/konnectivity-agent-f524q" Nov 03 01:50:49.653375 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.653236 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0801f13c-d14a-4b9e-b45a-a51af48d17c3-etc-selinux\") pod \"aws-ebs-csi-driver-node-xdpjb\" (UID: \"0801f13c-d14a-4b9e-b45a-a51af48d17c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdpjb" Nov 03 01:50:49.653375 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.653270 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-cnibin\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:50:49.653375 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.653304 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-host-var-lib-cni-multus\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:50:49.653375 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.653329 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/56117bce-d686-4c24-98c8-c0ff122778ac-ovnkube-config\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:50:49.653375 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.653351 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/56117bce-d686-4c24-98c8-c0ff122778ac-ovn-node-metrics-cert\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:50:49.653564 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.653382 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c3f175d-932c-42be-9660-e4fd3dceaf49-tmp-dir\") pod \"node-resolver-7zdvv\" (UID: \"7c3f175d-932c-42be-9660-e4fd3dceaf49\") " pod="openshift-dns/node-resolver-7zdvv" Nov 03 01:50:49.653564 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.653408 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/592adf6b-1edd-4f5e-88a2-6ed92895f22a-serviceca\") pod \"node-ca-gjs9f\" (UID: \"592adf6b-1edd-4f5e-88a2-6ed92895f22a\") " pod="openshift-image-registry/node-ca-gjs9f" Nov 03 01:50:49.653564 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.653427 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9152cf36-4997-470e-8902-802a96ebc2c7-system-cni-dir\") pod \"multus-additional-cni-plugins-gwlr2\" (UID: \"9152cf36-4997-470e-8902-802a96ebc2c7\") " pod="openshift-multus/multus-additional-cni-plugins-gwlr2" Nov 03 01:50:49.653564 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.653450 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-etc-kubernetes\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:50:49.653564 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.653471 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-log-socket\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:50:49.653564 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.653487 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/df76407a-ce41-4a4a-b51e-8420b817f97c-etc-systemd\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:50:49.653564 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.653514 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2l49\" (UniqueName: \"kubernetes.io/projected/dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b-kube-api-access-l2l49\") pod \"network-metrics-daemon-vzfhc\" (UID: \"dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b\") " pod="openshift-multus/network-metrics-daemon-vzfhc" Nov 03 01:50:49.653564 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.653545 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0801f13c-d14a-4b9e-b45a-a51af48d17c3-sys-fs\") pod \"aws-ebs-csi-driver-node-xdpjb\" (UID: \"0801f13c-d14a-4b9e-b45a-a51af48d17c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdpjb" Nov 03 01:50:49.653564 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.653563 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/df76407a-ce41-4a4a-b51e-8420b817f97c-sys\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:50:49.653829 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.653588 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/df76407a-ce41-4a4a-b51e-8420b817f97c-host\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:50:49.653829 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.653611 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-host-run-netns\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:50:49.653829 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.653641 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-host-run-multus-certs\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:50:49.653829 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.653664 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9152cf36-4997-470e-8902-802a96ebc2c7-cnibin\") pod \"multus-additional-cni-plugins-gwlr2\" (UID: \"9152cf36-4997-470e-8902-802a96ebc2c7\") " pod="openshift-multus/multus-additional-cni-plugins-gwlr2" Nov 03 01:50:49.653829 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.653688 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/df76407a-ce41-4a4a-b51e-8420b817f97c-var-lib-kubelet\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:50:49.653829 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.653748 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-multus-conf-dir\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:50:49.653829 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.653774 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-node-log\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:50:49.653829 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.653814 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kgsp\" (UniqueName: \"kubernetes.io/projected/56117bce-d686-4c24-98c8-c0ff122778ac-kube-api-access-5kgsp\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:50:49.654060 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.653847 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw55q\" (UniqueName: \"kubernetes.io/projected/592adf6b-1edd-4f5e-88a2-6ed92895f22a-kube-api-access-bw55q\") pod \"node-ca-gjs9f\" (UID: \"592adf6b-1edd-4f5e-88a2-6ed92895f22a\") " pod="openshift-image-registry/node-ca-gjs9f" Nov 03 01:50:49.654060 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.653866 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-systemd-units\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:50:49.654060 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.653882 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/56117bce-d686-4c24-98c8-c0ff122778ac-env-overrides\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:50:49.654060 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.653909 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7c3f175d-932c-42be-9660-e4fd3dceaf49-hosts-file\") pod \"node-resolver-7zdvv\" (UID: \"7c3f175d-932c-42be-9660-e4fd3dceaf49\") " pod="openshift-dns/node-resolver-7zdvv" Nov 03 01:50:49.654060 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.653939 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7d2438ed-bfc2-4639-b87d-84689a82123e-konnectivity-ca\") pod \"konnectivity-agent-f524q\" (UID: \"7d2438ed-bfc2-4639-b87d-84689a82123e\") " pod="kube-system/konnectivity-agent-f524q" Nov 03 01:50:49.654060 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.653963 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b-metrics-certs\") pod \"network-metrics-daemon-vzfhc\" (UID: \"dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b\") " pod="openshift-multus/network-metrics-daemon-vzfhc" Nov 03 01:50:49.654060 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.653990 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-host-slash\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:50:49.654060 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.654011 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqp5v\" (UniqueName: \"kubernetes.io/projected/0801f13c-d14a-4b9e-b45a-a51af48d17c3-kube-api-access-qqp5v\") pod \"aws-ebs-csi-driver-node-xdpjb\" (UID: \"0801f13c-d14a-4b9e-b45a-a51af48d17c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdpjb" Nov 03 01:50:49.654060 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.654035 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df76407a-ce41-4a4a-b51e-8420b817f97c-tmp\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:50:49.654060 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.654056 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-hostroot\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:50:49.654331 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.654077 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-multus-daemon-config\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:50:49.654331 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.654094 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0801f13c-d14a-4b9e-b45a-a51af48d17c3-registration-dir\") pod \"aws-ebs-csi-driver-node-xdpjb\" (UID: \"0801f13c-d14a-4b9e-b45a-a51af48d17c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdpjb" Nov 03 01:50:49.654331 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.654123 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0801f13c-d14a-4b9e-b45a-a51af48d17c3-device-dir\") pod \"aws-ebs-csi-driver-node-xdpjb\" (UID: \"0801f13c-d14a-4b9e-b45a-a51af48d17c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdpjb" Nov 03 01:50:49.654331 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.654139 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/df76407a-ce41-4a4a-b51e-8420b817f97c-etc-kubernetes\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:50:49.654331 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.654187 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/df76407a-ce41-4a4a-b51e-8420b817f97c-etc-sysctl-conf\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:50:49.680871 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.680842 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 01:50:49.680871 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.680860 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 01:50:49.681030 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.680860 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pc8bh\"" Nov 03 01:50:49.681030 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.680851 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 01:50:49.681723 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.681678 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 01:50:49.681723 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.681704 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jwmm7\"" Nov 03 01:50:49.681860 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.681757 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 01:50:49.684049 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.684026 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 01:50:49.684049 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.684024 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 01:50:49.684254 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.684133 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-zq5dk\"" Nov 03 01:50:49.684254 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.684169 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 01:50:49.684254 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.684221 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 01:50:49.686089 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.686070 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 01:50:49.686296 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.686281 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-f7c55\"" Nov 03 01:50:49.686372 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.686353 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 01:50:49.732654 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.732632 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 01:50:49.732654 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.732634 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 01:50:49.733025 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.733007 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 01:50:49.733093 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.733051 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 01:50:49.733156 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.733103 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 01:50:49.733156 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.733116 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bqcwb\"" Nov 03 01:50:49.733260 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.733208 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 01:50:49.733260 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.733230 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7cgvv\"" Nov 03 01:50:49.733260 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.733231 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 01:50:49.733458 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.733442 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 01:50:49.733696 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.733684 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 01:50:49.735716 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.735703 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5qdp4\"" Nov 03 01:50:49.735782 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.735738 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 01:50:49.735892 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.735878 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 01:50:49.735986 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.735974 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-2mmqt\"" Nov 03 01:50:49.736070 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.736057 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 01:50:49.736193 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.736178 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 01:50:49.736259 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.736190 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 01:50:49.736259 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.736204 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 01:50:49.736259 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.736239 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 01:50:49.736259 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.736245 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hgghc\"" Nov 03 01:50:49.736722 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.736699 2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 01:50:49.736764 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.736745 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5hs9j" Nov 03 01:50:49.744621 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.744491 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 01:50:49.755233 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.755204 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqp5v\" (UniqueName: \"kubernetes.io/projected/0801f13c-d14a-4b9e-b45a-a51af48d17c3-kube-api-access-qqp5v\") pod \"aws-ebs-csi-driver-node-xdpjb\" (UID: \"0801f13c-d14a-4b9e-b45a-a51af48d17c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdpjb" Nov 03 01:50:49.755337 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.755246 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df76407a-ce41-4a4a-b51e-8420b817f97c-tmp\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:50:49.755337 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.755273 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-hostroot\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:50:49.755459 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.755336 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-hostroot\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:50:49.755459 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.755440 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-multus-daemon-config\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:50:49.755588 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.755466 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0801f13c-d14a-4b9e-b45a-a51af48d17c3-registration-dir\") pod \"aws-ebs-csi-driver-node-xdpjb\" (UID: \"0801f13c-d14a-4b9e-b45a-a51af48d17c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdpjb" Nov 03 01:50:49.755588 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.755491 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0801f13c-d14a-4b9e-b45a-a51af48d17c3-device-dir\") pod \"aws-ebs-csi-driver-node-xdpjb\" (UID: \"0801f13c-d14a-4b9e-b45a-a51af48d17c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdpjb" Nov 03 01:50:49.755588 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.755517 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/df76407a-ce41-4a4a-b51e-8420b817f97c-etc-kubernetes\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:50:49.755588 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.755542 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/df76407a-ce41-4a4a-b51e-8420b817f97c-etc-sysctl-conf\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:50:49.755588 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.755560 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0801f13c-d14a-4b9e-b45a-a51af48d17c3-device-dir\") pod \"aws-ebs-csi-driver-node-xdpjb\" (UID: \"0801f13c-d14a-4b9e-b45a-a51af48d17c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdpjb" Nov 03 01:50:49.755588 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.755570 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/df76407a-ce41-4a4a-b51e-8420b817f97c-run\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:50:49.755588 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.755570 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0801f13c-d14a-4b9e-b45a-a51af48d17c3-registration-dir\") pod \"aws-ebs-csi-driver-node-xdpjb\" (UID: \"0801f13c-d14a-4b9e-b45a-a51af48d17c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdpjb" Nov 03 01:50:49.755956 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.755610 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-system-cni-dir\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:50:49.755956 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.755653 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-multus-socket-dir-parent\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:50:49.755956 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.755677 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-etc-openvswitch\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:50:49.755956 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.755701 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/df76407a-ce41-4a4a-b51e-8420b817f97c-etc-kubernetes\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:50:49.755956 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.755706 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9152cf36-4997-470e-8902-802a96ebc2c7-cni-binary-copy\") pod \"multus-additional-cni-plugins-gwlr2\" (UID: \"9152cf36-4997-470e-8902-802a96ebc2c7\") " pod="openshift-multus/multus-additional-cni-plugins-gwlr2" Nov 03 01:50:49.755956 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.755615 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/df76407a-ce41-4a4a-b51e-8420b817f97c-run\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:50:49.755956 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.755734 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/df76407a-ce41-4a4a-b51e-8420b817f97c-etc-sysctl-conf\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:50:49.755956 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.755711 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-system-cni-dir\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:50:49.755956 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.755749 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9152cf36-4997-470e-8902-802a96ebc2c7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gwlr2\" (UID: \"9152cf36-4997-470e-8902-802a96ebc2c7\") " pod="openshift-multus/multus-additional-cni-plugins-gwlr2" Nov 03 01:50:49.755956 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.755793 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/df76407a-ce41-4a4a-b51e-8420b817f97c-lib-modules\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:50:49.755956 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.755797 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-multus-socket-dir-parent\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:50:49.755956 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.755797 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-etc-openvswitch\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:50:49.755956 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.755881 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/df76407a-ce41-4a4a-b51e-8420b817f97c-lib-modules\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:50:49.755956 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.755911 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-host-var-lib-kubelet\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:50:49.755956 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.755935 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ccfs\" (UniqueName: \"kubernetes.io/projected/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-kube-api-access-7ccfs\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:50:49.755956 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.755955 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/85148983-aa3f-45c2-a062-da2748dd9011-iptables-alerter-script\") pod \"iptables-alerter-vmjc5\" (UID: \"85148983-aa3f-45c2-a062-da2748dd9011\") " pod="openshift-network-operator/iptables-alerter-vmjc5" Nov 03 01:50:49.756613 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.755974 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9152cf36-4997-470e-8902-802a96ebc2c7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gwlr2\" (UID: \"9152cf36-4997-470e-8902-802a96ebc2c7\") " pod="openshift-multus/multus-additional-cni-plugins-gwlr2" Nov 03 01:50:49.756613 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.756005 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr2jw\" (UniqueName: \"kubernetes.io/projected/6ae1437e-8885-42a5-b092-3862cd404c60-kube-api-access-sr2jw\") pod \"network-check-target-f5qvs\" (UID: \"6ae1437e-8885-42a5-b092-3862cd404c60\") " pod="openshift-network-diagnostics/network-check-target-f5qvs" Nov 03 01:50:49.756613 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.756020 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0801f13c-d14a-4b9e-b45a-a51af48d17c3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xdpjb\" (UID: \"0801f13c-d14a-4b9e-b45a-a51af48d17c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdpjb" Nov 03 01:50:49.756613 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.756034 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/df76407a-ce41-4a4a-b51e-8420b817f97c-etc-modprobe-d\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:50:49.756613 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.756049 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/df76407a-ce41-4a4a-b51e-8420b817f97c-etc-sysctl-d\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:50:49.756613 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.756063 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-host-kubelet\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:50:49.756613 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.756079 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-run-systemd\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:50:49.756613 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.756099 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-var-lib-openvswitch\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:50:49.756613 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.756127 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-host-var-lib-cni-bin\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:50:49.756613 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.756146 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-run-ovn\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:50:49.756613 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.756169 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-host-run-ovn-kubernetes\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:50:49.756613 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.756188 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-host-cni-bin\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:50:49.756613 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.756203 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/592adf6b-1edd-4f5e-88a2-6ed92895f22a-host\") pod \"node-ca-gjs9f\" (UID: \"592adf6b-1edd-4f5e-88a2-6ed92895f22a\") " pod="openshift-image-registry/node-ca-gjs9f" Nov 03 01:50:49.756613 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.756218 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qmrm\" (UniqueName: \"kubernetes.io/projected/85148983-aa3f-45c2-a062-da2748dd9011-kube-api-access-2qmrm\") pod \"iptables-alerter-vmjc5\" (UID: \"85148983-aa3f-45c2-a062-da2748dd9011\") " pod="openshift-network-operator/iptables-alerter-vmjc5" Nov 03 01:50:49.756613 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.756233 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0801f13c-d14a-4b9e-b45a-a51af48d17c3-socket-dir\") pod \"aws-ebs-csi-driver-node-xdpjb\" (UID: \"0801f13c-d14a-4b9e-b45a-a51af48d17c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdpjb" Nov 03 01:50:49.756613 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.756246 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/df76407a-ce41-4a4a-b51e-8420b817f97c-etc-sysconfig\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:50:49.756613 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.756262 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/df76407a-ce41-4a4a-b51e-8420b817f97c-etc-tuned\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:50:49.757318 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.756279 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:50:49.757318 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.756302 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/56117bce-d686-4c24-98c8-c0ff122778ac-ovnkube-script-lib\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:50:49.757318 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.756317 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffwtk\" (UniqueName: \"kubernetes.io/projected/df76407a-ce41-4a4a-b51e-8420b817f97c-kube-api-access-ffwtk\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:50:49.757318 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.756342 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-multus-cni-dir\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:50:49.757318 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.756358 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-host-run-netns\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:50:49.757318 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.756374 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-run-openvswitch\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:50:49.757318 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.756390 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgvhv\" (UniqueName: \"kubernetes.io/projected/7c3f175d-932c-42be-9660-e4fd3dceaf49-kube-api-access-pgvhv\") pod \"node-resolver-7zdvv\" (UID: \"7c3f175d-932c-42be-9660-e4fd3dceaf49\") " pod="openshift-dns/node-resolver-7zdvv" Nov 03 01:50:49.757318 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.756439 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9152cf36-4997-470e-8902-802a96ebc2c7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gwlr2\" (UID: \"9152cf36-4997-470e-8902-802a96ebc2c7\") " pod="openshift-multus/multus-additional-cni-plugins-gwlr2" Nov 03 01:50:49.757318 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.756468 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-cni-binary-copy\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:50:49.757318 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.756495 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-host-run-k8s-cni-cncf-io\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:50:49.757318 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.756533 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-host-cni-netd\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:50:49.757318 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.756554 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfgdw\" (UniqueName: \"kubernetes.io/projected/9152cf36-4997-470e-8902-802a96ebc2c7-kube-api-access-bfgdw\") pod \"multus-additional-cni-plugins-gwlr2\" (UID: \"9152cf36-4997-470e-8902-802a96ebc2c7\") " pod="openshift-multus/multus-additional-cni-plugins-gwlr2" Nov 03 01:50:49.757318 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.756590 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/85148983-aa3f-45c2-a062-da2748dd9011-host-slash\") pod \"iptables-alerter-vmjc5\" (UID: \"85148983-aa3f-45c2-a062-da2748dd9011\") " pod="openshift-network-operator/iptables-alerter-vmjc5" Nov 03 01:50:49.757318 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.756606 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/592adf6b-1edd-4f5e-88a2-6ed92895f22a-host\") pod \"node-ca-gjs9f\" (UID: \"592adf6b-1edd-4f5e-88a2-6ed92895f22a\") " pod="openshift-image-registry/node-ca-gjs9f" Nov 03 01:50:49.757318 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.756619 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/28f00eb6-78c2-4d93-bde5-c1ea12601dbd-original-pull-secret\") pod \"global-pull-secret-syncer-kkngd\" (UID: \"28f00eb6-78c2-4d93-bde5-c1ea12601dbd\") " pod="kube-system/global-pull-secret-syncer-kkngd" Nov 03 01:50:49.757318 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.755936 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-host-var-lib-kubelet\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:50:49.757318 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.756637 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7d2438ed-bfc2-4639-b87d-84689a82123e-agent-certs\") pod \"konnectivity-agent-f524q\" (UID: \"7d2438ed-bfc2-4639-b87d-84689a82123e\") " pod="kube-system/konnectivity-agent-f524q" Nov 03 01:50:49.757974 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.756659 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0801f13c-d14a-4b9e-b45a-a51af48d17c3-etc-selinux\") pod \"aws-ebs-csi-driver-node-xdpjb\" (UID: \"0801f13c-d14a-4b9e-b45a-a51af48d17c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdpjb" Nov 03 01:50:49.757974 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.756675 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-cnibin\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:50:49.757974 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.756682 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-multus-daemon-config\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:50:49.757974 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.756721 2579 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 01:50:49.757974 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.756750 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-host-var-lib-cni-multus\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:50:49.757974 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.756689 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-host-var-lib-cni-multus\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:50:49.757974 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.756794 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/56117bce-d686-4c24-98c8-c0ff122778ac-ovnkube-config\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:50:49.757974 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.756824 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/56117bce-d686-4c24-98c8-c0ff122778ac-ovn-node-metrics-cert\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:50:49.757974 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.756851 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c3f175d-932c-42be-9660-e4fd3dceaf49-tmp-dir\") pod \"node-resolver-7zdvv\" (UID: \"7c3f175d-932c-42be-9660-e4fd3dceaf49\") " pod="openshift-dns/node-resolver-7zdvv" Nov 03 01:50:49.757974 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.756879 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/592adf6b-1edd-4f5e-88a2-6ed92895f22a-serviceca\") pod \"node-ca-gjs9f\" (UID: \"592adf6b-1edd-4f5e-88a2-6ed92895f22a\") " pod="openshift-image-registry/node-ca-gjs9f" Nov 03 01:50:49.757974 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.756907 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9152cf36-4997-470e-8902-802a96ebc2c7-system-cni-dir\") pod \"multus-additional-cni-plugins-gwlr2\" (UID: \"9152cf36-4997-470e-8902-802a96ebc2c7\") " pod="openshift-multus/multus-additional-cni-plugins-gwlr2" Nov 03 01:50:49.757974 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.756934 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-etc-kubernetes\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:50:49.757974 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.756958 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-log-socket\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:50:49.757974 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.756980 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0801f13c-d14a-4b9e-b45a-a51af48d17c3-socket-dir\") pod \"aws-ebs-csi-driver-node-xdpjb\" (UID: \"0801f13c-d14a-4b9e-b45a-a51af48d17c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdpjb" Nov 03 01:50:49.757974 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.756990 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/28f00eb6-78c2-4d93-bde5-c1ea12601dbd-kubelet-config\") pod \"global-pull-secret-syncer-kkngd\" (UID: \"28f00eb6-78c2-4d93-bde5-c1ea12601dbd\") " pod="kube-system/global-pull-secret-syncer-kkngd" Nov 03 01:50:49.757974 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.757023 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/df76407a-ce41-4a4a-b51e-8420b817f97c-etc-systemd\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:50:49.757974 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.757040 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/df76407a-ce41-4a4a-b51e-8420b817f97c-etc-sysconfig\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:50:49.757974 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.757054 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2l49\" (UniqueName: \"kubernetes.io/projected/dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b-kube-api-access-l2l49\") pod \"network-metrics-daemon-vzfhc\" (UID: \"dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b\") " pod="openshift-multus/network-metrics-daemon-vzfhc" Nov 03 01:50:49.758765 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.757082 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0801f13c-d14a-4b9e-b45a-a51af48d17c3-sys-fs\") pod \"aws-ebs-csi-driver-node-xdpjb\" (UID: \"0801f13c-d14a-4b9e-b45a-a51af48d17c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdpjb" Nov 03 01:50:49.758765 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.757111 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/df76407a-ce41-4a4a-b51e-8420b817f97c-sys\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:50:49.758765 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.757136 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/df76407a-ce41-4a4a-b51e-8420b817f97c-host\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:50:49.758765 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.757141 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/85148983-aa3f-45c2-a062-da2748dd9011-iptables-alerter-script\") pod \"iptables-alerter-vmjc5\" (UID: \"85148983-aa3f-45c2-a062-da2748dd9011\") " pod="openshift-network-operator/iptables-alerter-vmjc5" Nov 03 01:50:49.758765 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.757160 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-host-run-netns\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:50:49.758765 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.757189 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-host-run-multus-certs\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:50:49.758765 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.757213 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9152cf36-4997-470e-8902-802a96ebc2c7-cnibin\") pod \"multus-additional-cni-plugins-gwlr2\" (UID: \"9152cf36-4997-470e-8902-802a96ebc2c7\") " pod="openshift-multus/multus-additional-cni-plugins-gwlr2" Nov 03 01:50:49.758765 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.757240 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/df76407a-ce41-4a4a-b51e-8420b817f97c-var-lib-kubelet\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:50:49.758765 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.757258 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-multus-conf-dir\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:50:49.758765 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.757273 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-node-log\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:50:49.758765 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.757293 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kgsp\" (UniqueName: \"kubernetes.io/projected/56117bce-d686-4c24-98c8-c0ff122778ac-kube-api-access-5kgsp\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:50:49.758765 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.757309 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw55q\" (UniqueName: \"kubernetes.io/projected/592adf6b-1edd-4f5e-88a2-6ed92895f22a-kube-api-access-bw55q\") pod \"node-ca-gjs9f\" (UID: \"592adf6b-1edd-4f5e-88a2-6ed92895f22a\") " pod="openshift-image-registry/node-ca-gjs9f" Nov 03 01:50:49.758765 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.757326 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/28f00eb6-78c2-4d93-bde5-c1ea12601dbd-dbus\") pod \"global-pull-secret-syncer-kkngd\" (UID: \"28f00eb6-78c2-4d93-bde5-c1ea12601dbd\") " pod="kube-system/global-pull-secret-syncer-kkngd" Nov 03 01:50:49.758765 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.757342 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-systemd-units\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:50:49.758765 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.757358 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/56117bce-d686-4c24-98c8-c0ff122778ac-env-overrides\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:50:49.758765 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.757375 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7c3f175d-932c-42be-9660-e4fd3dceaf49-hosts-file\") pod \"node-resolver-7zdvv\" (UID: \"7c3f175d-932c-42be-9660-e4fd3dceaf49\") " pod="openshift-dns/node-resolver-7zdvv" Nov 03 01:50:49.758765 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.757396 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7d2438ed-bfc2-4639-b87d-84689a82123e-konnectivity-ca\") pod \"konnectivity-agent-f524q\" (UID: \"7d2438ed-bfc2-4639-b87d-84689a82123e\") " pod="kube-system/konnectivity-agent-f524q" Nov 03 01:50:49.759631 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.757431 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b-metrics-certs\") pod \"network-metrics-daemon-vzfhc\" (UID: \"dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b\") " pod="openshift-multus/network-metrics-daemon-vzfhc" Nov 03 01:50:49.759631 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.757491 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-host-slash\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:50:49.759631 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.757581 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-host-slash\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:50:49.759631 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.758037 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/56117bce-d686-4c24-98c8-c0ff122778ac-ovnkube-config\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:50:49.759631 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.759137 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9152cf36-4997-470e-8902-802a96ebc2c7-cni-binary-copy\") pod \"multus-additional-cni-plugins-gwlr2\" (UID: \"9152cf36-4997-470e-8902-802a96ebc2c7\") " pod="openshift-multus/multus-additional-cni-plugins-gwlr2" Nov 03 01:50:49.759631 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.759168 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9152cf36-4997-470e-8902-802a96ebc2c7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gwlr2\" (UID: \"9152cf36-4997-470e-8902-802a96ebc2c7\") " pod="openshift-multus/multus-additional-cni-plugins-gwlr2" Nov 03 01:50:49.759631 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.759453 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0801f13c-d14a-4b9e-b45a-a51af48d17c3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xdpjb\" (UID: \"0801f13c-d14a-4b9e-b45a-a51af48d17c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdpjb" Nov 03 01:50:49.759631 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.759546 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/df76407a-ce41-4a4a-b51e-8420b817f97c-etc-modprobe-d\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:50:49.759631 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.759623 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/df76407a-ce41-4a4a-b51e-8420b817f97c-etc-sysctl-d\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:50:49.760157 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.759663 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-host-kubelet\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:50:49.760157 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.759702 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-run-systemd\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:50:49.760157 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.759740 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-var-lib-openvswitch\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:50:49.760157 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.759794 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-host-var-lib-cni-bin\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:50:49.760157 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.759834 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-run-ovn\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:50:49.760157 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.759870 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-host-run-ovn-kubernetes\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:50:49.760157 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.759911 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-host-cni-bin\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:50:49.760157 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.759997 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-cni-binary-copy\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:50:49.760157 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.760055 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:50:49.760644 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.760625 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/56117bce-d686-4c24-98c8-c0ff122778ac-ovnkube-script-lib\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:50:49.760907 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.760747 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-run-openvswitch\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:50:49.761009 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.760965 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-multus-cni-dir\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:50:49.761083 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.761008 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-host-run-netns\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:50:49.761144 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.761083 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-host-run-multus-certs\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:50:49.761144 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.761134 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9152cf36-4997-470e-8902-802a96ebc2c7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gwlr2\" (UID: \"9152cf36-4997-470e-8902-802a96ebc2c7\") " pod="openshift-multus/multus-additional-cni-plugins-gwlr2" Nov 03 01:50:49.761263 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.761139 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9152cf36-4997-470e-8902-802a96ebc2c7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gwlr2\" (UID: \"9152cf36-4997-470e-8902-802a96ebc2c7\") " pod="openshift-multus/multus-additional-cni-plugins-gwlr2" Nov 03 01:50:49.761263 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.761229 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-multus-conf-dir\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:50:49.761386 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.761300 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-node-log\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:50:49.761386 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.761374 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/df76407a-ce41-4a4a-b51e-8420b817f97c-etc-tuned\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:50:49.761517 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.761475 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-host-run-k8s-cni-cncf-io\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:50:49.761517 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.761490 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df76407a-ce41-4a4a-b51e-8420b817f97c-tmp\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:50:49.761634 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.761550 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-host-cni-netd\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:50:49.761634 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.761553 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c3f175d-932c-42be-9660-e4fd3dceaf49-tmp-dir\") pod \"node-resolver-7zdvv\" (UID: \"7c3f175d-932c-42be-9660-e4fd3dceaf49\") " pod="openshift-dns/node-resolver-7zdvv" Nov 03 01:50:49.761634 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.761595 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/85148983-aa3f-45c2-a062-da2748dd9011-host-slash\") pod \"iptables-alerter-vmjc5\" (UID: \"85148983-aa3f-45c2-a062-da2748dd9011\") " pod="openshift-network-operator/iptables-alerter-vmjc5" Nov 03 01:50:49.761811 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.761645 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0801f13c-d14a-4b9e-b45a-a51af48d17c3-etc-selinux\") pod \"aws-ebs-csi-driver-node-xdpjb\" (UID: \"0801f13c-d14a-4b9e-b45a-a51af48d17c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdpjb" Nov 03 01:50:49.761811 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.761654 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/df76407a-ce41-4a4a-b51e-8420b817f97c-etc-systemd\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:50:49.761811 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.761699 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-cnibin\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:50:49.761811 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.761720 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9152cf36-4997-470e-8902-802a96ebc2c7-system-cni-dir\") pod \"multus-additional-cni-plugins-gwlr2\" (UID: \"9152cf36-4997-470e-8902-802a96ebc2c7\") " pod="openshift-multus/multus-additional-cni-plugins-gwlr2" Nov 03 01:50:49.761811 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.761731 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9152cf36-4997-470e-8902-802a96ebc2c7-cnibin\") pod \"multus-additional-cni-plugins-gwlr2\" (UID: \"9152cf36-4997-470e-8902-802a96ebc2c7\") " pod="openshift-multus/multus-additional-cni-plugins-gwlr2" Nov 03 01:50:49.761811 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.761747 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/df76407a-ce41-4a4a-b51e-8420b817f97c-var-lib-kubelet\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:50:49.761811 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.761764 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-etc-kubernetes\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:50:49.761811 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.761799 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-log-socket\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:50:49.762129 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.761876 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7c3f175d-932c-42be-9660-e4fd3dceaf49-hosts-file\") pod \"node-resolver-7zdvv\" (UID: \"7c3f175d-932c-42be-9660-e4fd3dceaf49\") " pod="openshift-dns/node-resolver-7zdvv" Nov 03 01:50:49.762129 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.761937 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/df76407a-ce41-4a4a-b51e-8420b817f97c-sys\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:50:49.762129 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.762101 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0801f13c-d14a-4b9e-b45a-a51af48d17c3-sys-fs\") pod \"aws-ebs-csi-driver-node-xdpjb\" (UID: \"0801f13c-d14a-4b9e-b45a-a51af48d17c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdpjb" Nov 03 01:50:49.762219 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.762138 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-systemd-units\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:50:49.762219 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:49.762212 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 01:50:49.762301 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:49.762289 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b-metrics-certs podName:dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b nodeName:}" failed. No retries permitted until 2026-11-03 01:50:50.262274876 +0000 UTC m=+2.265722319 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b-metrics-certs") pod "network-metrics-daemon-vzfhc" (UID: "dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 01:50:49.762344 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.762331 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/df76407a-ce41-4a4a-b51e-8420b817f97c-host\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:50:49.762378 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.762358 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-host-run-netns\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:50:49.762841 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.762812 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/56117bce-d686-4c24-98c8-c0ff122778ac-ovn-node-metrics-cert\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:50:49.763407 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.763370 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/56117bce-d686-4c24-98c8-c0ff122778ac-env-overrides\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:50:49.763551 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.763382 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/592adf6b-1edd-4f5e-88a2-6ed92895f22a-serviceca\") pod \"node-ca-gjs9f\" (UID: \"592adf6b-1edd-4f5e-88a2-6ed92895f22a\") " pod="openshift-image-registry/node-ca-gjs9f" Nov 03 01:50:49.763606 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.763556 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7d2438ed-bfc2-4639-b87d-84689a82123e-konnectivity-ca\") pod \"konnectivity-agent-f524q\" (UID: \"7d2438ed-bfc2-4639-b87d-84689a82123e\") " pod="kube-system/konnectivity-agent-f524q" Nov 03 01:50:49.763738 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.763722 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7d2438ed-bfc2-4639-b87d-84689a82123e-agent-certs\") pod \"konnectivity-agent-f524q\" (UID: \"7d2438ed-bfc2-4639-b87d-84689a82123e\") " pod="kube-system/konnectivity-agent-f524q" Nov 03 01:50:49.817811 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.817742 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqp5v\" (UniqueName: \"kubernetes.io/projected/0801f13c-d14a-4b9e-b45a-a51af48d17c3-kube-api-access-qqp5v\") pod \"aws-ebs-csi-driver-node-xdpjb\" (UID: \"0801f13c-d14a-4b9e-b45a-a51af48d17c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdpjb" Nov 03 01:50:49.817811 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.817756 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ccfs\" (UniqueName: \"kubernetes.io/projected/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-kube-api-access-7ccfs\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:50:49.817811 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.817760 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qmrm\" (UniqueName: \"kubernetes.io/projected/85148983-aa3f-45c2-a062-da2748dd9011-kube-api-access-2qmrm\") pod \"iptables-alerter-vmjc5\" (UID: \"85148983-aa3f-45c2-a062-da2748dd9011\") " pod="openshift-network-operator/iptables-alerter-vmjc5" Nov 03 01:50:49.818761 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:49.818745 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 01:50:49.818805 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:49.818766 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 01:50:49.818805 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:49.818776 2579 projected.go:196] Error preparing data for projected volume kube-api-access-sr2jw for pod openshift-network-diagnostics/network-check-target-f5qvs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 01:50:49.818868 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:49.818831 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ae1437e-8885-42a5-b092-3862cd404c60-kube-api-access-sr2jw podName:6ae1437e-8885-42a5-b092-3862cd404c60 nodeName:}" failed. No retries permitted until 2026-11-03 01:50:50.318815611 +0000 UTC m=+2.322263071 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-sr2jw" (UniqueName: "kubernetes.io/projected/6ae1437e-8885-42a5-b092-3862cd404c60-kube-api-access-sr2jw") pod "network-check-target-f5qvs" (UID: "6ae1437e-8885-42a5-b092-3862cd404c60") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 01:50:49.823257 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.823226 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgvhv\" (UniqueName: \"kubernetes.io/projected/7c3f175d-932c-42be-9660-e4fd3dceaf49-kube-api-access-pgvhv\") pod \"node-resolver-7zdvv\" (UID: \"7c3f175d-932c-42be-9660-e4fd3dceaf49\") " pod="openshift-dns/node-resolver-7zdvv" Nov 03 01:50:49.823716 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.823688 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffwtk\" (UniqueName: \"kubernetes.io/projected/df76407a-ce41-4a4a-b51e-8420b817f97c-kube-api-access-ffwtk\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:50:49.824049 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.824023 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw55q\" (UniqueName: \"kubernetes.io/projected/592adf6b-1edd-4f5e-88a2-6ed92895f22a-kube-api-access-bw55q\") pod \"node-ca-gjs9f\" (UID: \"592adf6b-1edd-4f5e-88a2-6ed92895f22a\") " pod="openshift-image-registry/node-ca-gjs9f" Nov 03 01:50:49.824123 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.824090 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kgsp\" (UniqueName: \"kubernetes.io/projected/56117bce-d686-4c24-98c8-c0ff122778ac-kube-api-access-5kgsp\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:50:49.824254 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.824232 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfgdw\" (UniqueName: \"kubernetes.io/projected/9152cf36-4997-470e-8902-802a96ebc2c7-kube-api-access-bfgdw\") pod \"multus-additional-cni-plugins-gwlr2\" (UID: \"9152cf36-4997-470e-8902-802a96ebc2c7\") " pod="openshift-multus/multus-additional-cni-plugins-gwlr2" Nov 03 01:50:49.824941 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.824920 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2l49\" (UniqueName: \"kubernetes.io/projected/dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b-kube-api-access-l2l49\") pod \"network-metrics-daemon-vzfhc\" (UID: \"dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b\") " pod="openshift-multus/network-metrics-daemon-vzfhc" Nov 03 01:50:49.858164 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.858124 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/28f00eb6-78c2-4d93-bde5-c1ea12601dbd-original-pull-secret\") pod \"global-pull-secret-syncer-kkngd\" (UID: \"28f00eb6-78c2-4d93-bde5-c1ea12601dbd\") " pod="kube-system/global-pull-secret-syncer-kkngd" Nov 03 01:50:49.858392 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:49.858291 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 01:50:49.858392 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.858348 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/28f00eb6-78c2-4d93-bde5-c1ea12601dbd-kubelet-config\") pod \"global-pull-secret-syncer-kkngd\" (UID: \"28f00eb6-78c2-4d93-bde5-c1ea12601dbd\") " pod="kube-system/global-pull-secret-syncer-kkngd" Nov 03 01:50:49.858392 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:49.858361 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28f00eb6-78c2-4d93-bde5-c1ea12601dbd-original-pull-secret podName:28f00eb6-78c2-4d93-bde5-c1ea12601dbd nodeName:}" failed. No retries permitted until 2026-11-03 01:50:50.35833892 +0000 UTC m=+2.361786381 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/28f00eb6-78c2-4d93-bde5-c1ea12601dbd-original-pull-secret") pod "global-pull-secret-syncer-kkngd" (UID: "28f00eb6-78c2-4d93-bde5-c1ea12601dbd") : object "kube-system"/"original-pull-secret" not registered Nov 03 01:50:49.858618 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.858427 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/28f00eb6-78c2-4d93-bde5-c1ea12601dbd-kubelet-config\") pod \"global-pull-secret-syncer-kkngd\" (UID: \"28f00eb6-78c2-4d93-bde5-c1ea12601dbd\") " pod="kube-system/global-pull-secret-syncer-kkngd" Nov 03 01:50:49.858618 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.858459 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/28f00eb6-78c2-4d93-bde5-c1ea12601dbd-dbus\") pod \"global-pull-secret-syncer-kkngd\" (UID: \"28f00eb6-78c2-4d93-bde5-c1ea12601dbd\") " pod="kube-system/global-pull-secret-syncer-kkngd" Nov 03 01:50:49.858618 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.858572 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/28f00eb6-78c2-4d93-bde5-c1ea12601dbd-dbus\") pod \"global-pull-secret-syncer-kkngd\" (UID: \"28f00eb6-78c2-4d93-bde5-c1ea12601dbd\") " pod="kube-system/global-pull-secret-syncer-kkngd" Nov 03 01:50:49.955543 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.955513 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdpjb" Nov 03 01:50:49.959756 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.959732 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:50:49.988180 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:49.988145 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-trr75" Nov 03 01:50:50.012559 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:50.012534 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vmjc5" Nov 03 01:50:50.043705 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:50.043675 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:50:50.046030 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:50.046002 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-f524q" Nov 03 01:50:50.080258 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:50.080193 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gwlr2" Nov 03 01:50:50.082484 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:50.082461 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7zdvv" Nov 03 01:50:50.083690 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:50.083668 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gjs9f" Nov 03 01:50:50.176680 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:50.176638 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b5ab86e21798d22090c574595774d10.slice/crio-988bc0719feedc4d376b3edfbe25039b9faf76ee6d89c0e2e9ba1ea8ad53553a WatchSource:0}: Error finding container 988bc0719feedc4d376b3edfbe25039b9faf76ee6d89c0e2e9ba1ea8ad53553a: Status 404 returned error can't find the container with id 988bc0719feedc4d376b3edfbe25039b9faf76ee6d89c0e2e9ba1ea8ad53553a Nov 03 01:50:50.177056 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:50.177029 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1cefe6117be002b0e139e82f22d4917.slice/crio-535f8f7d5b0302230f474cf92cfcadc6365f660f9f7339ab151a81e4890cea8e WatchSource:0}: Error finding container 535f8f7d5b0302230f474cf92cfcadc6365f660f9f7339ab151a81e4890cea8e: Status 404 returned error can't find the container with id 535f8f7d5b0302230f474cf92cfcadc6365f660f9f7339ab151a81e4890cea8e Nov 03 01:50:50.181885 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:50.181866 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 01:50:50.362754 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:50.362661 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/28f00eb6-78c2-4d93-bde5-c1ea12601dbd-original-pull-secret\") pod \"global-pull-secret-syncer-kkngd\" (UID: \"28f00eb6-78c2-4d93-bde5-c1ea12601dbd\") " pod="kube-system/global-pull-secret-syncer-kkngd" Nov 03 01:50:50.362754 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:50.362706 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b-metrics-certs\") pod \"network-metrics-daemon-vzfhc\" (UID: \"dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b\") " pod="openshift-multus/network-metrics-daemon-vzfhc" Nov 03 01:50:50.362754 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:50.362730 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr2jw\" (UniqueName: \"kubernetes.io/projected/6ae1437e-8885-42a5-b092-3862cd404c60-kube-api-access-sr2jw\") pod \"network-check-target-f5qvs\" (UID: \"6ae1437e-8885-42a5-b092-3862cd404c60\") " pod="openshift-network-diagnostics/network-check-target-f5qvs" Nov 03 01:50:50.363209 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:50.362819 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 01:50:50.363209 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:50.362874 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28f00eb6-78c2-4d93-bde5-c1ea12601dbd-original-pull-secret podName:28f00eb6-78c2-4d93-bde5-c1ea12601dbd nodeName:}" failed. No retries permitted until 2026-11-03 01:50:51.362858818 +0000 UTC m=+3.366306261 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/28f00eb6-78c2-4d93-bde5-c1ea12601dbd-original-pull-secret") pod "global-pull-secret-syncer-kkngd" (UID: "28f00eb6-78c2-4d93-bde5-c1ea12601dbd") : object "kube-system"/"original-pull-secret" not registered Nov 03 01:50:50.363209 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:50.362820 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 01:50:50.363209 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:50.362827 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 01:50:50.363209 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:50.362916 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 01:50:50.363209 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:50.362926 2579 projected.go:196] Error preparing data for projected volume kube-api-access-sr2jw for pod openshift-network-diagnostics/network-check-target-f5qvs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 01:50:50.363209 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:50.362907 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b-metrics-certs podName:dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b nodeName:}" failed. No retries permitted until 2026-11-03 01:50:51.362899092 +0000 UTC m=+3.366346535 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b-metrics-certs") pod "network-metrics-daemon-vzfhc" (UID: "dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 01:50:50.363209 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:50.362972 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ae1437e-8885-42a5-b092-3862cd404c60-kube-api-access-sr2jw podName:6ae1437e-8885-42a5-b092-3862cd404c60 nodeName:}" failed. No retries permitted until 2026-11-03 01:50:51.362960724 +0000 UTC m=+3.366408166 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-sr2jw" (UniqueName: "kubernetes.io/projected/6ae1437e-8885-42a5-b092-3862cd404c60-kube-api-access-sr2jw") pod "network-check-target-f5qvs" (UID: "6ae1437e-8885-42a5-b092-3862cd404c60") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 01:50:50.475328 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:50.475262 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-140.us-west-2.compute.internal" event={"ID":"d1cefe6117be002b0e139e82f22d4917","Type":"ContainerStarted","Data":"535f8f7d5b0302230f474cf92cfcadc6365f660f9f7339ab151a81e4890cea8e"} Nov 03 01:50:50.476352 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:50.476325 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-140.us-west-2.compute.internal" event={"ID":"3b5ab86e21798d22090c574595774d10","Type":"ContainerStarted","Data":"988bc0719feedc4d376b3edfbe25039b9faf76ee6d89c0e2e9ba1ea8ad53553a"} Nov 03 01:50:50.740103 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:50.739996 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 01:45:50 +0000 UTC" deadline="2028-08-14 23:20:53.099669396 +0000 UTC" Nov 03 01:50:50.740103 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:50.740046 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15621h30m2.359625955s" Nov 03 01:50:50.791805 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:50.791771 2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 01:50:50.895775 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:50.895744 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf76407a_ce41_4a4a_b51e_8420b817f97c.slice/crio-79d01368f40af632c77386836e9b55b47dfdf3598f3f9128db82d00f4d32cbab WatchSource:0}: Error finding container 79d01368f40af632c77386836e9b55b47dfdf3598f3f9128db82d00f4d32cbab: Status 404 returned error can't find the container with id 79d01368f40af632c77386836e9b55b47dfdf3598f3f9128db82d00f4d32cbab Nov 03 01:50:50.912938 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:50.912909 2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 01:50:50.920174 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:50.920149 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod987bbde2_6a6a_44fc_b5e4_b0197bcecd8b.slice/crio-69597725f815b11f8be350787f47840c1fcd6e2ffc45c95addf48313ec7cd46d WatchSource:0}: Error finding container 69597725f815b11f8be350787f47840c1fcd6e2ffc45c95addf48313ec7cd46d: Status 404 returned error can't find the container with id 69597725f815b11f8be350787f47840c1fcd6e2ffc45c95addf48313ec7cd46d Nov 03 01:50:50.932963 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:50.932934 2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 01:50:50.940614 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:50.940587 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0801f13c_d14a_4b9e_b45a_a51af48d17c3.slice/crio-da8b4e239c1214ccf41568209cfff8db65166c4b47549e0e45e287564408e64e WatchSource:0}: Error finding container da8b4e239c1214ccf41568209cfff8db65166c4b47549e0e45e287564408e64e: Status 404 returned error can't find the container with id da8b4e239c1214ccf41568209cfff8db65166c4b47549e0e45e287564408e64e Nov 03 01:50:50.968054 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:50.968025 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56117bce_d686_4c24_98c8_c0ff122778ac.slice/crio-19bbae04ef2fe7675bd877cca8a8cac40e2e02ce17e8440de5bf5c040ffd0e83 WatchSource:0}: Error finding container 19bbae04ef2fe7675bd877cca8a8cac40e2e02ce17e8440de5bf5c040ffd0e83: Status 404 returned error can't find the container with id 19bbae04ef2fe7675bd877cca8a8cac40e2e02ce17e8440de5bf5c040ffd0e83 Nov 03 01:50:51.017726 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:51.017684 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85148983_aa3f_45c2_a062_da2748dd9011.slice/crio-6e7abde09bd8790efbf067d9fa68b368e263d7f206db973770b8389ef07cbbf0 WatchSource:0}: Error finding container 6e7abde09bd8790efbf067d9fa68b368e263d7f206db973770b8389ef07cbbf0: Status 404 returned error can't find the container with id 6e7abde09bd8790efbf067d9fa68b368e263d7f206db973770b8389ef07cbbf0 Nov 03 01:50:51.018268 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:51.018240 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c3f175d_932c_42be_9660_e4fd3dceaf49.slice/crio-f666cadc2b74c94a27dc69fcae7d613fd3a4651e984de98a8076baf26c9abcbc WatchSource:0}: Error finding container f666cadc2b74c94a27dc69fcae7d613fd3a4651e984de98a8076baf26c9abcbc: Status 404 returned error can't find the container with id f666cadc2b74c94a27dc69fcae7d613fd3a4651e984de98a8076baf26c9abcbc Nov 03 01:50:51.040715 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:51.040684 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9152cf36_4997_470e_8902_802a96ebc2c7.slice/crio-f618f2b28711a3231ea6e344c22c588e935e8fa1b8fb39eda474a018c73fc5ba WatchSource:0}: Error finding container f618f2b28711a3231ea6e344c22c588e935e8fa1b8fb39eda474a018c73fc5ba: Status 404 returned error can't find the container with id f618f2b28711a3231ea6e344c22c588e935e8fa1b8fb39eda474a018c73fc5ba Nov 03 01:50:51.065976 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:51.065947 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d2438ed_bfc2_4639_b87d_84689a82123e.slice/crio-fb9d1cd812647e2f4378d1305d22e70c4bc844ac53832a289468b8e7e922b5ea WatchSource:0}: Error finding container fb9d1cd812647e2f4378d1305d22e70c4bc844ac53832a289468b8e7e922b5ea: Status 404 returned error can't find the container with id fb9d1cd812647e2f4378d1305d22e70c4bc844ac53832a289468b8e7e922b5ea Nov 03 01:50:51.118295 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:50:51.118258 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod592adf6b_1edd_4f5e_88a2_6ed92895f22a.slice/crio-6d44cceeee48b2e37bdb5bdef30b12a3578a06e24872cbbcb5fa45a898aab4c0 WatchSource:0}: Error finding container 6d44cceeee48b2e37bdb5bdef30b12a3578a06e24872cbbcb5fa45a898aab4c0: Status 404 returned error can't find the container with id 6d44cceeee48b2e37bdb5bdef30b12a3578a06e24872cbbcb5fa45a898aab4c0 Nov 03 01:50:51.370703 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:51.370615 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b-metrics-certs\") pod \"network-metrics-daemon-vzfhc\" (UID: \"dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b\") " pod="openshift-multus/network-metrics-daemon-vzfhc" Nov 03 01:50:51.370703 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:51.370667 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr2jw\" (UniqueName: \"kubernetes.io/projected/6ae1437e-8885-42a5-b092-3862cd404c60-kube-api-access-sr2jw\") pod \"network-check-target-f5qvs\" (UID: \"6ae1437e-8885-42a5-b092-3862cd404c60\") " pod="openshift-network-diagnostics/network-check-target-f5qvs" Nov 03 01:50:51.371499 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:51.370717 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/28f00eb6-78c2-4d93-bde5-c1ea12601dbd-original-pull-secret\") pod \"global-pull-secret-syncer-kkngd\" (UID: \"28f00eb6-78c2-4d93-bde5-c1ea12601dbd\") " pod="kube-system/global-pull-secret-syncer-kkngd" Nov 03 01:50:51.371499 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:51.370816 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 01:50:51.371499 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:51.370869 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 01:50:51.371499 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:51.370888 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 01:50:51.371499 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:51.370890 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b-metrics-certs podName:dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b nodeName:}" failed. No retries permitted until 2026-11-03 01:50:53.370870732 +0000 UTC m=+5.374318191 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b-metrics-certs") pod "network-metrics-daemon-vzfhc" (UID: "dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 01:50:51.371499 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:51.370901 2579 projected.go:196] Error preparing data for projected volume kube-api-access-sr2jw for pod openshift-network-diagnostics/network-check-target-f5qvs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 01:50:51.371499 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:51.370827 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 01:50:51.371499 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:51.370951 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ae1437e-8885-42a5-b092-3862cd404c60-kube-api-access-sr2jw podName:6ae1437e-8885-42a5-b092-3862cd404c60 nodeName:}" failed. No retries permitted until 2026-11-03 01:50:53.37093348 +0000 UTC m=+5.374380945 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-sr2jw" (UniqueName: "kubernetes.io/projected/6ae1437e-8885-42a5-b092-3862cd404c60-kube-api-access-sr2jw") pod "network-check-target-f5qvs" (UID: "6ae1437e-8885-42a5-b092-3862cd404c60") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 01:50:51.371499 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:51.370972 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28f00eb6-78c2-4d93-bde5-c1ea12601dbd-original-pull-secret podName:28f00eb6-78c2-4d93-bde5-c1ea12601dbd nodeName:}" failed. No retries permitted until 2026-11-03 01:50:53.37096204 +0000 UTC m=+5.374409497 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/28f00eb6-78c2-4d93-bde5-c1ea12601dbd-original-pull-secret") pod "global-pull-secret-syncer-kkngd" (UID: "28f00eb6-78c2-4d93-bde5-c1ea12601dbd") : object "kube-system"/"original-pull-secret" not registered Nov 03 01:50:51.471695 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:51.471661 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5qvs" Nov 03 01:50:51.471868 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:51.471709 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kkngd" Nov 03 01:50:51.471868 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:51.471767 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vzfhc" Nov 03 01:50:51.471868 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:51.471783 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5qvs" podUID="6ae1437e-8885-42a5-b092-3862cd404c60" Nov 03 01:50:51.471868 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:51.471854 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vzfhc" podUID="dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b" Nov 03 01:50:51.472072 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:51.471922 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kkngd" podUID="28f00eb6-78c2-4d93-bde5-c1ea12601dbd" Nov 03 01:50:51.478980 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:51.478951 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-f524q" event={"ID":"7d2438ed-bfc2-4639-b87d-84689a82123e","Type":"ContainerStarted","Data":"fb9d1cd812647e2f4378d1305d22e70c4bc844ac53832a289468b8e7e922b5ea"} Nov 03 01:50:51.480244 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:51.480211 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gwlr2" event={"ID":"9152cf36-4997-470e-8902-802a96ebc2c7","Type":"ContainerStarted","Data":"f618f2b28711a3231ea6e344c22c588e935e8fa1b8fb39eda474a018c73fc5ba"} Nov 03 01:50:51.481368 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:51.481329 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vmjc5" event={"ID":"85148983-aa3f-45c2-a062-da2748dd9011","Type":"ContainerStarted","Data":"6e7abde09bd8790efbf067d9fa68b368e263d7f206db973770b8389ef07cbbf0"} Nov 03 01:50:51.482641 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:51.482621 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x452r" event={"ID":"56117bce-d686-4c24-98c8-c0ff122778ac","Type":"ContainerStarted","Data":"19bbae04ef2fe7675bd877cca8a8cac40e2e02ce17e8440de5bf5c040ffd0e83"} Nov 03 01:50:51.483789 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:51.483768 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7zdvv" event={"ID":"7c3f175d-932c-42be-9660-e4fd3dceaf49","Type":"ContainerStarted","Data":"f666cadc2b74c94a27dc69fcae7d613fd3a4651e984de98a8076baf26c9abcbc"} Nov 03 01:50:51.484959 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:51.484935 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdpjb" event={"ID":"0801f13c-d14a-4b9e-b45a-a51af48d17c3","Type":"ContainerStarted","Data":"da8b4e239c1214ccf41568209cfff8db65166c4b47549e0e45e287564408e64e"} Nov 03 01:50:51.486068 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:51.486049 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-trr75" event={"ID":"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b","Type":"ContainerStarted","Data":"69597725f815b11f8be350787f47840c1fcd6e2ffc45c95addf48313ec7cd46d"} Nov 03 01:50:51.487143 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:51.487111 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" event={"ID":"df76407a-ce41-4a4a-b51e-8420b817f97c","Type":"ContainerStarted","Data":"79d01368f40af632c77386836e9b55b47dfdf3598f3f9128db82d00f4d32cbab"} Nov 03 01:50:51.488326 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:51.488303 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gjs9f" event={"ID":"592adf6b-1edd-4f5e-88a2-6ed92895f22a","Type":"ContainerStarted","Data":"6d44cceeee48b2e37bdb5bdef30b12a3578a06e24872cbbcb5fa45a898aab4c0"} Nov 03 01:50:51.741210 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:51.741097 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 01:45:50 +0000 UTC" deadline="2028-03-30 00:50:33.851544559 +0000 UTC" Nov 03 01:50:51.741210 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:51.741162 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12310h59m42.110386641s" Nov 03 01:50:53.390501 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:53.389358 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b-metrics-certs\") pod \"network-metrics-daemon-vzfhc\" (UID: \"dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b\") " pod="openshift-multus/network-metrics-daemon-vzfhc" Nov 03 01:50:53.390501 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:53.389433 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr2jw\" (UniqueName: \"kubernetes.io/projected/6ae1437e-8885-42a5-b092-3862cd404c60-kube-api-access-sr2jw\") pod \"network-check-target-f5qvs\" (UID: \"6ae1437e-8885-42a5-b092-3862cd404c60\") " pod="openshift-network-diagnostics/network-check-target-f5qvs" Nov 03 01:50:53.390501 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:53.389484 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/28f00eb6-78c2-4d93-bde5-c1ea12601dbd-original-pull-secret\") pod \"global-pull-secret-syncer-kkngd\" (UID: \"28f00eb6-78c2-4d93-bde5-c1ea12601dbd\") " pod="kube-system/global-pull-secret-syncer-kkngd" Nov 03 01:50:53.390501 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:53.389618 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 01:50:53.390501 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:53.389703 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28f00eb6-78c2-4d93-bde5-c1ea12601dbd-original-pull-secret podName:28f00eb6-78c2-4d93-bde5-c1ea12601dbd nodeName:}" failed. No retries permitted until 2026-11-03 01:50:57.389683238 +0000 UTC m=+9.393130698 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/28f00eb6-78c2-4d93-bde5-c1ea12601dbd-original-pull-secret") pod "global-pull-secret-syncer-kkngd" (UID: "28f00eb6-78c2-4d93-bde5-c1ea12601dbd") : object "kube-system"/"original-pull-secret" not registered Nov 03 01:50:53.390501 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:53.390170 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 01:50:53.390501 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:53.390233 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b-metrics-certs podName:dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b nodeName:}" failed. No retries permitted until 2026-11-03 01:50:57.390215113 +0000 UTC m=+9.393662576 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b-metrics-certs") pod "network-metrics-daemon-vzfhc" (UID: "dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 01:50:53.390501 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:53.390354 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 01:50:53.390501 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:53.390371 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 01:50:53.390501 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:53.390383 2579 projected.go:196] Error preparing data for projected volume kube-api-access-sr2jw for pod openshift-network-diagnostics/network-check-target-f5qvs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 01:50:53.390501 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:53.390421 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ae1437e-8885-42a5-b092-3862cd404c60-kube-api-access-sr2jw podName:6ae1437e-8885-42a5-b092-3862cd404c60 nodeName:}" failed. No retries permitted until 2026-11-03 01:50:57.390408214 +0000 UTC m=+9.393855673 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-sr2jw" (UniqueName: "kubernetes.io/projected/6ae1437e-8885-42a5-b092-3862cd404c60-kube-api-access-sr2jw") pod "network-check-target-f5qvs" (UID: "6ae1437e-8885-42a5-b092-3862cd404c60") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 01:50:53.472187 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:53.472146 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kkngd" Nov 03 01:50:53.472429 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:53.472363 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kkngd" podUID="28f00eb6-78c2-4d93-bde5-c1ea12601dbd" Nov 03 01:50:53.472551 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:53.472456 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vzfhc" Nov 03 01:50:53.472616 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:53.472557 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vzfhc" podUID="dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b" Nov 03 01:50:53.472670 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:53.472620 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5qvs" Nov 03 01:50:53.472742 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:53.472713 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5qvs" podUID="6ae1437e-8885-42a5-b092-3862cd404c60" Nov 03 01:50:55.471724 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:55.471689 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kkngd" Nov 03 01:50:55.472449 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:55.471825 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kkngd" podUID="28f00eb6-78c2-4d93-bde5-c1ea12601dbd" Nov 03 01:50:55.472449 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:55.471898 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vzfhc" Nov 03 01:50:55.472449 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:55.471993 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vzfhc" podUID="dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b" Nov 03 01:50:55.472449 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:55.472041 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5qvs" Nov 03 01:50:55.472449 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:55.472105 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5qvs" podUID="6ae1437e-8885-42a5-b092-3862cd404c60" Nov 03 01:50:55.515392 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:55.515199 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-140.us-west-2.compute.internal" event={"ID":"d1cefe6117be002b0e139e82f22d4917","Type":"ContainerStarted","Data":"4324339d1bde2f5a0ca57f44f779b47a0fe0177f77f6d2827d9bb41f35384908"} Nov 03 01:50:55.522600 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:55.521889 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gwlr2" event={"ID":"9152cf36-4997-470e-8902-802a96ebc2c7","Type":"ContainerStarted","Data":"a1ad295aeb842338664a10f33a39f2ad700aee049a86a69790085fc0e2e1316a"} Nov 03 01:50:55.530209 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:55.530177 2579 generic.go:356] "Generic (PLEG): container finished" podID="3b5ab86e21798d22090c574595774d10" containerID="0d8c0999791f3327b46bc8895969e4f93e864da713eedc74f06cd76721c12a72" exitCode=0 Nov 03 01:50:55.530306 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:55.530230 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-140.us-west-2.compute.internal" event={"ID":"3b5ab86e21798d22090c574595774d10","Type":"ContainerDied","Data":"0d8c0999791f3327b46bc8895969e4f93e864da713eedc74f06cd76721c12a72"} Nov 03 01:50:55.637018 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:55.636952 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-140.us-west-2.compute.internal" podStartSLOduration=6.636934316 podStartE2EDuration="6.636934316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:50:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:50:55.636692512 +0000 UTC m=+7.640139976" watchObservedRunningTime="2026-11-03 01:50:55.636934316 +0000 UTC m=+7.640381763" Nov 03 01:50:56.537620 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:56.537433 2579 generic.go:356] "Generic (PLEG): container finished" podID="9152cf36-4997-470e-8902-802a96ebc2c7" containerID="a1ad295aeb842338664a10f33a39f2ad700aee049a86a69790085fc0e2e1316a" exitCode=0 Nov 03 01:50:56.537620 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:56.537528 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gwlr2" event={"ID":"9152cf36-4997-470e-8902-802a96ebc2c7","Type":"ContainerDied","Data":"a1ad295aeb842338664a10f33a39f2ad700aee049a86a69790085fc0e2e1316a"} Nov 03 01:50:56.545461 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:56.545362 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-140.us-west-2.compute.internal" event={"ID":"3b5ab86e21798d22090c574595774d10","Type":"ContainerStarted","Data":"25cf6dcfecb0fc1b8717378e9fd9fb8a071982c9d5c4602699f25be38c3df528"} Nov 03 01:50:57.429016 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:57.428899 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/28f00eb6-78c2-4d93-bde5-c1ea12601dbd-original-pull-secret\") pod \"global-pull-secret-syncer-kkngd\" (UID: \"28f00eb6-78c2-4d93-bde5-c1ea12601dbd\") " pod="kube-system/global-pull-secret-syncer-kkngd" Nov 03 01:50:57.429016 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:57.428997 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b-metrics-certs\") pod \"network-metrics-daemon-vzfhc\" (UID: \"dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b\") " pod="openshift-multus/network-metrics-daemon-vzfhc" Nov 03 01:50:57.429246 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:57.429062 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr2jw\" (UniqueName: \"kubernetes.io/projected/6ae1437e-8885-42a5-b092-3862cd404c60-kube-api-access-sr2jw\") pod \"network-check-target-f5qvs\" (UID: \"6ae1437e-8885-42a5-b092-3862cd404c60\") " pod="openshift-network-diagnostics/network-check-target-f5qvs" Nov 03 01:50:57.429246 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:57.429236 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 01:50:57.429349 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:57.429251 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 01:50:57.429349 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:57.429262 2579 projected.go:196] Error preparing data for projected volume kube-api-access-sr2jw for pod openshift-network-diagnostics/network-check-target-f5qvs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 01:50:57.429349 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:57.429309 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ae1437e-8885-42a5-b092-3862cd404c60-kube-api-access-sr2jw podName:6ae1437e-8885-42a5-b092-3862cd404c60 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:05.429295419 +0000 UTC m=+17.432742877 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-sr2jw" (UniqueName: "kubernetes.io/projected/6ae1437e-8885-42a5-b092-3862cd404c60-kube-api-access-sr2jw") pod "network-check-target-f5qvs" (UID: "6ae1437e-8885-42a5-b092-3862cd404c60") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 01:50:57.429536 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:57.429503 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 01:50:57.429594 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:57.429561 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b-metrics-certs podName:dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b nodeName:}" failed. No retries permitted until 2026-11-03 01:51:05.429545507 +0000 UTC m=+17.432992960 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b-metrics-certs") pod "network-metrics-daemon-vzfhc" (UID: "dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 01:50:57.429594 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:57.429559 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 01:50:57.429705 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:57.429631 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28f00eb6-78c2-4d93-bde5-c1ea12601dbd-original-pull-secret podName:28f00eb6-78c2-4d93-bde5-c1ea12601dbd nodeName:}" failed. No retries permitted until 2026-11-03 01:51:05.429610353 +0000 UTC m=+17.433057808 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/28f00eb6-78c2-4d93-bde5-c1ea12601dbd-original-pull-secret") pod "global-pull-secret-syncer-kkngd" (UID: "28f00eb6-78c2-4d93-bde5-c1ea12601dbd") : object "kube-system"/"original-pull-secret" not registered Nov 03 01:50:57.471945 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:57.471893 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kkngd" Nov 03 01:50:57.472125 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:57.472071 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kkngd" podUID="28f00eb6-78c2-4d93-bde5-c1ea12601dbd" Nov 03 01:50:57.472213 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:57.472191 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vzfhc" Nov 03 01:50:57.472350 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:57.472329 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vzfhc" podUID="dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b" Nov 03 01:50:57.472448 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:57.472402 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5qvs" Nov 03 01:50:57.472506 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:57.472489 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5qvs" podUID="6ae1437e-8885-42a5-b092-3862cd404c60" Nov 03 01:50:57.555696 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:57.555649 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-f524q" event={"ID":"7d2438ed-bfc2-4639-b87d-84689a82123e","Type":"ContainerStarted","Data":"b0ea5dfb04e682b3a4f23e5dd0dedb51da60ef2df5196a815716ca54fd1293a8"} Nov 03 01:50:57.677678 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:57.677616 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-140.us-west-2.compute.internal" podStartSLOduration=8.677596003 podStartE2EDuration="8.677596003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:50:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:50:56.668602375 +0000 UTC m=+8.672049842" watchObservedRunningTime="2026-11-03 01:50:57.677596003 +0000 UTC m=+9.681043470" Nov 03 01:50:57.807855 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:57.807830 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-f524q" Nov 03 01:50:57.808244 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:57.808223 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-f524q" Nov 03 01:50:57.941184 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:57.941113 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-f524q" podStartSLOduration=3.633186139 podStartE2EDuration="8.941089676s" totalImagesPullingTime="5.307903537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:50:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:50:57.678625709 +0000 UTC m=+9.682073214" watchObservedRunningTime="2026-11-03 01:50:57.941089676 +0000 UTC m=+9.944537188" Nov 03 01:50:58.561014 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:58.560977 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdpjb" event={"ID":"0801f13c-d14a-4b9e-b45a-a51af48d17c3","Type":"ContainerStarted","Data":"31ee1b24e47a9ee36e43b7c3740252ec977cdd48fd547ceb22ea5d8560a7fe61"} Nov 03 01:50:58.564460 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:58.564429 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gjs9f" event={"ID":"592adf6b-1edd-4f5e-88a2-6ed92895f22a","Type":"ContainerStarted","Data":"506c30789cd6624cae67ed7f5007e67e40423b9f72c8308b6be87025ff9942dc"} Nov 03 01:50:59.472376 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:59.472326 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kkngd" Nov 03 01:50:59.472625 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:59.472593 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kkngd" podUID="28f00eb6-78c2-4d93-bde5-c1ea12601dbd" Nov 03 01:50:59.472750 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:59.472682 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vzfhc" Nov 03 01:50:59.472842 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:59.472820 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vzfhc" podUID="dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b" Nov 03 01:50:59.472907 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:59.472856 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5qvs" Nov 03 01:50:59.472981 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:50:59.472932 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5qvs" podUID="6ae1437e-8885-42a5-b092-3862cd404c60" Nov 03 01:50:59.566087 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:50:59.566058 2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 01:51:00.197903 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:00.197876 2579 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 01:51:00.426143 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:00.425998 2579 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T01:51:00.197891969Z","UUID":"ba7346f7-cc02-4124-bbda-d2526393fede","Handler":null,"Name":"","Endpoint":""} Nov 03 01:51:00.427740 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:00.427709 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 01:51:00.427740 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:00.427744 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 01:51:00.571592 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:00.571557 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gwlr2" event={"ID":"9152cf36-4997-470e-8902-802a96ebc2c7","Type":"ContainerStarted","Data":"7f39f434749ee4bec89a3ec1c468c53f1dbe225a0811178a618924663c8df8f1"} Nov 03 01:51:00.573375 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:00.573326 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7zdvv" event={"ID":"7c3f175d-932c-42be-9660-e4fd3dceaf49","Type":"ContainerStarted","Data":"0b06bf3f305586d88730a2d07b5376d48dbda47191339b82620250f9d3a4c8db"} Nov 03 01:51:00.575847 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:00.575815 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdpjb" event={"ID":"0801f13c-d14a-4b9e-b45a-a51af48d17c3","Type":"ContainerStarted","Data":"9a8b6b0e835b509505d4fbf3a9e5861b8af47c7f2c045010b64acbb57ba6238f"} Nov 03 01:51:00.577835 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:00.577470 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-trr75" event={"ID":"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b","Type":"ContainerStarted","Data":"6e0ba775e13e441bac970bb3f3c7312ce63076b80401dcd970023da12e1faae7"} Nov 03 01:51:00.686620 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:00.686578 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-7zdvv" podStartSLOduration=2.640450502 podStartE2EDuration="11.686564426s" totalImagesPullingTime="9.046113924s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:50:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:51:00.686531518 +0000 UTC m=+12.689978981" watchObservedRunningTime="2026-11-03 01:51:00.686564426 +0000 UTC m=+12.690011879" Nov 03 01:51:00.686745 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:00.686725 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-gjs9f" podStartSLOduration=5.465047385 podStartE2EDuration="11.686718198s" totalImagesPullingTime="6.221670813s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:50:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:50:58.679034504 +0000 UTC m=+10.682481972" watchObservedRunningTime="2026-11-03 01:51:00.686718198 +0000 UTC m=+12.690165660" Nov 03 01:51:00.802973 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:00.802914 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-trr75" podStartSLOduration=3.185089229 podStartE2EDuration="11.802895958s" totalImagesPullingTime="8.617806729s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:50:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:51:00.802067753 +0000 UTC m=+12.805515220" watchObservedRunningTime="2026-11-03 01:51:00.802895958 +0000 UTC m=+12.806343421" Nov 03 01:51:01.472343 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:01.472304 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kkngd" Nov 03 01:51:01.472532 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:01.472511 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kkngd" podUID="28f00eb6-78c2-4d93-bde5-c1ea12601dbd" Nov 03 01:51:01.472853 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:01.472838 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vzfhc" Nov 03 01:51:01.472968 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:01.472953 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vzfhc" podUID="dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b" Nov 03 01:51:01.473019 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:01.473005 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5qvs" Nov 03 01:51:01.473087 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:01.473072 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5qvs" podUID="6ae1437e-8885-42a5-b092-3862cd404c60" Nov 03 01:51:01.581167 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:01.581139 2579 generic.go:356] "Generic (PLEG): container finished" podID="9152cf36-4997-470e-8902-802a96ebc2c7" containerID="7f39f434749ee4bec89a3ec1c468c53f1dbe225a0811178a618924663c8df8f1" exitCode=0 Nov 03 01:51:01.581517 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:01.581219 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gwlr2" event={"ID":"9152cf36-4997-470e-8902-802a96ebc2c7","Type":"ContainerDied","Data":"7f39f434749ee4bec89a3ec1c468c53f1dbe225a0811178a618924663c8df8f1"} Nov 03 01:51:01.585288 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:01.584726 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vmjc5" event={"ID":"85148983-aa3f-45c2-a062-da2748dd9011","Type":"ContainerStarted","Data":"391bf568245733ccef68c53369c01576800c35eb96e285b201637328e5da8049"} Nov 03 01:51:01.968614 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:01.968560 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vmjc5" podStartSLOduration=3.806524721 podStartE2EDuration="12.968541417s" totalImagesPullingTime="9.162016696s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:50:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:51:01.963941274 +0000 UTC m=+13.967388740" watchObservedRunningTime="2026-11-03 01:51:01.968541417 +0000 UTC m=+13.971988885" Nov 03 01:51:02.588226 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:02.587972 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdpjb" event={"ID":"0801f13c-d14a-4b9e-b45a-a51af48d17c3","Type":"ContainerStarted","Data":"a2bfa2972016df64de316c468379f329889551be673a7582e0a0037bcddbfa84"} Nov 03 01:51:02.589383 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:02.589345 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" event={"ID":"df76407a-ce41-4a4a-b51e-8420b817f97c","Type":"ContainerStarted","Data":"6f84c442b57d2aa0f283e8c986fd5b1bd1c68806596370d4d997dfa4287a2acb"} Nov 03 01:51:02.704750 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:02.704692 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdpjb" podStartSLOduration=3.125204024 podStartE2EDuration="13.704674565s" totalImagesPullingTime="10.579470541s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:50:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:51:02.704267141 +0000 UTC m=+14.707714603" watchObservedRunningTime="2026-11-03 01:51:02.704674565 +0000 UTC m=+14.708122030" Nov 03 01:51:02.819044 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:02.819001 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" podStartSLOduration=2.7318821890000002 podStartE2EDuration="13.818988191s" totalImagesPullingTime="11.087106002s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:50:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:51:02.81873022 +0000 UTC m=+14.822177695" watchObservedRunningTime="2026-11-03 01:51:02.818988191 +0000 UTC m=+14.822435653" Nov 03 01:51:03.472007 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:03.471978 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kkngd" Nov 03 01:51:03.472226 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:03.472091 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vzfhc" Nov 03 01:51:03.472226 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:03.472103 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kkngd" podUID="28f00eb6-78c2-4d93-bde5-c1ea12601dbd" Nov 03 01:51:03.472226 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:03.472204 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vzfhc" podUID="dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b" Nov 03 01:51:03.472424 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:03.472227 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5qvs" Nov 03 01:51:03.472424 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:03.472307 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5qvs" podUID="6ae1437e-8885-42a5-b092-3862cd404c60" Nov 03 01:51:03.592885 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:03.592856 2579 generic.go:356] "Generic (PLEG): container finished" podID="9152cf36-4997-470e-8902-802a96ebc2c7" containerID="ef3a3ad4dcd0b7b6726fd15c67345b4bd8d407e35a646a94eff195a8ee59092d" exitCode=0 Nov 03 01:51:03.593457 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:03.592948 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gwlr2" event={"ID":"9152cf36-4997-470e-8902-802a96ebc2c7","Type":"ContainerDied","Data":"ef3a3ad4dcd0b7b6726fd15c67345b4bd8d407e35a646a94eff195a8ee59092d"} Nov 03 01:51:05.472407 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:05.472336 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kkngd" Nov 03 01:51:05.473174 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:05.472442 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5qvs" Nov 03 01:51:05.473174 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:05.472557 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kkngd" podUID="28f00eb6-78c2-4d93-bde5-c1ea12601dbd" Nov 03 01:51:05.473174 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:05.472612 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vzfhc" Nov 03 01:51:05.473174 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:05.472716 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5qvs" podUID="6ae1437e-8885-42a5-b092-3862cd404c60" Nov 03 01:51:05.473174 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:05.472812 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vzfhc" podUID="dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b" Nov 03 01:51:05.502531 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:05.502493 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr2jw\" (UniqueName: \"kubernetes.io/projected/6ae1437e-8885-42a5-b092-3862cd404c60-kube-api-access-sr2jw\") pod \"network-check-target-f5qvs\" (UID: \"6ae1437e-8885-42a5-b092-3862cd404c60\") " pod="openshift-network-diagnostics/network-check-target-f5qvs" Nov 03 01:51:05.502698 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:05.502554 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/28f00eb6-78c2-4d93-bde5-c1ea12601dbd-original-pull-secret\") pod \"global-pull-secret-syncer-kkngd\" (UID: \"28f00eb6-78c2-4d93-bde5-c1ea12601dbd\") " pod="kube-system/global-pull-secret-syncer-kkngd" Nov 03 01:51:05.502698 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:05.502583 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b-metrics-certs\") pod \"network-metrics-daemon-vzfhc\" (UID: \"dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b\") " pod="openshift-multus/network-metrics-daemon-vzfhc" Nov 03 01:51:05.502698 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:05.502652 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 01:51:05.502698 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:05.502680 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 01:51:05.502698 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:05.502693 2579 projected.go:196] Error preparing data for projected volume kube-api-access-sr2jw for pod openshift-network-diagnostics/network-check-target-f5qvs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 01:51:05.502698 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:05.502693 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 01:51:05.502892 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:05.502746 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b-metrics-certs podName:dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b nodeName:}" failed. No retries permitted until 2026-11-03 01:51:21.502732022 +0000 UTC m=+33.506179464 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b-metrics-certs") pod "network-metrics-daemon-vzfhc" (UID: "dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 01:51:05.502892 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:05.502759 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ae1437e-8885-42a5-b092-3862cd404c60-kube-api-access-sr2jw podName:6ae1437e-8885-42a5-b092-3862cd404c60 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:21.502753645 +0000 UTC m=+33.506201087 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-sr2jw" (UniqueName: "kubernetes.io/projected/6ae1437e-8885-42a5-b092-3862cd404c60-kube-api-access-sr2jw") pod "network-check-target-f5qvs" (UID: "6ae1437e-8885-42a5-b092-3862cd404c60") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 01:51:05.502892 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:05.502693 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 01:51:05.502892 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:05.502787 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28f00eb6-78c2-4d93-bde5-c1ea12601dbd-original-pull-secret podName:28f00eb6-78c2-4d93-bde5-c1ea12601dbd nodeName:}" failed. No retries permitted until 2026-11-03 01:51:21.502781651 +0000 UTC m=+33.506229093 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/28f00eb6-78c2-4d93-bde5-c1ea12601dbd-original-pull-secret") pod "global-pull-secret-syncer-kkngd" (UID: "28f00eb6-78c2-4d93-bde5-c1ea12601dbd") : object "kube-system"/"original-pull-secret" not registered Nov 03 01:51:05.600786 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:05.600675 2579 generic.go:356] "Generic (PLEG): container finished" podID="9152cf36-4997-470e-8902-802a96ebc2c7" containerID="ee19309bbbeb838da62c2138b0fd04e48db6c4e4c2e7463b36f475442b9abf6b" exitCode=0 Nov 03 01:51:05.600786 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:05.600728 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gwlr2" event={"ID":"9152cf36-4997-470e-8902-802a96ebc2c7","Type":"ContainerDied","Data":"ee19309bbbeb838da62c2138b0fd04e48db6c4e4c2e7463b36f475442b9abf6b"} Nov 03 01:51:06.810824 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:06.810793 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-f524q" Nov 03 01:51:06.811302 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:06.810949 2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 01:51:06.811628 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:06.811614 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-f524q" Nov 03 01:51:07.472386 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:07.472184 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kkngd" Nov 03 01:51:07.472468 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:07.472239 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vzfhc" Nov 03 01:51:07.472527 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:07.472501 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kkngd" podUID="28f00eb6-78c2-4d93-bde5-c1ea12601dbd" Nov 03 01:51:07.472565 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:07.472244 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5qvs" Nov 03 01:51:07.472594 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:07.472573 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vzfhc" podUID="dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b" Nov 03 01:51:07.472645 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:07.472626 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5qvs" podUID="6ae1437e-8885-42a5-b092-3862cd404c60" Nov 03 01:51:07.608800 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:07.608555 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x452r" event={"ID":"56117bce-d686-4c24-98c8-c0ff122778ac","Type":"ContainerStarted","Data":"e6c5a37985ab990ad184abb30d8f50b1c2aaaa2faa3d1b4374efbb476bd735bd"} Nov 03 01:51:07.608800 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:07.608695 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x452r" event={"ID":"56117bce-d686-4c24-98c8-c0ff122778ac","Type":"ContainerStarted","Data":"3eea4c80220c8f1432f39540c0fce278a563285bb9a70fbffb74561ca7be2f2b"} Nov 03 01:51:07.608800 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:07.608745 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x452r" event={"ID":"56117bce-d686-4c24-98c8-c0ff122778ac","Type":"ContainerStarted","Data":"868f43d3f2c92057f7021e77092dcb94407e235b7b21a410c975843c8bc55553"} Nov 03 01:51:07.608800 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:07.608795 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x452r" event={"ID":"56117bce-d686-4c24-98c8-c0ff122778ac","Type":"ContainerStarted","Data":"379e1c01fb6eb8a75f4b2f75c4122ff5dddcd5289ac59bd25273c6c24bf77fe7"} Nov 03 01:51:08.612853 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:08.612648 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x452r" event={"ID":"56117bce-d686-4c24-98c8-c0ff122778ac","Type":"ContainerStarted","Data":"37724f6d1478271bb8645d01dbb7d71340e3c9ea53b622f4048fb31210cd0db0"} Nov 03 01:51:08.613317 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:08.612909 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x452r" event={"ID":"56117bce-d686-4c24-98c8-c0ff122778ac","Type":"ContainerStarted","Data":"cbfa01292794b04103ef425b5326dfdd21648d6f8ea0f323cb092c4a7cf74cbb"} Nov 03 01:51:08.615207 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:08.615183 2579 generic.go:356] "Generic (PLEG): container finished" podID="9152cf36-4997-470e-8902-802a96ebc2c7" containerID="ace6e3a8d61e46c606efc566bae17f1879a9fb1930a5ccb382963042173a04c5" exitCode=0 Nov 03 01:51:08.615301 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:08.615215 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gwlr2" event={"ID":"9152cf36-4997-470e-8902-802a96ebc2c7","Type":"ContainerDied","Data":"ace6e3a8d61e46c606efc566bae17f1879a9fb1930a5ccb382963042173a04c5"} Nov 03 01:51:09.471811 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:09.471773 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kkngd" Nov 03 01:51:09.471811 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:09.471816 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5qvs" Nov 03 01:51:09.472042 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:09.471902 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kkngd" podUID="28f00eb6-78c2-4d93-bde5-c1ea12601dbd" Nov 03 01:51:09.472042 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:09.471950 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vzfhc" Nov 03 01:51:09.472132 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:09.472053 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5qvs" podUID="6ae1437e-8885-42a5-b092-3862cd404c60" Nov 03 01:51:09.472165 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:09.472145 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vzfhc" podUID="dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b" Nov 03 01:51:09.620431 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:09.620397 2579 generic.go:356] "Generic (PLEG): container finished" podID="9152cf36-4997-470e-8902-802a96ebc2c7" containerID="2cde1738af27fdb2d6b0d1a8a95c5a7ece144ca35e2b7549038be52669e42954" exitCode=0 Nov 03 01:51:09.620431 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:09.620432 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gwlr2" event={"ID":"9152cf36-4997-470e-8902-802a96ebc2c7","Type":"ContainerDied","Data":"2cde1738af27fdb2d6b0d1a8a95c5a7ece144ca35e2b7549038be52669e42954"} Nov 03 01:51:10.625692 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:10.625660 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gwlr2" event={"ID":"9152cf36-4997-470e-8902-802a96ebc2c7","Type":"ContainerStarted","Data":"d8ad192d23a7deb068c19f72098614c9c33a02445dd939f32e0cfb50c702eee2"} Nov 03 01:51:10.628662 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:10.628638 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x452r" event={"ID":"56117bce-d686-4c24-98c8-c0ff122778ac","Type":"ContainerStarted","Data":"738d0d58dbed6c26ccf56dd2529b827c3b3f3b25e89603db896b5fa3b6d6642e"} Nov 03 01:51:10.744552 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:10.744504 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-gwlr2" podStartSLOduration=8.102384145 podStartE2EDuration="21.744491544s" totalImagesPullingTime="13.642107399s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:50:49 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:51:10.743760167 +0000 UTC m=+22.747207629" watchObservedRunningTime="2026-11-03 01:51:10.744491544 +0000 UTC m=+22.747939007" Nov 03 01:51:11.471643 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:11.471603 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5qvs" Nov 03 01:51:11.471821 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:11.471708 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kkngd" Nov 03 01:51:11.471821 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:11.471718 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5qvs" podUID="6ae1437e-8885-42a5-b092-3862cd404c60" Nov 03 01:51:11.471821 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:11.471789 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vzfhc" Nov 03 01:51:11.471936 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:11.471788 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kkngd" podUID="28f00eb6-78c2-4d93-bde5-c1ea12601dbd" Nov 03 01:51:11.471936 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:11.471866 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vzfhc" podUID="dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b" Nov 03 01:51:12.636153 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:12.636115 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x452r" event={"ID":"56117bce-d686-4c24-98c8-c0ff122778ac","Type":"ContainerStarted","Data":"deb13c16d2302045eed4dbc9dcb2bcda737c0fd5987f9b83c7f74562c3894241"} Nov 03 01:51:12.637045 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:12.636422 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:51:12.637045 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:12.636485 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:51:12.637045 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:12.636517 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:51:12.651774 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:12.651747 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:51:12.651917 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:12.651846 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:51:12.912720 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:12.912618 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-x452r" podStartSLOduration=7.603557354 podStartE2EDuration="23.912604116s" totalImagesPullingTime="16.309046762s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:50:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:51:12.912097626 +0000 UTC m=+24.915545087" watchObservedRunningTime="2026-11-03 01:51:12.912604116 +0000 UTC m=+24.916051579" Nov 03 01:51:13.471918 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:13.471884 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5qvs" Nov 03 01:51:13.471918 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:13.471914 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kkngd" Nov 03 01:51:13.472131 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:13.471886 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vzfhc" Nov 03 01:51:13.472131 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:13.471999 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5qvs" podUID="6ae1437e-8885-42a5-b092-3862cd404c60" Nov 03 01:51:13.472131 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:13.472078 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kkngd" podUID="28f00eb6-78c2-4d93-bde5-c1ea12601dbd" Nov 03 01:51:13.472218 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:13.472142 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vzfhc" podUID="dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b" Nov 03 01:51:14.598988 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:14.598786 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kkngd"] Nov 03 01:51:14.599397 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:14.599080 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kkngd" Nov 03 01:51:14.599397 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:14.599196 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kkngd" podUID="28f00eb6-78c2-4d93-bde5-c1ea12601dbd" Nov 03 01:51:14.602163 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:14.602135 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vzfhc"] Nov 03 01:51:14.602310 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:14.602242 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vzfhc" Nov 03 01:51:14.602394 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:14.602373 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vzfhc" podUID="dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b" Nov 03 01:51:14.602643 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:14.602619 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-f5qvs"] Nov 03 01:51:14.602728 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:14.602716 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5qvs" Nov 03 01:51:14.602806 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:14.602789 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5qvs" podUID="6ae1437e-8885-42a5-b092-3862cd404c60" Nov 03 01:51:14.969015 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:14.968940 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7zdvv_7c3f175d-932c-42be-9660-e4fd3dceaf49/dns-node-resolver/0.log" Nov 03 01:51:16.014113 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:16.014082 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gjs9f_592adf6b-1edd-4f5e-88a2-6ed92895f22a/node-ca/0.log" Nov 03 01:51:16.472038 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:16.472000 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5qvs" Nov 03 01:51:16.472263 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:16.472012 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vzfhc" Nov 03 01:51:16.472263 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:16.472117 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kkngd" Nov 03 01:51:16.472263 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:16.472124 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5qvs" podUID="6ae1437e-8885-42a5-b092-3862cd404c60" Nov 03 01:51:16.472263 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:16.472192 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vzfhc" podUID="dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b" Nov 03 01:51:16.472513 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:16.472257 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kkngd" podUID="28f00eb6-78c2-4d93-bde5-c1ea12601dbd" Nov 03 01:51:18.472597 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:18.472560 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vzfhc" Nov 03 01:51:18.473068 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:18.472661 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vzfhc" podUID="dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b" Nov 03 01:51:18.473068 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:18.472752 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5qvs" Nov 03 01:51:18.473068 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:18.472865 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5qvs" podUID="6ae1437e-8885-42a5-b092-3862cd404c60" Nov 03 01:51:18.473068 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:18.472884 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kkngd" Nov 03 01:51:18.473068 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:18.473001 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kkngd" podUID="28f00eb6-78c2-4d93-bde5-c1ea12601dbd" Nov 03 01:51:20.471949 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:20.471911 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5qvs" Nov 03 01:51:20.472411 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:20.471923 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vzfhc" Nov 03 01:51:20.472411 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:20.472039 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5qvs" podUID="6ae1437e-8885-42a5-b092-3862cd404c60" Nov 03 01:51:20.472411 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:20.472053 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kkngd" Nov 03 01:51:20.472411 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:20.472138 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vzfhc" podUID="dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b" Nov 03 01:51:20.472411 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:20.472203 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kkngd" podUID="28f00eb6-78c2-4d93-bde5-c1ea12601dbd" Nov 03 01:51:21.514308 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:21.514269 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/28f00eb6-78c2-4d93-bde5-c1ea12601dbd-original-pull-secret\") pod \"global-pull-secret-syncer-kkngd\" (UID: \"28f00eb6-78c2-4d93-bde5-c1ea12601dbd\") " pod="kube-system/global-pull-secret-syncer-kkngd" Nov 03 01:51:21.514308 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:21.514317 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b-metrics-certs\") pod \"network-metrics-daemon-vzfhc\" (UID: \"dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b\") " pod="openshift-multus/network-metrics-daemon-vzfhc" Nov 03 01:51:21.514824 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:21.514421 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 01:51:21.514824 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:21.514442 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 01:51:21.514824 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:21.514442 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr2jw\" (UniqueName: \"kubernetes.io/projected/6ae1437e-8885-42a5-b092-3862cd404c60-kube-api-access-sr2jw\") pod \"network-check-target-f5qvs\" (UID: \"6ae1437e-8885-42a5-b092-3862cd404c60\") " pod="openshift-network-diagnostics/network-check-target-f5qvs" Nov 03 01:51:21.514824 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:21.514488 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b-metrics-certs podName:dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b nodeName:}" failed. No retries permitted until 2026-11-03 01:51:53.514472435 +0000 UTC m=+65.517919896 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b-metrics-certs") pod "network-metrics-daemon-vzfhc" (UID: "dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 01:51:21.514824 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:21.514509 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28f00eb6-78c2-4d93-bde5-c1ea12601dbd-original-pull-secret podName:28f00eb6-78c2-4d93-bde5-c1ea12601dbd nodeName:}" failed. No retries permitted until 2026-11-03 01:51:53.514502839 +0000 UTC m=+65.517950280 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/28f00eb6-78c2-4d93-bde5-c1ea12601dbd-original-pull-secret") pod "global-pull-secret-syncer-kkngd" (UID: "28f00eb6-78c2-4d93-bde5-c1ea12601dbd") : object "kube-system"/"original-pull-secret" not registered Nov 03 01:51:21.514824 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:21.514521 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 01:51:21.514824 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:21.514538 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 01:51:21.514824 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:21.514550 2579 projected.go:196] Error preparing data for projected volume kube-api-access-sr2jw for pod openshift-network-diagnostics/network-check-target-f5qvs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 01:51:21.514824 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:21.514596 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ae1437e-8885-42a5-b092-3862cd404c60-kube-api-access-sr2jw podName:6ae1437e-8885-42a5-b092-3862cd404c60 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:53.514581339 +0000 UTC m=+65.518028796 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-sr2jw" (UniqueName: "kubernetes.io/projected/6ae1437e-8885-42a5-b092-3862cd404c60-kube-api-access-sr2jw") pod "network-check-target-f5qvs" (UID: "6ae1437e-8885-42a5-b092-3862cd404c60") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 01:51:22.471451 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:22.471424 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kkngd" Nov 03 01:51:22.471451 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:22.471448 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vzfhc" Nov 03 01:51:22.471709 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:22.471459 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5qvs" Nov 03 01:51:22.471709 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:22.471516 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kkngd" podUID="28f00eb6-78c2-4d93-bde5-c1ea12601dbd" Nov 03 01:51:22.471709 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:22.471632 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vzfhc" podUID="dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b" Nov 03 01:51:22.471709 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:22.471680 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5qvs" podUID="6ae1437e-8885-42a5-b092-3862cd404c60" Nov 03 01:51:24.472229 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:24.472144 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5qvs" Nov 03 01:51:24.472643 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:24.472161 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kkngd" Nov 03 01:51:24.472643 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:24.472254 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5qvs" podUID="6ae1437e-8885-42a5-b092-3862cd404c60" Nov 03 01:51:24.472643 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:24.472307 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vzfhc" Nov 03 01:51:24.472643 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:24.472436 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kkngd" podUID="28f00eb6-78c2-4d93-bde5-c1ea12601dbd" Nov 03 01:51:24.472643 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:24.472496 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vzfhc" podUID="dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b" Nov 03 01:51:26.471978 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:26.471946 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5qvs" Nov 03 01:51:26.472434 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:26.471948 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vzfhc" Nov 03 01:51:26.472434 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:26.472056 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5qvs" podUID="6ae1437e-8885-42a5-b092-3862cd404c60" Nov 03 01:51:26.472434 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:26.472111 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vzfhc" podUID="dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b" Nov 03 01:51:26.472434 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:26.471961 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kkngd" Nov 03 01:51:26.472434 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:26.472183 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kkngd" podUID="28f00eb6-78c2-4d93-bde5-c1ea12601dbd" Nov 03 01:51:28.472601 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:28.472573 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vzfhc" Nov 03 01:51:28.473081 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:28.472671 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5qvs" Nov 03 01:51:28.473081 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:28.472683 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vzfhc" podUID="dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b" Nov 03 01:51:28.473081 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:28.472782 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kkngd" Nov 03 01:51:28.473081 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:28.472796 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5qvs" podUID="6ae1437e-8885-42a5-b092-3862cd404c60" Nov 03 01:51:28.473081 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:28.472855 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kkngd" podUID="28f00eb6-78c2-4d93-bde5-c1ea12601dbd" Nov 03 01:51:30.471479 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:30.471445 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vzfhc" Nov 03 01:51:30.471862 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:30.471451 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5qvs" Nov 03 01:51:30.471862 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:30.471556 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vzfhc" podUID="dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b" Nov 03 01:51:30.471862 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:30.471460 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kkngd" Nov 03 01:51:30.471862 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:30.471634 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5qvs" podUID="6ae1437e-8885-42a5-b092-3862cd404c60" Nov 03 01:51:30.471862 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:30.471691 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kkngd" podUID="28f00eb6-78c2-4d93-bde5-c1ea12601dbd" Nov 03 01:51:32.474305 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:32.474278 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kkngd" Nov 03 01:51:32.474677 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:32.474421 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vzfhc" Nov 03 01:51:32.474677 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:32.474425 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kkngd" podUID="28f00eb6-78c2-4d93-bde5-c1ea12601dbd" Nov 03 01:51:32.474677 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:32.474520 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vzfhc" podUID="dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b" Nov 03 01:51:32.474677 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:32.474567 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5qvs" Nov 03 01:51:32.474677 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:32.474611 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5qvs" podUID="6ae1437e-8885-42a5-b092-3862cd404c60" Nov 03 01:51:34.472452 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:34.472425 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vzfhc" Nov 03 01:51:34.472452 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:34.472438 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5qvs" Nov 03 01:51:34.472914 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:34.472523 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vzfhc" podUID="dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b" Nov 03 01:51:34.472914 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:34.472598 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5qvs" podUID="6ae1437e-8885-42a5-b092-3862cd404c60" Nov 03 01:51:34.472914 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:34.472632 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kkngd" Nov 03 01:51:34.472914 ip-10-0-140-140 kubenswrapper[2579]: E1103 01:51:34.472684 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kkngd" podUID="28f00eb6-78c2-4d93-bde5-c1ea12601dbd" Nov 03 01:51:36.290426 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.290402 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-140.us-west-2.compute.internal" event="NodeReady" Nov 03 01:51:36.290771 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.290536 2579 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 01:51:36.450076 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.450041 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-t2dwr"] Nov 03 01:51:36.452351 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.452334 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-2d49s"] Nov 03 01:51:36.452477 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.452457 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-t2dwr" Nov 03 01:51:36.454919 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.454900 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2d49s" Nov 03 01:51:36.459566 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.457536 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dmrb8"] Nov 03 01:51:36.462271 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.462253 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-t2dwr"] Nov 03 01:51:36.462349 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.462282 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dmrb8"] Nov 03 01:51:36.462404 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.462349 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dmrb8" Nov 03 01:51:36.471838 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.471795 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kkngd" Nov 03 01:51:36.471838 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.471803 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vzfhc" Nov 03 01:51:36.471974 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.471797 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5qvs" Nov 03 01:51:36.497928 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.497904 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2d49s"] Nov 03 01:51:36.505622 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.505603 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 01:51:36.505811 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.505794 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bsx6s\"" Nov 03 01:51:36.505914 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.505811 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 01:51:36.507740 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.507727 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 01:51:36.507801 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.507767 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 01:51:36.507801 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.507774 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-v5ks7\"" Nov 03 01:51:36.508097 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.508084 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 01:51:36.605873 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.605839 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 01:51:36.606047 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.606034 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jwnvd\"" Nov 03 01:51:36.606118 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.606036 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 01:51:36.606118 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.606063 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 01:51:36.606118 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.606086 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 01:51:36.606243 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.606124 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 01:51:36.606243 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.606207 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 01:51:36.606243 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.606222 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-zxsvp\"" Nov 03 01:51:36.606353 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.606266 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-wpc48\"" Nov 03 01:51:36.606353 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.606284 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 01:51:36.606353 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.606325 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-btxgv\"" Nov 03 01:51:36.606473 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.606381 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 01:51:36.616758 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.616739 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3bc52103-b80b-49a8-9bed-15b196fdef12-crio-socket\") pod \"insights-runtime-extractor-dmrb8\" (UID: \"3bc52103-b80b-49a8-9bed-15b196fdef12\") " pod="openshift-insights/insights-runtime-extractor-dmrb8" Nov 03 01:51:36.616814 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.616780 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtzct\" (UniqueName: \"kubernetes.io/projected/0f363d1e-561d-43ae-9d53-379c86b2a558-kube-api-access-dtzct\") pod \"ingress-canary-2d49s\" (UID: \"0f363d1e-561d-43ae-9d53-379c86b2a558\") " pod="openshift-ingress-canary/ingress-canary-2d49s" Nov 03 01:51:36.616814 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.616804 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3bc52103-b80b-49a8-9bed-15b196fdef12-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dmrb8\" (UID: \"3bc52103-b80b-49a8-9bed-15b196fdef12\") " pod="openshift-insights/insights-runtime-extractor-dmrb8" Nov 03 01:51:36.616882 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.616855 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3bc52103-b80b-49a8-9bed-15b196fdef12-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dmrb8\" (UID: \"3bc52103-b80b-49a8-9bed-15b196fdef12\") " pod="openshift-insights/insights-runtime-extractor-dmrb8" Nov 03 01:51:36.616925 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.616894 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0f363d1e-561d-43ae-9d53-379c86b2a558-cert\") pod \"ingress-canary-2d49s\" (UID: \"0f363d1e-561d-43ae-9d53-379c86b2a558\") " pod="openshift-ingress-canary/ingress-canary-2d49s" Nov 03 01:51:36.616972 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.616925 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3bc52103-b80b-49a8-9bed-15b196fdef12-data-volume\") pod \"insights-runtime-extractor-dmrb8\" (UID: \"3bc52103-b80b-49a8-9bed-15b196fdef12\") " pod="openshift-insights/insights-runtime-extractor-dmrb8" Nov 03 01:51:36.617040 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.617026 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52b546b3-7bf6-4239-91cb-c8aaca1c9d18-tmp-dir\") pod \"dns-default-t2dwr\" (UID: \"52b546b3-7bf6-4239-91cb-c8aaca1c9d18\") " pod="openshift-dns/dns-default-t2dwr" Nov 03 01:51:36.617087 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.617050 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqvvj\" (UniqueName: \"kubernetes.io/projected/52b546b3-7bf6-4239-91cb-c8aaca1c9d18-kube-api-access-pqvvj\") pod \"dns-default-t2dwr\" (UID: \"52b546b3-7bf6-4239-91cb-c8aaca1c9d18\") " pod="openshift-dns/dns-default-t2dwr" Nov 03 01:51:36.617172 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.617157 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/52b546b3-7bf6-4239-91cb-c8aaca1c9d18-config-volume\") pod \"dns-default-t2dwr\" (UID: \"52b546b3-7bf6-4239-91cb-c8aaca1c9d18\") " pod="openshift-dns/dns-default-t2dwr" Nov 03 01:51:36.617225 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.617181 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/52b546b3-7bf6-4239-91cb-c8aaca1c9d18-metrics-tls\") pod \"dns-default-t2dwr\" (UID: \"52b546b3-7bf6-4239-91cb-c8aaca1c9d18\") " pod="openshift-dns/dns-default-t2dwr" Nov 03 01:51:36.617267 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.617225 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk6qs\" (UniqueName: \"kubernetes.io/projected/3bc52103-b80b-49a8-9bed-15b196fdef12-kube-api-access-xk6qs\") pod \"insights-runtime-extractor-dmrb8\" (UID: \"3bc52103-b80b-49a8-9bed-15b196fdef12\") " pod="openshift-insights/insights-runtime-extractor-dmrb8" Nov 03 01:51:36.718490 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.718460 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtzct\" (UniqueName: \"kubernetes.io/projected/0f363d1e-561d-43ae-9d53-379c86b2a558-kube-api-access-dtzct\") pod \"ingress-canary-2d49s\" (UID: \"0f363d1e-561d-43ae-9d53-379c86b2a558\") " pod="openshift-ingress-canary/ingress-canary-2d49s" Nov 03 01:51:36.718490 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.718491 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3bc52103-b80b-49a8-9bed-15b196fdef12-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dmrb8\" (UID: \"3bc52103-b80b-49a8-9bed-15b196fdef12\") " pod="openshift-insights/insights-runtime-extractor-dmrb8" Nov 03 01:51:36.718675 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.718512 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3bc52103-b80b-49a8-9bed-15b196fdef12-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dmrb8\" (UID: \"3bc52103-b80b-49a8-9bed-15b196fdef12\") " pod="openshift-insights/insights-runtime-extractor-dmrb8" Nov 03 01:51:36.718715 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.718701 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0f363d1e-561d-43ae-9d53-379c86b2a558-cert\") pod \"ingress-canary-2d49s\" (UID: \"0f363d1e-561d-43ae-9d53-379c86b2a558\") " pod="openshift-ingress-canary/ingress-canary-2d49s" Nov 03 01:51:36.718747 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.718731 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3bc52103-b80b-49a8-9bed-15b196fdef12-data-volume\") pod \"insights-runtime-extractor-dmrb8\" (UID: \"3bc52103-b80b-49a8-9bed-15b196fdef12\") " pod="openshift-insights/insights-runtime-extractor-dmrb8" Nov 03 01:51:36.718791 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.718748 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52b546b3-7bf6-4239-91cb-c8aaca1c9d18-tmp-dir\") pod \"dns-default-t2dwr\" (UID: \"52b546b3-7bf6-4239-91cb-c8aaca1c9d18\") " pod="openshift-dns/dns-default-t2dwr" Nov 03 01:51:36.718825 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.718794 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqvvj\" (UniqueName: \"kubernetes.io/projected/52b546b3-7bf6-4239-91cb-c8aaca1c9d18-kube-api-access-pqvvj\") pod \"dns-default-t2dwr\" (UID: \"52b546b3-7bf6-4239-91cb-c8aaca1c9d18\") " pod="openshift-dns/dns-default-t2dwr" Nov 03 01:51:36.718869 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.718845 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/52b546b3-7bf6-4239-91cb-c8aaca1c9d18-config-volume\") pod \"dns-default-t2dwr\" (UID: \"52b546b3-7bf6-4239-91cb-c8aaca1c9d18\") " pod="openshift-dns/dns-default-t2dwr" Nov 03 01:51:36.718869 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.718862 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/52b546b3-7bf6-4239-91cb-c8aaca1c9d18-metrics-tls\") pod \"dns-default-t2dwr\" (UID: \"52b546b3-7bf6-4239-91cb-c8aaca1c9d18\") " pod="openshift-dns/dns-default-t2dwr" Nov 03 01:51:36.718954 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.718901 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk6qs\" (UniqueName: \"kubernetes.io/projected/3bc52103-b80b-49a8-9bed-15b196fdef12-kube-api-access-xk6qs\") pod \"insights-runtime-extractor-dmrb8\" (UID: \"3bc52103-b80b-49a8-9bed-15b196fdef12\") " pod="openshift-insights/insights-runtime-extractor-dmrb8" Nov 03 01:51:36.718954 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.718934 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3bc52103-b80b-49a8-9bed-15b196fdef12-crio-socket\") pod \"insights-runtime-extractor-dmrb8\" (UID: \"3bc52103-b80b-49a8-9bed-15b196fdef12\") " pod="openshift-insights/insights-runtime-extractor-dmrb8" Nov 03 01:51:36.719050 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.719018 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3bc52103-b80b-49a8-9bed-15b196fdef12-crio-socket\") pod \"insights-runtime-extractor-dmrb8\" (UID: \"3bc52103-b80b-49a8-9bed-15b196fdef12\") " pod="openshift-insights/insights-runtime-extractor-dmrb8" Nov 03 01:51:36.719099 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.719052 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3bc52103-b80b-49a8-9bed-15b196fdef12-data-volume\") pod \"insights-runtime-extractor-dmrb8\" (UID: \"3bc52103-b80b-49a8-9bed-15b196fdef12\") " pod="openshift-insights/insights-runtime-extractor-dmrb8" Nov 03 01:51:36.719166 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.719145 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52b546b3-7bf6-4239-91cb-c8aaca1c9d18-tmp-dir\") pod \"dns-default-t2dwr\" (UID: \"52b546b3-7bf6-4239-91cb-c8aaca1c9d18\") " pod="openshift-dns/dns-default-t2dwr" Nov 03 01:51:36.719219 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.719199 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3bc52103-b80b-49a8-9bed-15b196fdef12-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dmrb8\" (UID: \"3bc52103-b80b-49a8-9bed-15b196fdef12\") " pod="openshift-insights/insights-runtime-extractor-dmrb8" Nov 03 01:51:36.719432 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.719411 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/52b546b3-7bf6-4239-91cb-c8aaca1c9d18-config-volume\") pod \"dns-default-t2dwr\" (UID: \"52b546b3-7bf6-4239-91cb-c8aaca1c9d18\") " pod="openshift-dns/dns-default-t2dwr" Nov 03 01:51:36.722304 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.722253 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3bc52103-b80b-49a8-9bed-15b196fdef12-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dmrb8\" (UID: \"3bc52103-b80b-49a8-9bed-15b196fdef12\") " pod="openshift-insights/insights-runtime-extractor-dmrb8" Nov 03 01:51:36.722304 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.722290 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/52b546b3-7bf6-4239-91cb-c8aaca1c9d18-metrics-tls\") pod \"dns-default-t2dwr\" (UID: \"52b546b3-7bf6-4239-91cb-c8aaca1c9d18\") " pod="openshift-dns/dns-default-t2dwr" Nov 03 01:51:36.722431 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.722307 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0f363d1e-561d-43ae-9d53-379c86b2a558-cert\") pod \"ingress-canary-2d49s\" (UID: \"0f363d1e-561d-43ae-9d53-379c86b2a558\") " pod="openshift-ingress-canary/ingress-canary-2d49s" Nov 03 01:51:36.776440 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.776416 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk6qs\" (UniqueName: \"kubernetes.io/projected/3bc52103-b80b-49a8-9bed-15b196fdef12-kube-api-access-xk6qs\") pod \"insights-runtime-extractor-dmrb8\" (UID: \"3bc52103-b80b-49a8-9bed-15b196fdef12\") " pod="openshift-insights/insights-runtime-extractor-dmrb8" Nov 03 01:51:36.776560 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.776513 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqvvj\" (UniqueName: \"kubernetes.io/projected/52b546b3-7bf6-4239-91cb-c8aaca1c9d18-kube-api-access-pqvvj\") pod \"dns-default-t2dwr\" (UID: \"52b546b3-7bf6-4239-91cb-c8aaca1c9d18\") " pod="openshift-dns/dns-default-t2dwr" Nov 03 01:51:36.776733 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.776717 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtzct\" (UniqueName: \"kubernetes.io/projected/0f363d1e-561d-43ae-9d53-379c86b2a558-kube-api-access-dtzct\") pod \"ingress-canary-2d49s\" (UID: \"0f363d1e-561d-43ae-9d53-379c86b2a558\") " pod="openshift-ingress-canary/ingress-canary-2d49s" Nov 03 01:51:36.782940 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:36.782921 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dmrb8" Nov 03 01:51:37.070276 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:37.070247 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-t2dwr" Nov 03 01:51:37.073117 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:37.073098 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2d49s" Nov 03 01:51:37.158037 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:37.158006 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dmrb8"] Nov 03 01:51:37.161451 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:51:37.161416 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bc52103_b80b_49a8_9bed_15b196fdef12.slice/crio-9c5979a417e760236a272153330856b889b67f0f680390ecfa5423d5211d9978 WatchSource:0}: Error finding container 9c5979a417e760236a272153330856b889b67f0f680390ecfa5423d5211d9978: Status 404 returned error can't find the container with id 9c5979a417e760236a272153330856b889b67f0f680390ecfa5423d5211d9978 Nov 03 01:51:37.440631 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:37.440557 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-t2dwr"] Nov 03 01:51:37.441140 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:37.441035 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2d49s"] Nov 03 01:51:37.443282 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:51:37.443258 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52b546b3_7bf6_4239_91cb_c8aaca1c9d18.slice/crio-1deb7a08cfcf5ea6b79d48b1886cf9b399b6c4fcc5c2f45bee5545d9a477ec65 WatchSource:0}: Error finding container 1deb7a08cfcf5ea6b79d48b1886cf9b399b6c4fcc5c2f45bee5545d9a477ec65: Status 404 returned error can't find the container with id 1deb7a08cfcf5ea6b79d48b1886cf9b399b6c4fcc5c2f45bee5545d9a477ec65 Nov 03 01:51:37.443808 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:51:37.443789 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f363d1e_561d_43ae_9d53_379c86b2a558.slice/crio-502e7f3026c2bb58bfb48aae412d650bacf07e07855d86ada27a101dd0326349 WatchSource:0}: Error finding container 502e7f3026c2bb58bfb48aae412d650bacf07e07855d86ada27a101dd0326349: Status 404 returned error can't find the container with id 502e7f3026c2bb58bfb48aae412d650bacf07e07855d86ada27a101dd0326349 Nov 03 01:51:37.687677 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:37.687641 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-t2dwr" event={"ID":"52b546b3-7bf6-4239-91cb-c8aaca1c9d18","Type":"ContainerStarted","Data":"1deb7a08cfcf5ea6b79d48b1886cf9b399b6c4fcc5c2f45bee5545d9a477ec65"} Nov 03 01:51:37.689086 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:37.689059 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dmrb8" event={"ID":"3bc52103-b80b-49a8-9bed-15b196fdef12","Type":"ContainerStarted","Data":"b59c1638e89bbdadd046faf36f6b7032ee372f77d4f7d1483cc5713d37b2c66a"} Nov 03 01:51:37.689184 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:37.689115 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dmrb8" event={"ID":"3bc52103-b80b-49a8-9bed-15b196fdef12","Type":"ContainerStarted","Data":"9c5979a417e760236a272153330856b889b67f0f680390ecfa5423d5211d9978"} Nov 03 01:51:37.690122 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:37.690103 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2d49s" event={"ID":"0f363d1e-561d-43ae-9d53-379c86b2a558","Type":"ContainerStarted","Data":"502e7f3026c2bb58bfb48aae412d650bacf07e07855d86ada27a101dd0326349"} Nov 03 01:51:38.695046 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:38.695012 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dmrb8" event={"ID":"3bc52103-b80b-49a8-9bed-15b196fdef12","Type":"ContainerStarted","Data":"4a0e8d41451bc2480d3a6db4022041875e33489cbac4e124a14ffebee1623fa1"} Nov 03 01:51:40.049277 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:40.049246 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-jmj4g"] Nov 03 01:51:40.068013 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:40.067985 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-jmj4g" Nov 03 01:51:40.105877 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:40.105847 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-jmj4g"] Nov 03 01:51:40.155878 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:40.155847 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 03 01:51:40.156125 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:40.156088 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-x6kj8\"" Nov 03 01:51:40.172589 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:40.172566 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 01:51:40.172715 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:40.172597 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 03 01:51:40.172715 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:40.172673 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 01:51:40.206866 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:40.206843 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 01:51:40.246208 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:40.246177 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5d514c70-f1a7-462a-bd3e-6f5bdc016c39-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-jmj4g\" (UID: \"5d514c70-f1a7-462a-bd3e-6f5bdc016c39\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jmj4g" Nov 03 01:51:40.246208 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:40.246212 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5d514c70-f1a7-462a-bd3e-6f5bdc016c39-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-jmj4g\" (UID: \"5d514c70-f1a7-462a-bd3e-6f5bdc016c39\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jmj4g" Nov 03 01:51:40.246475 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:40.246232 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d514c70-f1a7-462a-bd3e-6f5bdc016c39-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-jmj4g\" (UID: \"5d514c70-f1a7-462a-bd3e-6f5bdc016c39\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jmj4g" Nov 03 01:51:40.246475 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:40.246383 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhl6w\" (UniqueName: \"kubernetes.io/projected/5d514c70-f1a7-462a-bd3e-6f5bdc016c39-kube-api-access-dhl6w\") pod \"prometheus-operator-9cd89b74c-jmj4g\" (UID: \"5d514c70-f1a7-462a-bd3e-6f5bdc016c39\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jmj4g" Nov 03 01:51:40.347844 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:40.347763 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5d514c70-f1a7-462a-bd3e-6f5bdc016c39-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-jmj4g\" (UID: \"5d514c70-f1a7-462a-bd3e-6f5bdc016c39\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jmj4g" Nov 03 01:51:40.347844 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:40.347801 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5d514c70-f1a7-462a-bd3e-6f5bdc016c39-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-jmj4g\" (UID: \"5d514c70-f1a7-462a-bd3e-6f5bdc016c39\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jmj4g" Nov 03 01:51:40.347844 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:40.347821 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d514c70-f1a7-462a-bd3e-6f5bdc016c39-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-jmj4g\" (UID: \"5d514c70-f1a7-462a-bd3e-6f5bdc016c39\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jmj4g" Nov 03 01:51:40.348097 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:40.347877 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhl6w\" (UniqueName: \"kubernetes.io/projected/5d514c70-f1a7-462a-bd3e-6f5bdc016c39-kube-api-access-dhl6w\") pod \"prometheus-operator-9cd89b74c-jmj4g\" (UID: \"5d514c70-f1a7-462a-bd3e-6f5bdc016c39\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jmj4g" Nov 03 01:51:40.349126 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:40.349070 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d514c70-f1a7-462a-bd3e-6f5bdc016c39-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-jmj4g\" (UID: \"5d514c70-f1a7-462a-bd3e-6f5bdc016c39\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jmj4g" Nov 03 01:51:40.350903 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:40.350871 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5d514c70-f1a7-462a-bd3e-6f5bdc016c39-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-jmj4g\" (UID: \"5d514c70-f1a7-462a-bd3e-6f5bdc016c39\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jmj4g" Nov 03 01:51:40.351127 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:40.351107 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5d514c70-f1a7-462a-bd3e-6f5bdc016c39-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-jmj4g\" (UID: \"5d514c70-f1a7-462a-bd3e-6f5bdc016c39\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jmj4g" Nov 03 01:51:40.406086 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:40.406046 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhl6w\" (UniqueName: \"kubernetes.io/projected/5d514c70-f1a7-462a-bd3e-6f5bdc016c39-kube-api-access-dhl6w\") pod \"prometheus-operator-9cd89b74c-jmj4g\" (UID: \"5d514c70-f1a7-462a-bd3e-6f5bdc016c39\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jmj4g" Nov 03 01:51:40.408690 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:40.408665 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-jmj4g" Nov 03 01:51:40.801007 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:40.800970 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-jmj4g"] Nov 03 01:51:40.834653 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:51:40.834626 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d514c70_f1a7_462a_bd3e_6f5bdc016c39.slice/crio-bc2201422c99cde1c72025b313b97433d58fce7e26f4b82febbb95752ea56185 WatchSource:0}: Error finding container bc2201422c99cde1c72025b313b97433d58fce7e26f4b82febbb95752ea56185: Status 404 returned error can't find the container with id bc2201422c99cde1c72025b313b97433d58fce7e26f4b82febbb95752ea56185 Nov 03 01:51:41.704441 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:41.704403 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-jmj4g" event={"ID":"5d514c70-f1a7-462a-bd3e-6f5bdc016c39","Type":"ContainerStarted","Data":"bc2201422c99cde1c72025b313b97433d58fce7e26f4b82febbb95752ea56185"} Nov 03 01:51:41.705999 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:41.705979 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-t2dwr" event={"ID":"52b546b3-7bf6-4239-91cb-c8aaca1c9d18","Type":"ContainerStarted","Data":"203ea3c9d2a6167f2886fac0653441c25a0232f3837033f291a4b00db7766f09"} Nov 03 01:51:41.706102 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:41.706023 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-t2dwr" event={"ID":"52b546b3-7bf6-4239-91cb-c8aaca1c9d18","Type":"ContainerStarted","Data":"51f2a7f17a7185a0e9155e822d11edbba8fde671104396c258c7ec3e008a97e5"} Nov 03 01:51:41.706102 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:41.706073 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-t2dwr" Nov 03 01:51:41.707874 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:41.707836 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dmrb8" event={"ID":"3bc52103-b80b-49a8-9bed-15b196fdef12","Type":"ContainerStarted","Data":"f6f2916c24964fe911e595ac88896c43e0a040793f8f75e54969431531f514de"} Nov 03 01:51:41.709010 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:41.708993 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2d49s" event={"ID":"0f363d1e-561d-43ae-9d53-379c86b2a558","Type":"ContainerStarted","Data":"7ebd3336f4f77a3158371b0e23b1fb4d82e171bc4db89fef85c4c1eb0058ac4c"} Nov 03 01:51:41.821502 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:41.821453 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-t2dwr" podStartSLOduration=2.40130794 podStartE2EDuration="5.821439789s" totalImagesPullingTime="3.420131849s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:51:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:51:41.820445141 +0000 UTC m=+53.823892603" watchObservedRunningTime="2026-11-03 01:51:41.821439789 +0000 UTC m=+53.824887252" Nov 03 01:51:41.936865 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:41.936808 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-2d49s" podStartSLOduration=2.089983633 podStartE2EDuration="5.936795213s" totalImagesPullingTime="3.84681158s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:51:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:51:41.936089755 +0000 UTC m=+53.939537218" watchObservedRunningTime="2026-11-03 01:51:41.936795213 +0000 UTC m=+53.940242672" Nov 03 01:51:42.051570 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:42.051518 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-dmrb8" podStartSLOduration=2.505632054 podStartE2EDuration="6.051505193s" totalImagesPullingTime="3.545873139s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:51:36 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:51:42.051018003 +0000 UTC m=+54.054465463" watchObservedRunningTime="2026-11-03 01:51:42.051505193 +0000 UTC m=+54.054952655" Nov 03 01:51:42.347179 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:42.347108 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-59fbbdf587-6nm8q"] Nov 03 01:51:42.349944 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:42.349928 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59fbbdf587-6nm8q" Nov 03 01:51:42.405976 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:42.405943 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 03 01:51:42.406177 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:42.406161 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 03 01:51:42.455633 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:42.455600 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 03 01:51:42.455633 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:42.455600 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 03 01:51:42.455874 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:42.455638 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-t7b59\"" Nov 03 01:51:42.455874 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:42.455653 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 03 01:51:42.455874 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:42.455604 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 03 01:51:42.455874 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:42.455665 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 03 01:51:42.456742 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:42.456721 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59fbbdf587-6nm8q"] Nov 03 01:51:42.465285 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:42.465260 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-tmp\") pod \"console-59fbbdf587-6nm8q\" (UID: \"3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6\") " pod="openshift-console/console-59fbbdf587-6nm8q" Nov 03 01:51:42.465404 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:42.465345 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmhdf\" (UniqueName: \"kubernetes.io/projected/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-kube-api-access-bmhdf\") pod \"console-59fbbdf587-6nm8q\" (UID: \"3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6\") " pod="openshift-console/console-59fbbdf587-6nm8q" Nov 03 01:51:42.465527 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:42.465507 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-console-oauth-config\") pod \"console-59fbbdf587-6nm8q\" (UID: \"3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6\") " pod="openshift-console/console-59fbbdf587-6nm8q" Nov 03 01:51:42.465602 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:42.465585 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-console-config\") pod \"console-59fbbdf587-6nm8q\" (UID: \"3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6\") " pod="openshift-console/console-59fbbdf587-6nm8q" Nov 03 01:51:42.465658 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:42.465608 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-service-ca\") pod \"console-59fbbdf587-6nm8q\" (UID: \"3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6\") " pod="openshift-console/console-59fbbdf587-6nm8q" Nov 03 01:51:42.465658 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:42.465633 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-oauth-serving-cert\") pod \"console-59fbbdf587-6nm8q\" (UID: \"3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6\") " pod="openshift-console/console-59fbbdf587-6nm8q" Nov 03 01:51:42.465778 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:42.465763 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-console-serving-cert\") pod \"console-59fbbdf587-6nm8q\" (UID: \"3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6\") " pod="openshift-console/console-59fbbdf587-6nm8q" Nov 03 01:51:42.567139 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:42.567098 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-console-config\") pod \"console-59fbbdf587-6nm8q\" (UID: \"3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6\") " pod="openshift-console/console-59fbbdf587-6nm8q" Nov 03 01:51:42.567139 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:42.567144 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-service-ca\") pod \"console-59fbbdf587-6nm8q\" (UID: \"3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6\") " pod="openshift-console/console-59fbbdf587-6nm8q" Nov 03 01:51:42.567386 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:42.567167 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-oauth-serving-cert\") pod \"console-59fbbdf587-6nm8q\" (UID: \"3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6\") " pod="openshift-console/console-59fbbdf587-6nm8q" Nov 03 01:51:42.567386 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:42.567217 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-console-serving-cert\") pod \"console-59fbbdf587-6nm8q\" (UID: \"3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6\") " pod="openshift-console/console-59fbbdf587-6nm8q" Nov 03 01:51:42.567386 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:42.567253 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-tmp\") pod \"console-59fbbdf587-6nm8q\" (UID: \"3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6\") " pod="openshift-console/console-59fbbdf587-6nm8q" Nov 03 01:51:42.567386 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:42.567284 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmhdf\" (UniqueName: \"kubernetes.io/projected/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-kube-api-access-bmhdf\") pod \"console-59fbbdf587-6nm8q\" (UID: \"3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6\") " pod="openshift-console/console-59fbbdf587-6nm8q" Nov 03 01:51:42.567386 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:42.567318 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-console-oauth-config\") pod \"console-59fbbdf587-6nm8q\" (UID: \"3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6\") " pod="openshift-console/console-59fbbdf587-6nm8q" Nov 03 01:51:42.567972 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:42.567923 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-tmp\") pod \"console-59fbbdf587-6nm8q\" (UID: \"3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6\") " pod="openshift-console/console-59fbbdf587-6nm8q" Nov 03 01:51:42.567972 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:42.567925 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-console-config\") pod \"console-59fbbdf587-6nm8q\" (UID: \"3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6\") " pod="openshift-console/console-59fbbdf587-6nm8q" Nov 03 01:51:42.568138 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:42.567964 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-service-ca\") pod \"console-59fbbdf587-6nm8q\" (UID: \"3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6\") " pod="openshift-console/console-59fbbdf587-6nm8q" Nov 03 01:51:42.568138 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:42.567995 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-oauth-serving-cert\") pod \"console-59fbbdf587-6nm8q\" (UID: \"3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6\") " pod="openshift-console/console-59fbbdf587-6nm8q" Nov 03 01:51:42.569805 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:42.569777 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-console-serving-cert\") pod \"console-59fbbdf587-6nm8q\" (UID: \"3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6\") " pod="openshift-console/console-59fbbdf587-6nm8q" Nov 03 01:51:42.569928 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:42.569906 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-console-oauth-config\") pod \"console-59fbbdf587-6nm8q\" (UID: \"3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6\") " pod="openshift-console/console-59fbbdf587-6nm8q" Nov 03 01:51:42.625139 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:42.625082 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmhdf\" (UniqueName: \"kubernetes.io/projected/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-kube-api-access-bmhdf\") pod \"console-59fbbdf587-6nm8q\" (UID: \"3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6\") " pod="openshift-console/console-59fbbdf587-6nm8q" Nov 03 01:51:42.660394 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:42.660346 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59fbbdf587-6nm8q" Nov 03 01:51:43.034807 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:43.034734 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59fbbdf587-6nm8q"] Nov 03 01:51:43.067676 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:51:43.067646 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fde4b63_e3dc_4b4e_97a5_ad6dc8fe48f6.slice/crio-6c90f04210adfeedaf7fc82f96407539f7eef4c1ced7c1f75c523b11d27cc513 WatchSource:0}: Error finding container 6c90f04210adfeedaf7fc82f96407539f7eef4c1ced7c1f75c523b11d27cc513: Status 404 returned error can't find the container with id 6c90f04210adfeedaf7fc82f96407539f7eef4c1ced7c1f75c523b11d27cc513 Nov 03 01:51:43.717241 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:43.717199 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59fbbdf587-6nm8q" event={"ID":"3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6","Type":"ContainerStarted","Data":"6c90f04210adfeedaf7fc82f96407539f7eef4c1ced7c1f75c523b11d27cc513"} Nov 03 01:51:43.718817 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:43.718793 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-jmj4g" event={"ID":"5d514c70-f1a7-462a-bd3e-6f5bdc016c39","Type":"ContainerStarted","Data":"5791e42c8810dcd232b2c5ac74b208d1b64ae9859ea31315a921097779cb7cf9"} Nov 03 01:51:43.718945 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:43.718840 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-jmj4g" event={"ID":"5d514c70-f1a7-462a-bd3e-6f5bdc016c39","Type":"ContainerStarted","Data":"0dce5afd6a0d4e2ce4c43b41d2449309df4d07c7f40d917627a0c1d2a1b7dd6e"} Nov 03 01:51:43.834698 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:43.834646 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-jmj4g" podStartSLOduration=1.617679741 podStartE2EDuration="3.834632133s" totalImagesPullingTime="2.216952392s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:51:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:51:43.833670354 +0000 UTC m=+55.837117816" watchObservedRunningTime="2026-11-03 01:51:43.834632133 +0000 UTC m=+55.838079595" Nov 03 01:51:44.651298 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:44.651270 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:51:46.406777 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:46.406744 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2q7kh"] Nov 03 01:51:46.410259 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:46.410223 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2q7kh" Nov 03 01:51:46.463412 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:46.463378 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 01:51:46.463573 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:46.463454 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 01:51:46.500895 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:46.500862 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4480cca6-b2e9-4840-ac8c-a88a40ed90f2-node-exporter-textfile\") pod \"node-exporter-2q7kh\" (UID: \"4480cca6-b2e9-4840-ac8c-a88a40ed90f2\") " pod="openshift-monitoring/node-exporter-2q7kh" Nov 03 01:51:46.501071 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:46.500902 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4480cca6-b2e9-4840-ac8c-a88a40ed90f2-node-exporter-tls\") pod \"node-exporter-2q7kh\" (UID: \"4480cca6-b2e9-4840-ac8c-a88a40ed90f2\") " pod="openshift-monitoring/node-exporter-2q7kh" Nov 03 01:51:46.501071 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:46.501007 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4480cca6-b2e9-4840-ac8c-a88a40ed90f2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2q7kh\" (UID: \"4480cca6-b2e9-4840-ac8c-a88a40ed90f2\") " pod="openshift-monitoring/node-exporter-2q7kh" Nov 03 01:51:46.501071 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:46.501060 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4480cca6-b2e9-4840-ac8c-a88a40ed90f2-metrics-client-ca\") pod \"node-exporter-2q7kh\" (UID: \"4480cca6-b2e9-4840-ac8c-a88a40ed90f2\") " pod="openshift-monitoring/node-exporter-2q7kh" Nov 03 01:51:46.501226 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:46.501091 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4480cca6-b2e9-4840-ac8c-a88a40ed90f2-root\") pod \"node-exporter-2q7kh\" (UID: \"4480cca6-b2e9-4840-ac8c-a88a40ed90f2\") " pod="openshift-monitoring/node-exporter-2q7kh" Nov 03 01:51:46.501226 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:46.501127 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4480cca6-b2e9-4840-ac8c-a88a40ed90f2-sys\") pod \"node-exporter-2q7kh\" (UID: \"4480cca6-b2e9-4840-ac8c-a88a40ed90f2\") " pod="openshift-monitoring/node-exporter-2q7kh" Nov 03 01:51:46.501226 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:46.501166 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4480cca6-b2e9-4840-ac8c-a88a40ed90f2-node-exporter-wtmp\") pod \"node-exporter-2q7kh\" (UID: \"4480cca6-b2e9-4840-ac8c-a88a40ed90f2\") " pod="openshift-monitoring/node-exporter-2q7kh" Nov 03 01:51:46.501226 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:46.501194 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4480cca6-b2e9-4840-ac8c-a88a40ed90f2-node-exporter-accelerators-collector-config\") pod \"node-exporter-2q7kh\" (UID: \"4480cca6-b2e9-4840-ac8c-a88a40ed90f2\") " pod="openshift-monitoring/node-exporter-2q7kh" Nov 03 01:51:46.501226 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:46.501220 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkn5x\" (UniqueName: \"kubernetes.io/projected/4480cca6-b2e9-4840-ac8c-a88a40ed90f2-kube-api-access-bkn5x\") pod \"node-exporter-2q7kh\" (UID: \"4480cca6-b2e9-4840-ac8c-a88a40ed90f2\") " pod="openshift-monitoring/node-exporter-2q7kh" Nov 03 01:51:46.514227 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:46.514192 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-8rp9w\"" Nov 03 01:51:46.544385 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:46.544338 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 01:51:46.601673 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:46.601640 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4480cca6-b2e9-4840-ac8c-a88a40ed90f2-node-exporter-wtmp\") pod \"node-exporter-2q7kh\" (UID: \"4480cca6-b2e9-4840-ac8c-a88a40ed90f2\") " pod="openshift-monitoring/node-exporter-2q7kh" Nov 03 01:51:46.601842 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:46.601682 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4480cca6-b2e9-4840-ac8c-a88a40ed90f2-node-exporter-accelerators-collector-config\") pod \"node-exporter-2q7kh\" (UID: \"4480cca6-b2e9-4840-ac8c-a88a40ed90f2\") " pod="openshift-monitoring/node-exporter-2q7kh" Nov 03 01:51:46.601842 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:46.601709 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkn5x\" (UniqueName: \"kubernetes.io/projected/4480cca6-b2e9-4840-ac8c-a88a40ed90f2-kube-api-access-bkn5x\") pod \"node-exporter-2q7kh\" (UID: \"4480cca6-b2e9-4840-ac8c-a88a40ed90f2\") " pod="openshift-monitoring/node-exporter-2q7kh" Nov 03 01:51:46.601842 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:46.601747 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4480cca6-b2e9-4840-ac8c-a88a40ed90f2-node-exporter-textfile\") pod \"node-exporter-2q7kh\" (UID: \"4480cca6-b2e9-4840-ac8c-a88a40ed90f2\") " pod="openshift-monitoring/node-exporter-2q7kh" Nov 03 01:51:46.601842 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:46.601766 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4480cca6-b2e9-4840-ac8c-a88a40ed90f2-node-exporter-tls\") pod \"node-exporter-2q7kh\" (UID: \"4480cca6-b2e9-4840-ac8c-a88a40ed90f2\") " pod="openshift-monitoring/node-exporter-2q7kh" Nov 03 01:51:46.601842 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:46.601809 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4480cca6-b2e9-4840-ac8c-a88a40ed90f2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2q7kh\" (UID: \"4480cca6-b2e9-4840-ac8c-a88a40ed90f2\") " pod="openshift-monitoring/node-exporter-2q7kh" Nov 03 01:51:46.601842 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:46.601843 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4480cca6-b2e9-4840-ac8c-a88a40ed90f2-metrics-client-ca\") pod \"node-exporter-2q7kh\" (UID: \"4480cca6-b2e9-4840-ac8c-a88a40ed90f2\") " pod="openshift-monitoring/node-exporter-2q7kh" Nov 03 01:51:46.602117 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:46.601874 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4480cca6-b2e9-4840-ac8c-a88a40ed90f2-node-exporter-wtmp\") pod \"node-exporter-2q7kh\" (UID: \"4480cca6-b2e9-4840-ac8c-a88a40ed90f2\") " pod="openshift-monitoring/node-exporter-2q7kh" Nov 03 01:51:46.602117 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:46.602073 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4480cca6-b2e9-4840-ac8c-a88a40ed90f2-node-exporter-textfile\") pod \"node-exporter-2q7kh\" (UID: \"4480cca6-b2e9-4840-ac8c-a88a40ed90f2\") " pod="openshift-monitoring/node-exporter-2q7kh" Nov 03 01:51:46.602217 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:46.602120 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4480cca6-b2e9-4840-ac8c-a88a40ed90f2-root\") pod \"node-exporter-2q7kh\" (UID: \"4480cca6-b2e9-4840-ac8c-a88a40ed90f2\") " pod="openshift-monitoring/node-exporter-2q7kh" Nov 03 01:51:46.602217 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:46.602164 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4480cca6-b2e9-4840-ac8c-a88a40ed90f2-sys\") pod \"node-exporter-2q7kh\" (UID: \"4480cca6-b2e9-4840-ac8c-a88a40ed90f2\") " pod="openshift-monitoring/node-exporter-2q7kh" Nov 03 01:51:46.602319 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:46.602239 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4480cca6-b2e9-4840-ac8c-a88a40ed90f2-sys\") pod \"node-exporter-2q7kh\" (UID: \"4480cca6-b2e9-4840-ac8c-a88a40ed90f2\") " pod="openshift-monitoring/node-exporter-2q7kh" Nov 03 01:51:46.602319 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:46.602255 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4480cca6-b2e9-4840-ac8c-a88a40ed90f2-root\") pod \"node-exporter-2q7kh\" (UID: \"4480cca6-b2e9-4840-ac8c-a88a40ed90f2\") " pod="openshift-monitoring/node-exporter-2q7kh" Nov 03 01:51:46.602447 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:46.602373 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4480cca6-b2e9-4840-ac8c-a88a40ed90f2-metrics-client-ca\") pod \"node-exporter-2q7kh\" (UID: \"4480cca6-b2e9-4840-ac8c-a88a40ed90f2\") " pod="openshift-monitoring/node-exporter-2q7kh" Nov 03 01:51:46.602498 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:46.602457 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4480cca6-b2e9-4840-ac8c-a88a40ed90f2-node-exporter-accelerators-collector-config\") pod \"node-exporter-2q7kh\" (UID: \"4480cca6-b2e9-4840-ac8c-a88a40ed90f2\") " pod="openshift-monitoring/node-exporter-2q7kh" Nov 03 01:51:46.611432 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:46.604338 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4480cca6-b2e9-4840-ac8c-a88a40ed90f2-node-exporter-tls\") pod \"node-exporter-2q7kh\" (UID: \"4480cca6-b2e9-4840-ac8c-a88a40ed90f2\") " pod="openshift-monitoring/node-exporter-2q7kh" Nov 03 01:51:46.611432 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:46.604488 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4480cca6-b2e9-4840-ac8c-a88a40ed90f2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2q7kh\" (UID: \"4480cca6-b2e9-4840-ac8c-a88a40ed90f2\") " pod="openshift-monitoring/node-exporter-2q7kh" Nov 03 01:51:46.661942 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:46.661861 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkn5x\" (UniqueName: \"kubernetes.io/projected/4480cca6-b2e9-4840-ac8c-a88a40ed90f2-kube-api-access-bkn5x\") pod \"node-exporter-2q7kh\" (UID: \"4480cca6-b2e9-4840-ac8c-a88a40ed90f2\") " pod="openshift-monitoring/node-exporter-2q7kh" Nov 03 01:51:46.729039 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:46.729008 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2q7kh" Nov 03 01:51:46.747496 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:51:46.747455 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4480cca6_b2e9_4840_ac8c_a88a40ed90f2.slice/crio-d8aa584797783a6981d1de93f65e4b22f1e462cdccc7c73acaa94ca5530a26f4 WatchSource:0}: Error finding container d8aa584797783a6981d1de93f65e4b22f1e462cdccc7c73acaa94ca5530a26f4: Status 404 returned error can't find the container with id d8aa584797783a6981d1de93f65e4b22f1e462cdccc7c73acaa94ca5530a26f4 Nov 03 01:51:47.733559 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:47.733516 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59fbbdf587-6nm8q" event={"ID":"3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6","Type":"ContainerStarted","Data":"e7e44cb46e26b5bbb607e7a166a8f0c8378543626fe3447c00a07938b800f84a"} Nov 03 01:51:47.734714 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:47.734691 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2q7kh" event={"ID":"4480cca6-b2e9-4840-ac8c-a88a40ed90f2","Type":"ContainerStarted","Data":"d8aa584797783a6981d1de93f65e4b22f1e462cdccc7c73acaa94ca5530a26f4"} Nov 03 01:51:47.849851 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:47.849801 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-59fbbdf587-6nm8q" podStartSLOduration=1.972439472 podStartE2EDuration="5.849789269s" totalImagesPullingTime="3.877349797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:51:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:51:47.848755593 +0000 UTC m=+59.852203055" watchObservedRunningTime="2026-11-03 01:51:47.849789269 +0000 UTC m=+59.853236731" Nov 03 01:51:48.632885 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:48.632855 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 01:51:48.636057 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:48.636041 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:51:48.672324 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:48.672297 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 01:51:48.690073 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:48.690051 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 03 01:51:48.693835 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:48.693808 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 03 01:51:48.693930 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:48.693886 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 03 01:51:48.693930 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:48.693888 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 03 01:51:48.693930 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:48.693900 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 03 01:51:48.693930 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:48.693921 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 03 01:51:48.694130 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:48.693924 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-rvjqm\"" Nov 03 01:51:48.694130 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:48.693928 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 03 01:51:48.695523 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:48.695506 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 03 01:51:48.716446 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:48.716390 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d7d07f7-27c2-489a-8e44-fe366584a267-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:51:48.716446 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:48.716420 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:51:48.716446 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:48.716436 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-config-volume\") pod \"alertmanager-main-0\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:51:48.716626 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:48.716455 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3d7d07f7-27c2-489a-8e44-fe366584a267-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:51:48.716626 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:48.716553 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3d7d07f7-27c2-489a-8e44-fe366584a267-config-out\") pod \"alertmanager-main-0\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:51:48.716713 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:48.716635 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3d7d07f7-27c2-489a-8e44-fe366584a267-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:51:48.716713 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:48.716689 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:51:48.716812 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:48.716720 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d7d07f7-27c2-489a-8e44-fe366584a267-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:51:48.716812 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:48.716776 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:51:48.716877 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:48.716864 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:51:48.716914 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:48.716891 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-web-config\") pod \"alertmanager-main-0\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:51:48.716948 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:48.716929 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:51:48.716986 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:48.716950 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl9gt\" (UniqueName: \"kubernetes.io/projected/3d7d07f7-27c2-489a-8e44-fe366584a267-kube-api-access-nl9gt\") pod \"alertmanager-main-0\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:51:48.739237 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:48.739208 2579 generic.go:356] "Generic (PLEG): container finished" podID="4480cca6-b2e9-4840-ac8c-a88a40ed90f2" containerID="6de7e466197c3677347a4b99ad4c74967961b793939e0e414ccec863c44b0ce0" exitCode=0 Nov 03 01:51:48.739628 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:48.739236 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2q7kh" event={"ID":"4480cca6-b2e9-4840-ac8c-a88a40ed90f2","Type":"ContainerDied","Data":"6de7e466197c3677347a4b99ad4c74967961b793939e0e414ccec863c44b0ce0"} Nov 03 01:51:48.793045 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:48.793022 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 03 01:51:48.818187 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:48.818163 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:51:48.818291 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:48.818207 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d7d07f7-27c2-489a-8e44-fe366584a267-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:51:48.818291 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:48.818258 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:51:48.818423 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:48.818330 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:51:48.818423 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:48.818406 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-web-config\") pod \"alertmanager-main-0\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:51:48.818518 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:48.818444 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:51:48.818518 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:48.818467 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl9gt\" (UniqueName: \"kubernetes.io/projected/3d7d07f7-27c2-489a-8e44-fe366584a267-kube-api-access-nl9gt\") pod \"alertmanager-main-0\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:51:48.818604 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:48.818519 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d7d07f7-27c2-489a-8e44-fe366584a267-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:51:48.818604 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:48.818547 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:51:48.818604 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:48.818578 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-config-volume\") pod \"alertmanager-main-0\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:51:48.818785 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:48.818762 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3d7d07f7-27c2-489a-8e44-fe366584a267-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:51:48.818859 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:48.818843 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3d7d07f7-27c2-489a-8e44-fe366584a267-config-out\") pod \"alertmanager-main-0\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:51:48.818920 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:48.818905 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3d7d07f7-27c2-489a-8e44-fe366584a267-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:51:48.818955 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:48.818945 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d7d07f7-27c2-489a-8e44-fe366584a267-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:51:48.819563 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:48.819521 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3d7d07f7-27c2-489a-8e44-fe366584a267-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:51:48.821310 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:48.821278 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d7d07f7-27c2-489a-8e44-fe366584a267-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:51:48.821791 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:48.821765 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-web-config\") pod \"alertmanager-main-0\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:51:48.821887 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:48.821820 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3d7d07f7-27c2-489a-8e44-fe366584a267-config-out\") pod \"alertmanager-main-0\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:51:48.821947 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:48.821919 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-config-volume\") pod \"alertmanager-main-0\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:51:48.821994 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:48.821954 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:51:48.822116 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:48.822090 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:51:48.822510 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:48.822491 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3d7d07f7-27c2-489a-8e44-fe366584a267-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:51:48.822677 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:48.822657 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:51:48.823588 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:48.823572 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:51:48.823763 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:48.823748 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:51:48.875594 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:48.875569 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl9gt\" (UniqueName: \"kubernetes.io/projected/3d7d07f7-27c2-489a-8e44-fe366584a267-kube-api-access-nl9gt\") pod \"alertmanager-main-0\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:51:48.972970 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:48.972890 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:51:49.392863 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:49.392840 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 01:51:49.743973 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:49.743880 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2q7kh" event={"ID":"4480cca6-b2e9-4840-ac8c-a88a40ed90f2","Type":"ContainerStarted","Data":"b3e372f57ce3eb0e6b4671d0af95cc05da771da38e473eb3171008b91e4f01e9"} Nov 03 01:51:49.743973 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:49.743940 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2q7kh" event={"ID":"4480cca6-b2e9-4840-ac8c-a88a40ed90f2","Type":"ContainerStarted","Data":"4b01ba18c69420154d1a6ca48c773c56d929b183228778a857616b597ca5e4c3"} Nov 03 01:51:49.745132 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:49.745107 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3d7d07f7-27c2-489a-8e44-fe366584a267","Type":"ContainerStarted","Data":"e6b490f1544909557410be518cb9d934c6f2996cede5301488d9934ab887b297"} Nov 03 01:51:51.355556 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:51.355513 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2q7kh" podStartSLOduration=3.59047947 podStartE2EDuration="5.355498536s" totalImagesPullingTime="1.699319053s" totalInitContainerRuntime="65.700013ms" isStatefulPod=true podCreationTimestamp="2026-11-03 01:51:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:51:49.860493844 +0000 UTC m=+61.863941306" watchObservedRunningTime="2026-11-03 01:51:51.355498536 +0000 UTC m=+63.358946337" Nov 03 01:51:51.355834 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:51.355809 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-b984f6b6b-g6f2h"] Nov 03 01:51:51.359980 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:51.359963 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-b984f6b6b-g6f2h" Nov 03 01:51:51.416635 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:51.416607 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 03 01:51:51.430515 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:51.430481 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-s796x\"" Nov 03 01:51:51.430613 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:51.430559 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 03 01:51:51.438997 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:51.438973 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e32389d5-705f-4202-98ea-6fc6a5ad5d54-secret-metrics-server-tls\") pod \"metrics-server-b984f6b6b-g6f2h\" (UID: \"e32389d5-705f-4202-98ea-6fc6a5ad5d54\") " pod="openshift-monitoring/metrics-server-b984f6b6b-g6f2h" Nov 03 01:51:51.439084 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:51.439003 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv9fz\" (UniqueName: \"kubernetes.io/projected/e32389d5-705f-4202-98ea-6fc6a5ad5d54-kube-api-access-kv9fz\") pod \"metrics-server-b984f6b6b-g6f2h\" (UID: \"e32389d5-705f-4202-98ea-6fc6a5ad5d54\") " pod="openshift-monitoring/metrics-server-b984f6b6b-g6f2h" Nov 03 01:51:51.439084 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:51.439030 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e32389d5-705f-4202-98ea-6fc6a5ad5d54-secret-metrics-server-client-certs\") pod \"metrics-server-b984f6b6b-g6f2h\" (UID: \"e32389d5-705f-4202-98ea-6fc6a5ad5d54\") " pod="openshift-monitoring/metrics-server-b984f6b6b-g6f2h" Nov 03 01:51:51.439084 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:51.439045 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e32389d5-705f-4202-98ea-6fc6a5ad5d54-client-ca-bundle\") pod \"metrics-server-b984f6b6b-g6f2h\" (UID: \"e32389d5-705f-4202-98ea-6fc6a5ad5d54\") " pod="openshift-monitoring/metrics-server-b984f6b6b-g6f2h" Nov 03 01:51:51.439190 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:51.439119 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e32389d5-705f-4202-98ea-6fc6a5ad5d54-audit-log\") pod \"metrics-server-b984f6b6b-g6f2h\" (UID: \"e32389d5-705f-4202-98ea-6fc6a5ad5d54\") " pod="openshift-monitoring/metrics-server-b984f6b6b-g6f2h" Nov 03 01:51:51.439190 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:51.439153 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e32389d5-705f-4202-98ea-6fc6a5ad5d54-metrics-server-audit-profiles\") pod \"metrics-server-b984f6b6b-g6f2h\" (UID: \"e32389d5-705f-4202-98ea-6fc6a5ad5d54\") " pod="openshift-monitoring/metrics-server-b984f6b6b-g6f2h" Nov 03 01:51:51.439258 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:51.439239 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e32389d5-705f-4202-98ea-6fc6a5ad5d54-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-b984f6b6b-g6f2h\" (UID: \"e32389d5-705f-4202-98ea-6fc6a5ad5d54\") " pod="openshift-monitoring/metrics-server-b984f6b6b-g6f2h" Nov 03 01:51:51.467544 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:51.467519 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 03 01:51:51.467544 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:51.467541 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-5gb0tdoj89a63\"" Nov 03 01:51:51.467706 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:51.467639 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 03 01:51:51.468511 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:51.468494 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-b984f6b6b-g6f2h"] Nov 03 01:51:51.539702 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:51.539674 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e32389d5-705f-4202-98ea-6fc6a5ad5d54-secret-metrics-server-tls\") pod \"metrics-server-b984f6b6b-g6f2h\" (UID: \"e32389d5-705f-4202-98ea-6fc6a5ad5d54\") " pod="openshift-monitoring/metrics-server-b984f6b6b-g6f2h" Nov 03 01:51:51.539863 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:51.539707 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv9fz\" (UniqueName: \"kubernetes.io/projected/e32389d5-705f-4202-98ea-6fc6a5ad5d54-kube-api-access-kv9fz\") pod \"metrics-server-b984f6b6b-g6f2h\" (UID: \"e32389d5-705f-4202-98ea-6fc6a5ad5d54\") " pod="openshift-monitoring/metrics-server-b984f6b6b-g6f2h" Nov 03 01:51:51.539863 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:51.539733 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e32389d5-705f-4202-98ea-6fc6a5ad5d54-secret-metrics-server-client-certs\") pod \"metrics-server-b984f6b6b-g6f2h\" (UID: \"e32389d5-705f-4202-98ea-6fc6a5ad5d54\") " pod="openshift-monitoring/metrics-server-b984f6b6b-g6f2h" Nov 03 01:51:51.539863 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:51.539748 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e32389d5-705f-4202-98ea-6fc6a5ad5d54-client-ca-bundle\") pod \"metrics-server-b984f6b6b-g6f2h\" (UID: \"e32389d5-705f-4202-98ea-6fc6a5ad5d54\") " pod="openshift-monitoring/metrics-server-b984f6b6b-g6f2h" Nov 03 01:51:51.540026 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:51.539868 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e32389d5-705f-4202-98ea-6fc6a5ad5d54-audit-log\") pod \"metrics-server-b984f6b6b-g6f2h\" (UID: \"e32389d5-705f-4202-98ea-6fc6a5ad5d54\") " pod="openshift-monitoring/metrics-server-b984f6b6b-g6f2h" Nov 03 01:51:51.540026 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:51.539901 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e32389d5-705f-4202-98ea-6fc6a5ad5d54-metrics-server-audit-profiles\") pod \"metrics-server-b984f6b6b-g6f2h\" (UID: \"e32389d5-705f-4202-98ea-6fc6a5ad5d54\") " pod="openshift-monitoring/metrics-server-b984f6b6b-g6f2h" Nov 03 01:51:51.540026 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:51.539965 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e32389d5-705f-4202-98ea-6fc6a5ad5d54-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-b984f6b6b-g6f2h\" (UID: \"e32389d5-705f-4202-98ea-6fc6a5ad5d54\") " pod="openshift-monitoring/metrics-server-b984f6b6b-g6f2h" Nov 03 01:51:51.540276 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:51.540255 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e32389d5-705f-4202-98ea-6fc6a5ad5d54-audit-log\") pod \"metrics-server-b984f6b6b-g6f2h\" (UID: \"e32389d5-705f-4202-98ea-6fc6a5ad5d54\") " pod="openshift-monitoring/metrics-server-b984f6b6b-g6f2h" Nov 03 01:51:51.540640 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:51.540621 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e32389d5-705f-4202-98ea-6fc6a5ad5d54-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-b984f6b6b-g6f2h\" (UID: \"e32389d5-705f-4202-98ea-6fc6a5ad5d54\") " pod="openshift-monitoring/metrics-server-b984f6b6b-g6f2h" Nov 03 01:51:51.541016 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:51.540992 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e32389d5-705f-4202-98ea-6fc6a5ad5d54-metrics-server-audit-profiles\") pod \"metrics-server-b984f6b6b-g6f2h\" (UID: \"e32389d5-705f-4202-98ea-6fc6a5ad5d54\") " pod="openshift-monitoring/metrics-server-b984f6b6b-g6f2h" Nov 03 01:51:51.542119 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:51.542099 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e32389d5-705f-4202-98ea-6fc6a5ad5d54-secret-metrics-server-client-certs\") pod \"metrics-server-b984f6b6b-g6f2h\" (UID: \"e32389d5-705f-4202-98ea-6fc6a5ad5d54\") " pod="openshift-monitoring/metrics-server-b984f6b6b-g6f2h" Nov 03 01:51:51.542385 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:51.542345 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e32389d5-705f-4202-98ea-6fc6a5ad5d54-secret-metrics-server-tls\") pod \"metrics-server-b984f6b6b-g6f2h\" (UID: \"e32389d5-705f-4202-98ea-6fc6a5ad5d54\") " pod="openshift-monitoring/metrics-server-b984f6b6b-g6f2h" Nov 03 01:51:51.542432 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:51.542380 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e32389d5-705f-4202-98ea-6fc6a5ad5d54-client-ca-bundle\") pod \"metrics-server-b984f6b6b-g6f2h\" (UID: \"e32389d5-705f-4202-98ea-6fc6a5ad5d54\") " pod="openshift-monitoring/metrics-server-b984f6b6b-g6f2h" Nov 03 01:51:51.596975 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:51.596912 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv9fz\" (UniqueName: \"kubernetes.io/projected/e32389d5-705f-4202-98ea-6fc6a5ad5d54-kube-api-access-kv9fz\") pod \"metrics-server-b984f6b6b-g6f2h\" (UID: \"e32389d5-705f-4202-98ea-6fc6a5ad5d54\") " pod="openshift-monitoring/metrics-server-b984f6b6b-g6f2h" Nov 03 01:51:51.672625 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:51.672596 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-b984f6b6b-g6f2h" Nov 03 01:51:51.715089 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:51.715061 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-t2dwr" Nov 03 01:51:51.751698 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:51.751671 2579 generic.go:356] "Generic (PLEG): container finished" podID="3d7d07f7-27c2-489a-8e44-fe366584a267" containerID="0d9d5fbaa013bb1c1bc86e40f772c92f6173c111758e1557c9ffb7578af08cf3" exitCode=0 Nov 03 01:51:51.751820 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:51.751757 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3d7d07f7-27c2-489a-8e44-fe366584a267","Type":"ContainerDied","Data":"0d9d5fbaa013bb1c1bc86e40f772c92f6173c111758e1557c9ffb7578af08cf3"} Nov 03 01:51:52.015862 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:52.015774 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-699d68669f-9jg2q"] Nov 03 01:51:52.021336 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:52.021312 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-699d68669f-9jg2q" Nov 03 01:51:52.043704 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:52.043680 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/923b90de-5e45-4add-b092-91af16b7dbad-telemeter-client-tls\") pod \"telemeter-client-699d68669f-9jg2q\" (UID: \"923b90de-5e45-4add-b092-91af16b7dbad\") " pod="openshift-monitoring/telemeter-client-699d68669f-9jg2q" Nov 03 01:51:52.043837 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:52.043715 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/923b90de-5e45-4add-b092-91af16b7dbad-federate-client-tls\") pod \"telemeter-client-699d68669f-9jg2q\" (UID: \"923b90de-5e45-4add-b092-91af16b7dbad\") " pod="openshift-monitoring/telemeter-client-699d68669f-9jg2q" Nov 03 01:51:52.043837 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:52.043745 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/923b90de-5e45-4add-b092-91af16b7dbad-serving-certs-ca-bundle\") pod \"telemeter-client-699d68669f-9jg2q\" (UID: \"923b90de-5e45-4add-b092-91af16b7dbad\") " pod="openshift-monitoring/telemeter-client-699d68669f-9jg2q" Nov 03 01:51:52.043837 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:52.043762 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/923b90de-5e45-4add-b092-91af16b7dbad-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-699d68669f-9jg2q\" (UID: \"923b90de-5e45-4add-b092-91af16b7dbad\") " pod="openshift-monitoring/telemeter-client-699d68669f-9jg2q" Nov 03 01:51:52.043837 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:52.043821 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l75kg\" (UniqueName: \"kubernetes.io/projected/923b90de-5e45-4add-b092-91af16b7dbad-kube-api-access-l75kg\") pod \"telemeter-client-699d68669f-9jg2q\" (UID: \"923b90de-5e45-4add-b092-91af16b7dbad\") " pod="openshift-monitoring/telemeter-client-699d68669f-9jg2q" Nov 03 01:51:52.044005 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:52.043929 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/923b90de-5e45-4add-b092-91af16b7dbad-secret-telemeter-client\") pod \"telemeter-client-699d68669f-9jg2q\" (UID: \"923b90de-5e45-4add-b092-91af16b7dbad\") " pod="openshift-monitoring/telemeter-client-699d68669f-9jg2q" Nov 03 01:51:52.044046 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:52.044008 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/923b90de-5e45-4add-b092-91af16b7dbad-telemeter-trusted-ca-bundle\") pod \"telemeter-client-699d68669f-9jg2q\" (UID: \"923b90de-5e45-4add-b092-91af16b7dbad\") " pod="openshift-monitoring/telemeter-client-699d68669f-9jg2q" Nov 03 01:51:52.044046 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:52.044036 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/923b90de-5e45-4add-b092-91af16b7dbad-metrics-client-ca\") pod \"telemeter-client-699d68669f-9jg2q\" (UID: \"923b90de-5e45-4add-b092-91af16b7dbad\") " pod="openshift-monitoring/telemeter-client-699d68669f-9jg2q" Nov 03 01:51:52.073626 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:52.073597 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-699d68669f-9jg2q"] Nov 03 01:51:52.074817 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:52.074794 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 03 01:51:52.126062 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:52.126033 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 03 01:51:52.127050 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:52.127028 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-b984f6b6b-g6f2h"] Nov 03 01:51:52.127226 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:51:52.127206 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode32389d5_705f_4202_98ea_6fc6a5ad5d54.slice/crio-4f7e8ae3dd363ba52fbf24a93910e9605ace951fbbdf6e51650ffac0aafe7a77 WatchSource:0}: Error finding container 4f7e8ae3dd363ba52fbf24a93910e9605ace951fbbdf6e51650ffac0aafe7a77: Status 404 returned error can't find the container with id 4f7e8ae3dd363ba52fbf24a93910e9605ace951fbbdf6e51650ffac0aafe7a77 Nov 03 01:51:52.129522 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:52.129503 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 03 01:51:52.129600 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:52.129528 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 03 01:51:52.129600 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:52.129545 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 03 01:51:52.129600 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:52.129564 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 03 01:51:52.129703 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:52.129634 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-25h2j\"" Nov 03 01:51:52.145012 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:52.144991 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/923b90de-5e45-4add-b092-91af16b7dbad-serving-certs-ca-bundle\") pod \"telemeter-client-699d68669f-9jg2q\" (UID: \"923b90de-5e45-4add-b092-91af16b7dbad\") " pod="openshift-monitoring/telemeter-client-699d68669f-9jg2q" Nov 03 01:51:52.145116 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:52.145018 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/923b90de-5e45-4add-b092-91af16b7dbad-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-699d68669f-9jg2q\" (UID: \"923b90de-5e45-4add-b092-91af16b7dbad\") " pod="openshift-monitoring/telemeter-client-699d68669f-9jg2q" Nov 03 01:51:52.145116 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:52.145035 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l75kg\" (UniqueName: \"kubernetes.io/projected/923b90de-5e45-4add-b092-91af16b7dbad-kube-api-access-l75kg\") pod \"telemeter-client-699d68669f-9jg2q\" (UID: \"923b90de-5e45-4add-b092-91af16b7dbad\") " pod="openshift-monitoring/telemeter-client-699d68669f-9jg2q" Nov 03 01:51:52.145250 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:52.145219 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/923b90de-5e45-4add-b092-91af16b7dbad-secret-telemeter-client\") pod \"telemeter-client-699d68669f-9jg2q\" (UID: \"923b90de-5e45-4add-b092-91af16b7dbad\") " pod="openshift-monitoring/telemeter-client-699d68669f-9jg2q" Nov 03 01:51:52.145380 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:52.145278 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/923b90de-5e45-4add-b092-91af16b7dbad-telemeter-trusted-ca-bundle\") pod \"telemeter-client-699d68669f-9jg2q\" (UID: \"923b90de-5e45-4add-b092-91af16b7dbad\") " pod="openshift-monitoring/telemeter-client-699d68669f-9jg2q" Nov 03 01:51:52.145380 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:52.145328 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/923b90de-5e45-4add-b092-91af16b7dbad-metrics-client-ca\") pod \"telemeter-client-699d68669f-9jg2q\" (UID: \"923b90de-5e45-4add-b092-91af16b7dbad\") " pod="openshift-monitoring/telemeter-client-699d68669f-9jg2q" Nov 03 01:51:52.145497 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:52.145444 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/923b90de-5e45-4add-b092-91af16b7dbad-telemeter-client-tls\") pod \"telemeter-client-699d68669f-9jg2q\" (UID: \"923b90de-5e45-4add-b092-91af16b7dbad\") " pod="openshift-monitoring/telemeter-client-699d68669f-9jg2q" Nov 03 01:51:52.145497 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:52.145487 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/923b90de-5e45-4add-b092-91af16b7dbad-federate-client-tls\") pod \"telemeter-client-699d68669f-9jg2q\" (UID: \"923b90de-5e45-4add-b092-91af16b7dbad\") " pod="openshift-monitoring/telemeter-client-699d68669f-9jg2q" Nov 03 01:51:52.145901 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:52.145878 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/923b90de-5e45-4add-b092-91af16b7dbad-serving-certs-ca-bundle\") pod \"telemeter-client-699d68669f-9jg2q\" (UID: \"923b90de-5e45-4add-b092-91af16b7dbad\") " pod="openshift-monitoring/telemeter-client-699d68669f-9jg2q" Nov 03 01:51:52.146404 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:52.146239 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/923b90de-5e45-4add-b092-91af16b7dbad-metrics-client-ca\") pod \"telemeter-client-699d68669f-9jg2q\" (UID: \"923b90de-5e45-4add-b092-91af16b7dbad\") " pod="openshift-monitoring/telemeter-client-699d68669f-9jg2q" Nov 03 01:51:52.146404 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:52.146269 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/923b90de-5e45-4add-b092-91af16b7dbad-telemeter-trusted-ca-bundle\") pod \"telemeter-client-699d68669f-9jg2q\" (UID: \"923b90de-5e45-4add-b092-91af16b7dbad\") " pod="openshift-monitoring/telemeter-client-699d68669f-9jg2q" Nov 03 01:51:52.147583 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:52.147564 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/923b90de-5e45-4add-b092-91af16b7dbad-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-699d68669f-9jg2q\" (UID: \"923b90de-5e45-4add-b092-91af16b7dbad\") " pod="openshift-monitoring/telemeter-client-699d68669f-9jg2q" Nov 03 01:51:52.147763 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:52.147743 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/923b90de-5e45-4add-b092-91af16b7dbad-federate-client-tls\") pod \"telemeter-client-699d68669f-9jg2q\" (UID: \"923b90de-5e45-4add-b092-91af16b7dbad\") " pod="openshift-monitoring/telemeter-client-699d68669f-9jg2q" Nov 03 01:51:52.148013 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:52.147964 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/923b90de-5e45-4add-b092-91af16b7dbad-secret-telemeter-client\") pod \"telemeter-client-699d68669f-9jg2q\" (UID: \"923b90de-5e45-4add-b092-91af16b7dbad\") " pod="openshift-monitoring/telemeter-client-699d68669f-9jg2q" Nov 03 01:51:52.148079 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:52.148015 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/923b90de-5e45-4add-b092-91af16b7dbad-telemeter-client-tls\") pod \"telemeter-client-699d68669f-9jg2q\" (UID: \"923b90de-5e45-4add-b092-91af16b7dbad\") " pod="openshift-monitoring/telemeter-client-699d68669f-9jg2q" Nov 03 01:51:52.202299 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:52.202266 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l75kg\" (UniqueName: \"kubernetes.io/projected/923b90de-5e45-4add-b092-91af16b7dbad-kube-api-access-l75kg\") pod \"telemeter-client-699d68669f-9jg2q\" (UID: \"923b90de-5e45-4add-b092-91af16b7dbad\") " pod="openshift-monitoring/telemeter-client-699d68669f-9jg2q" Nov 03 01:51:52.336714 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:52.336681 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-699d68669f-9jg2q" Nov 03 01:51:52.661026 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:52.660953 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-59fbbdf587-6nm8q" Nov 03 01:51:52.661379 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:52.661024 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-59fbbdf587-6nm8q" Nov 03 01:51:52.665548 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:52.665526 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-59fbbdf587-6nm8q" Nov 03 01:51:52.755780 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:52.755737 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-b984f6b6b-g6f2h" event={"ID":"e32389d5-705f-4202-98ea-6fc6a5ad5d54","Type":"ContainerStarted","Data":"4f7e8ae3dd363ba52fbf24a93910e9605ace951fbbdf6e51650ffac0aafe7a77"} Nov 03 01:51:52.757519 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:52.757487 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-699d68669f-9jg2q"] Nov 03 01:51:52.759699 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:52.759674 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-59fbbdf587-6nm8q" Nov 03 01:51:52.760748 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:51:52.760719 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod923b90de_5e45_4add_b092_91af16b7dbad.slice/crio-48fdcde00f2d340a6ea75f58aefdbc17764a92e6033bf2530ab04d45395270ec WatchSource:0}: Error finding container 48fdcde00f2d340a6ea75f58aefdbc17764a92e6033bf2530ab04d45395270ec: Status 404 returned error can't find the container with id 48fdcde00f2d340a6ea75f58aefdbc17764a92e6033bf2530ab04d45395270ec Nov 03 01:51:53.106808 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:53.106774 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-676fb595d5-lmt4w"] Nov 03 01:51:53.109870 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:53.109849 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-676fb595d5-lmt4w" Nov 03 01:51:53.154310 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:53.154274 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/13fa5796-b3e6-4d1c-9759-88b7094d4b37-console-config\") pod \"console-676fb595d5-lmt4w\" (UID: \"13fa5796-b3e6-4d1c-9759-88b7094d4b37\") " pod="openshift-console/console-676fb595d5-lmt4w" Nov 03 01:51:53.154469 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:53.154319 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/13fa5796-b3e6-4d1c-9759-88b7094d4b37-service-ca\") pod \"console-676fb595d5-lmt4w\" (UID: \"13fa5796-b3e6-4d1c-9759-88b7094d4b37\") " pod="openshift-console/console-676fb595d5-lmt4w" Nov 03 01:51:53.154469 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:53.154340 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/13fa5796-b3e6-4d1c-9759-88b7094d4b37-tmp\") pod \"console-676fb595d5-lmt4w\" (UID: \"13fa5796-b3e6-4d1c-9759-88b7094d4b37\") " pod="openshift-console/console-676fb595d5-lmt4w" Nov 03 01:51:53.154469 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:53.154416 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb4tp\" (UniqueName: \"kubernetes.io/projected/13fa5796-b3e6-4d1c-9759-88b7094d4b37-kube-api-access-bb4tp\") pod \"console-676fb595d5-lmt4w\" (UID: \"13fa5796-b3e6-4d1c-9759-88b7094d4b37\") " pod="openshift-console/console-676fb595d5-lmt4w" Nov 03 01:51:53.154618 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:53.154535 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/13fa5796-b3e6-4d1c-9759-88b7094d4b37-console-oauth-config\") pod \"console-676fb595d5-lmt4w\" (UID: \"13fa5796-b3e6-4d1c-9759-88b7094d4b37\") " pod="openshift-console/console-676fb595d5-lmt4w" Nov 03 01:51:53.154618 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:53.154578 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/13fa5796-b3e6-4d1c-9759-88b7094d4b37-console-serving-cert\") pod \"console-676fb595d5-lmt4w\" (UID: \"13fa5796-b3e6-4d1c-9759-88b7094d4b37\") " pod="openshift-console/console-676fb595d5-lmt4w" Nov 03 01:51:53.154714 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:53.154699 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/13fa5796-b3e6-4d1c-9759-88b7094d4b37-oauth-serving-cert\") pod \"console-676fb595d5-lmt4w\" (UID: \"13fa5796-b3e6-4d1c-9759-88b7094d4b37\") " pod="openshift-console/console-676fb595d5-lmt4w" Nov 03 01:51:53.154759 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:53.154728 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13fa5796-b3e6-4d1c-9759-88b7094d4b37-trusted-ca-bundle\") pod \"console-676fb595d5-lmt4w\" (UID: \"13fa5796-b3e6-4d1c-9759-88b7094d4b37\") " pod="openshift-console/console-676fb595d5-lmt4w" Nov 03 01:51:53.166024 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:53.165994 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 03 01:51:53.166863 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:53.166842 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-676fb595d5-lmt4w"] Nov 03 01:51:53.256079 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:53.256050 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/13fa5796-b3e6-4d1c-9759-88b7094d4b37-console-config\") pod \"console-676fb595d5-lmt4w\" (UID: \"13fa5796-b3e6-4d1c-9759-88b7094d4b37\") " pod="openshift-console/console-676fb595d5-lmt4w" Nov 03 01:51:53.256079 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:53.256081 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/13fa5796-b3e6-4d1c-9759-88b7094d4b37-service-ca\") pod \"console-676fb595d5-lmt4w\" (UID: \"13fa5796-b3e6-4d1c-9759-88b7094d4b37\") " pod="openshift-console/console-676fb595d5-lmt4w" Nov 03 01:51:53.256297 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:53.256096 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/13fa5796-b3e6-4d1c-9759-88b7094d4b37-tmp\") pod \"console-676fb595d5-lmt4w\" (UID: \"13fa5796-b3e6-4d1c-9759-88b7094d4b37\") " pod="openshift-console/console-676fb595d5-lmt4w" Nov 03 01:51:53.256297 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:53.256113 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb4tp\" (UniqueName: \"kubernetes.io/projected/13fa5796-b3e6-4d1c-9759-88b7094d4b37-kube-api-access-bb4tp\") pod \"console-676fb595d5-lmt4w\" (UID: \"13fa5796-b3e6-4d1c-9759-88b7094d4b37\") " pod="openshift-console/console-676fb595d5-lmt4w" Nov 03 01:51:53.256297 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:53.256170 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/13fa5796-b3e6-4d1c-9759-88b7094d4b37-console-oauth-config\") pod \"console-676fb595d5-lmt4w\" (UID: \"13fa5796-b3e6-4d1c-9759-88b7094d4b37\") " pod="openshift-console/console-676fb595d5-lmt4w" Nov 03 01:51:53.256297 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:53.256221 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/13fa5796-b3e6-4d1c-9759-88b7094d4b37-console-serving-cert\") pod \"console-676fb595d5-lmt4w\" (UID: \"13fa5796-b3e6-4d1c-9759-88b7094d4b37\") " pod="openshift-console/console-676fb595d5-lmt4w" Nov 03 01:51:53.256609 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:53.256410 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/13fa5796-b3e6-4d1c-9759-88b7094d4b37-oauth-serving-cert\") pod \"console-676fb595d5-lmt4w\" (UID: \"13fa5796-b3e6-4d1c-9759-88b7094d4b37\") " pod="openshift-console/console-676fb595d5-lmt4w" Nov 03 01:51:53.256609 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:53.256451 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13fa5796-b3e6-4d1c-9759-88b7094d4b37-trusted-ca-bundle\") pod \"console-676fb595d5-lmt4w\" (UID: \"13fa5796-b3e6-4d1c-9759-88b7094d4b37\") " pod="openshift-console/console-676fb595d5-lmt4w" Nov 03 01:51:53.256609 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:53.256594 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/13fa5796-b3e6-4d1c-9759-88b7094d4b37-tmp\") pod \"console-676fb595d5-lmt4w\" (UID: \"13fa5796-b3e6-4d1c-9759-88b7094d4b37\") " pod="openshift-console/console-676fb595d5-lmt4w" Nov 03 01:51:53.256978 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:53.256923 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/13fa5796-b3e6-4d1c-9759-88b7094d4b37-console-config\") pod \"console-676fb595d5-lmt4w\" (UID: \"13fa5796-b3e6-4d1c-9759-88b7094d4b37\") " pod="openshift-console/console-676fb595d5-lmt4w" Nov 03 01:51:53.257167 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:53.257145 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/13fa5796-b3e6-4d1c-9759-88b7094d4b37-oauth-serving-cert\") pod \"console-676fb595d5-lmt4w\" (UID: \"13fa5796-b3e6-4d1c-9759-88b7094d4b37\") " pod="openshift-console/console-676fb595d5-lmt4w" Nov 03 01:51:53.257314 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:53.257222 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/13fa5796-b3e6-4d1c-9759-88b7094d4b37-service-ca\") pod \"console-676fb595d5-lmt4w\" (UID: \"13fa5796-b3e6-4d1c-9759-88b7094d4b37\") " pod="openshift-console/console-676fb595d5-lmt4w" Nov 03 01:51:53.257469 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:53.257452 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13fa5796-b3e6-4d1c-9759-88b7094d4b37-trusted-ca-bundle\") pod \"console-676fb595d5-lmt4w\" (UID: \"13fa5796-b3e6-4d1c-9759-88b7094d4b37\") " pod="openshift-console/console-676fb595d5-lmt4w" Nov 03 01:51:53.259348 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:53.259326 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/13fa5796-b3e6-4d1c-9759-88b7094d4b37-console-oauth-config\") pod \"console-676fb595d5-lmt4w\" (UID: \"13fa5796-b3e6-4d1c-9759-88b7094d4b37\") " pod="openshift-console/console-676fb595d5-lmt4w" Nov 03 01:51:53.259440 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:53.259350 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/13fa5796-b3e6-4d1c-9759-88b7094d4b37-console-serving-cert\") pod \"console-676fb595d5-lmt4w\" (UID: \"13fa5796-b3e6-4d1c-9759-88b7094d4b37\") " pod="openshift-console/console-676fb595d5-lmt4w" Nov 03 01:51:53.313551 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:53.313523 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb4tp\" (UniqueName: \"kubernetes.io/projected/13fa5796-b3e6-4d1c-9759-88b7094d4b37-kube-api-access-bb4tp\") pod \"console-676fb595d5-lmt4w\" (UID: \"13fa5796-b3e6-4d1c-9759-88b7094d4b37\") " pod="openshift-console/console-676fb595d5-lmt4w" Nov 03 01:51:53.422577 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:53.422273 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-676fb595d5-lmt4w" Nov 03 01:51:53.560211 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:53.560171 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/28f00eb6-78c2-4d93-bde5-c1ea12601dbd-original-pull-secret\") pod \"global-pull-secret-syncer-kkngd\" (UID: \"28f00eb6-78c2-4d93-bde5-c1ea12601dbd\") " pod="kube-system/global-pull-secret-syncer-kkngd" Nov 03 01:51:53.560396 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:53.560264 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b-metrics-certs\") pod \"network-metrics-daemon-vzfhc\" (UID: \"dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b\") " pod="openshift-multus/network-metrics-daemon-vzfhc" Nov 03 01:51:53.560396 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:53.560306 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr2jw\" (UniqueName: \"kubernetes.io/projected/6ae1437e-8885-42a5-b092-3862cd404c60-kube-api-access-sr2jw\") pod \"network-check-target-f5qvs\" (UID: \"6ae1437e-8885-42a5-b092-3862cd404c60\") " pod="openshift-network-diagnostics/network-check-target-f5qvs" Nov 03 01:51:53.612933 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:53.612899 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 01:51:53.612933 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:53.612933 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 01:51:53.613124 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:53.612991 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 01:51:53.623676 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:53.623649 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b-metrics-certs\") pod \"network-metrics-daemon-vzfhc\" (UID: \"dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b\") " pod="openshift-multus/network-metrics-daemon-vzfhc" Nov 03 01:51:53.624974 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:53.624944 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/28f00eb6-78c2-4d93-bde5-c1ea12601dbd-original-pull-secret\") pod \"global-pull-secret-syncer-kkngd\" (UID: \"28f00eb6-78c2-4d93-bde5-c1ea12601dbd\") " pod="kube-system/global-pull-secret-syncer-kkngd" Nov 03 01:51:53.673746 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:53.673664 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 01:51:53.684856 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:53.684794 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr2jw\" (UniqueName: \"kubernetes.io/projected/6ae1437e-8885-42a5-b092-3862cd404c60-kube-api-access-sr2jw\") pod \"network-check-target-f5qvs\" (UID: \"6ae1437e-8885-42a5-b092-3862cd404c60\") " pod="openshift-network-diagnostics/network-check-target-f5qvs" Nov 03 01:51:53.762986 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:53.762952 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-699d68669f-9jg2q" event={"ID":"923b90de-5e45-4add-b092-91af16b7dbad","Type":"ContainerStarted","Data":"48fdcde00f2d340a6ea75f58aefdbc17764a92e6033bf2530ab04d45395270ec"} Nov 03 01:51:53.804943 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:53.804910 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-676fb595d5-lmt4w"] Nov 03 01:51:53.808630 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:51:53.808600 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13fa5796_b3e6_4d1c_9759_88b7094d4b37.slice/crio-04fb7c6e6f597b086dc818c0595aee3f95569ab857b1f8ef044b5d76100b4942 WatchSource:0}: Error finding container 04fb7c6e6f597b086dc818c0595aee3f95569ab857b1f8ef044b5d76100b4942: Status 404 returned error can't find the container with id 04fb7c6e6f597b086dc818c0595aee3f95569ab857b1f8ef044b5d76100b4942 Nov 03 01:51:53.937670 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:53.937600 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-zxsvp\"" Nov 03 01:51:53.944653 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:53.944624 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-btxgv\"" Nov 03 01:51:53.945789 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:53.945763 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kkngd" Nov 03 01:51:53.946668 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:53.946646 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-wpc48\"" Nov 03 01:51:53.953080 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:53.953061 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vzfhc" Nov 03 01:51:53.955368 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:53.955299 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5qvs" Nov 03 01:51:54.340869 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:54.335663 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kkngd"] Nov 03 01:51:54.340869 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:51:54.337389 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28f00eb6_78c2_4d93_bde5_c1ea12601dbd.slice/crio-7f82c91811ed5d544dec301d9fdf527dfc4c44d87b6967fdcaf42c0d21897209 WatchSource:0}: Error finding container 7f82c91811ed5d544dec301d9fdf527dfc4c44d87b6967fdcaf42c0d21897209: Status 404 returned error can't find the container with id 7f82c91811ed5d544dec301d9fdf527dfc4c44d87b6967fdcaf42c0d21897209 Nov 03 01:51:54.385230 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:54.385204 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-f5qvs"] Nov 03 01:51:54.390257 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:54.390128 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vzfhc"] Nov 03 01:51:54.768205 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:54.768128 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-b984f6b6b-g6f2h" event={"ID":"e32389d5-705f-4202-98ea-6fc6a5ad5d54","Type":"ContainerStarted","Data":"4095e6f9cecfa68f2f273bdb111e19f22aab49b04964be3a3c2f52ef8f9aae0f"} Nov 03 01:51:54.769946 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:54.769913 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vzfhc" event={"ID":"dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b","Type":"ContainerStarted","Data":"b72c4f4a29c37e5578ac2dffd291c96fe82c9cf9453fcba1b9525237a59fa29b"} Nov 03 01:51:54.771738 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:54.771272 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-f5qvs" event={"ID":"6ae1437e-8885-42a5-b092-3862cd404c60","Type":"ContainerStarted","Data":"ce9b5c350d4575ec8ed292d16267ce993d863109ff54133fa02f2f49f41c879a"} Nov 03 01:51:54.772500 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:54.772480 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kkngd" event={"ID":"28f00eb6-78c2-4d93-bde5-c1ea12601dbd","Type":"ContainerStarted","Data":"7f82c91811ed5d544dec301d9fdf527dfc4c44d87b6967fdcaf42c0d21897209"} Nov 03 01:51:54.774058 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:54.774031 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-676fb595d5-lmt4w" event={"ID":"13fa5796-b3e6-4d1c-9759-88b7094d4b37","Type":"ContainerStarted","Data":"d499decdd20c159be35ca7f639b97bf63f9631407a3f6dde6ed02d2b27e21805"} Nov 03 01:51:54.774145 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:54.774062 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-676fb595d5-lmt4w" event={"ID":"13fa5796-b3e6-4d1c-9759-88b7094d4b37","Type":"ContainerStarted","Data":"04fb7c6e6f597b086dc818c0595aee3f95569ab857b1f8ef044b5d76100b4942"} Nov 03 01:51:54.776949 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:54.776927 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3d7d07f7-27c2-489a-8e44-fe366584a267","Type":"ContainerStarted","Data":"2a036232c737f1345f0379e4433a70dbbce6f1a0b055c8986f64f79b481c82a3"} Nov 03 01:51:54.777023 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:54.776973 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3d7d07f7-27c2-489a-8e44-fe366584a267","Type":"ContainerStarted","Data":"087add6fe689b6c7740d6995d0dcbb3dd5a93ce47965875f6a4a96e7b644cfd7"} Nov 03 01:51:54.777023 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:54.777000 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3d7d07f7-27c2-489a-8e44-fe366584a267","Type":"ContainerStarted","Data":"9ba45f94f3b5795d52d3c4aa689200dd500416f2f0ec4785d2d22b03676b8abb"} Nov 03 01:51:54.777023 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:54.777021 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3d7d07f7-27c2-489a-8e44-fe366584a267","Type":"ContainerStarted","Data":"9ba1dd1268281f89787a0064d6dd633f1a0d1c321c5397430616868590b663c5"} Nov 03 01:51:54.777159 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:54.777048 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3d7d07f7-27c2-489a-8e44-fe366584a267","Type":"ContainerStarted","Data":"5a89dbaf332edc96b69633d170776b866432066421f0079dc19cbeb24cc16b63"} Nov 03 01:51:54.886366 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:54.886305 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-b984f6b6b-g6f2h" podStartSLOduration=1.512210735 podStartE2EDuration="3.886291711s" totalImagesPullingTime="2.374080976s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:51:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:51:54.885874681 +0000 UTC m=+66.889322145" watchObservedRunningTime="2026-11-03 01:51:54.886291711 +0000 UTC m=+66.889739172" Nov 03 01:51:55.003558 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:55.003499 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-676fb595d5-lmt4w" podStartSLOduration=2.003482504 podStartE2EDuration="2.003482504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:51:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:51:55.002471513 +0000 UTC m=+67.005918988" watchObservedRunningTime="2026-11-03 01:51:55.003482504 +0000 UTC m=+67.006929969" Nov 03 01:51:56.787874 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:56.787778 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-699d68669f-9jg2q" event={"ID":"923b90de-5e45-4add-b092-91af16b7dbad","Type":"ContainerStarted","Data":"4d476527577f29756273c758334decdd910c58765eb88d19e3d8c20ad05f331a"} Nov 03 01:51:56.787874 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:56.787862 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-699d68669f-9jg2q" event={"ID":"923b90de-5e45-4add-b092-91af16b7dbad","Type":"ContainerStarted","Data":"5313041f9b23bf471923a735c54a4ec6b7017280d76e6d0076a8d87bdf6e11b7"} Nov 03 01:51:56.788394 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:56.787890 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-699d68669f-9jg2q" event={"ID":"923b90de-5e45-4add-b092-91af16b7dbad","Type":"ContainerStarted","Data":"b5abfd3e66df00a51ee5d28c5499da26685f8bd602a240688d55afedb46d0a1e"} Nov 03 01:51:56.791945 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:56.791920 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3d7d07f7-27c2-489a-8e44-fe366584a267","Type":"ContainerStarted","Data":"45237327ba6c01b1abbd14db2279979dfafd6ffd6fe0e98eddd6065d580bd86b"} Nov 03 01:51:56.989690 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:56.989633 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-699d68669f-9jg2q" podStartSLOduration=2.3791852110000002 podStartE2EDuration="5.989616002s" totalImagesPullingTime="3.610430791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:51:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:51:56.937837329 +0000 UTC m=+68.941284795" watchObservedRunningTime="2026-11-03 01:51:56.989616002 +0000 UTC m=+68.993063467" Nov 03 01:51:57.062230 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:57.062159 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.296217717 podStartE2EDuration="9.062139151s" totalImagesPullingTime="5.756239745s" totalInitContainerRuntime="9.681689ms" isStatefulPod=false podCreationTimestamp="2026-11-03 01:51:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:51:57.061308714 +0000 UTC m=+69.064756178" watchObservedRunningTime="2026-11-03 01:51:57.062139151 +0000 UTC m=+69.065586615" Nov 03 01:51:57.797337 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:57.797304 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vzfhc" event={"ID":"dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b","Type":"ContainerStarted","Data":"2072c28384d85ba22fa45282b584106c357928d1ce061caa3787506ee2a53db1"} Nov 03 01:51:57.797738 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:57.797380 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vzfhc" event={"ID":"dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b","Type":"ContainerStarted","Data":"e3a6af98c2ed591e808cbc0c699e46b4bda5283a1e094b8e10855a3ec1ef44e4"} Nov 03 01:51:57.912750 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:57.912692 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-vzfhc" podStartSLOduration=66.397958615 podStartE2EDuration="1m8.912677926s" totalImagesPullingTime="2.514719311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:50:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:51:57.912573341 +0000 UTC m=+69.916020803" watchObservedRunningTime="2026-11-03 01:51:57.912677926 +0000 UTC m=+69.916125371" Nov 03 01:51:58.920589 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:58.920552 2579 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-676fb595d5-lmt4w"] Nov 03 01:51:58.965868 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:58.965836 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f524q_7d2438ed-bfc2-4639-b87d-84689a82123e/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:58.971096 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:58.971070 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-69c467f7c8-cpfwh"] Nov 03 01:51:58.974233 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:58.974217 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69c467f7c8-cpfwh" Nov 03 01:51:59.019786 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:59.019762 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9kwt\" (UniqueName: \"kubernetes.io/projected/a4727262-eb15-4453-adf2-0fc6c2358508-kube-api-access-d9kwt\") pod \"console-69c467f7c8-cpfwh\" (UID: \"a4727262-eb15-4453-adf2-0fc6c2358508\") " pod="openshift-console/console-69c467f7c8-cpfwh" Nov 03 01:51:59.019925 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:59.019801 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a4727262-eb15-4453-adf2-0fc6c2358508-console-config\") pod \"console-69c467f7c8-cpfwh\" (UID: \"a4727262-eb15-4453-adf2-0fc6c2358508\") " pod="openshift-console/console-69c467f7c8-cpfwh" Nov 03 01:51:59.019925 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:59.019821 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a4727262-eb15-4453-adf2-0fc6c2358508-console-oauth-config\") pod \"console-69c467f7c8-cpfwh\" (UID: \"a4727262-eb15-4453-adf2-0fc6c2358508\") " pod="openshift-console/console-69c467f7c8-cpfwh" Nov 03 01:51:59.019925 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:59.019882 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a4727262-eb15-4453-adf2-0fc6c2358508-console-serving-cert\") pod \"console-69c467f7c8-cpfwh\" (UID: \"a4727262-eb15-4453-adf2-0fc6c2358508\") " pod="openshift-console/console-69c467f7c8-cpfwh" Nov 03 01:51:59.020012 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:59.019935 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a4727262-eb15-4453-adf2-0fc6c2358508-oauth-serving-cert\") pod \"console-69c467f7c8-cpfwh\" (UID: \"a4727262-eb15-4453-adf2-0fc6c2358508\") " pod="openshift-console/console-69c467f7c8-cpfwh" Nov 03 01:51:59.020044 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:59.020013 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4727262-eb15-4453-adf2-0fc6c2358508-tmp\") pod \"console-69c467f7c8-cpfwh\" (UID: \"a4727262-eb15-4453-adf2-0fc6c2358508\") " pod="openshift-console/console-69c467f7c8-cpfwh" Nov 03 01:51:59.020044 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:59.020031 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4727262-eb15-4453-adf2-0fc6c2358508-trusted-ca-bundle\") pod \"console-69c467f7c8-cpfwh\" (UID: \"a4727262-eb15-4453-adf2-0fc6c2358508\") " pod="openshift-console/console-69c467f7c8-cpfwh" Nov 03 01:51:59.020095 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:59.020049 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a4727262-eb15-4453-adf2-0fc6c2358508-service-ca\") pod \"console-69c467f7c8-cpfwh\" (UID: \"a4727262-eb15-4453-adf2-0fc6c2358508\") " pod="openshift-console/console-69c467f7c8-cpfwh" Nov 03 01:51:59.022547 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:59.022522 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69c467f7c8-cpfwh"] Nov 03 01:51:59.121176 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:59.121129 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9kwt\" (UniqueName: \"kubernetes.io/projected/a4727262-eb15-4453-adf2-0fc6c2358508-kube-api-access-d9kwt\") pod \"console-69c467f7c8-cpfwh\" (UID: \"a4727262-eb15-4453-adf2-0fc6c2358508\") " pod="openshift-console/console-69c467f7c8-cpfwh" Nov 03 01:51:59.121548 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:59.121199 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a4727262-eb15-4453-adf2-0fc6c2358508-console-config\") pod \"console-69c467f7c8-cpfwh\" (UID: \"a4727262-eb15-4453-adf2-0fc6c2358508\") " pod="openshift-console/console-69c467f7c8-cpfwh" Nov 03 01:51:59.121548 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:59.121224 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a4727262-eb15-4453-adf2-0fc6c2358508-console-oauth-config\") pod \"console-69c467f7c8-cpfwh\" (UID: \"a4727262-eb15-4453-adf2-0fc6c2358508\") " pod="openshift-console/console-69c467f7c8-cpfwh" Nov 03 01:51:59.121548 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:59.121255 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a4727262-eb15-4453-adf2-0fc6c2358508-console-serving-cert\") pod \"console-69c467f7c8-cpfwh\" (UID: \"a4727262-eb15-4453-adf2-0fc6c2358508\") " pod="openshift-console/console-69c467f7c8-cpfwh" Nov 03 01:51:59.121548 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:59.121272 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a4727262-eb15-4453-adf2-0fc6c2358508-oauth-serving-cert\") pod \"console-69c467f7c8-cpfwh\" (UID: \"a4727262-eb15-4453-adf2-0fc6c2358508\") " pod="openshift-console/console-69c467f7c8-cpfwh" Nov 03 01:51:59.121548 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:59.121315 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4727262-eb15-4453-adf2-0fc6c2358508-tmp\") pod \"console-69c467f7c8-cpfwh\" (UID: \"a4727262-eb15-4453-adf2-0fc6c2358508\") " pod="openshift-console/console-69c467f7c8-cpfwh" Nov 03 01:51:59.121548 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:59.121347 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4727262-eb15-4453-adf2-0fc6c2358508-trusted-ca-bundle\") pod \"console-69c467f7c8-cpfwh\" (UID: \"a4727262-eb15-4453-adf2-0fc6c2358508\") " pod="openshift-console/console-69c467f7c8-cpfwh" Nov 03 01:51:59.121548 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:59.121379 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a4727262-eb15-4453-adf2-0fc6c2358508-service-ca\") pod \"console-69c467f7c8-cpfwh\" (UID: \"a4727262-eb15-4453-adf2-0fc6c2358508\") " pod="openshift-console/console-69c467f7c8-cpfwh" Nov 03 01:51:59.121976 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:59.121796 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4727262-eb15-4453-adf2-0fc6c2358508-tmp\") pod \"console-69c467f7c8-cpfwh\" (UID: \"a4727262-eb15-4453-adf2-0fc6c2358508\") " pod="openshift-console/console-69c467f7c8-cpfwh" Nov 03 01:51:59.122261 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:59.122237 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a4727262-eb15-4453-adf2-0fc6c2358508-oauth-serving-cert\") pod \"console-69c467f7c8-cpfwh\" (UID: \"a4727262-eb15-4453-adf2-0fc6c2358508\") " pod="openshift-console/console-69c467f7c8-cpfwh" Nov 03 01:51:59.122351 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:59.122336 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a4727262-eb15-4453-adf2-0fc6c2358508-console-config\") pod \"console-69c467f7c8-cpfwh\" (UID: \"a4727262-eb15-4453-adf2-0fc6c2358508\") " pod="openshift-console/console-69c467f7c8-cpfwh" Nov 03 01:51:59.122566 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:59.122541 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4727262-eb15-4453-adf2-0fc6c2358508-trusted-ca-bundle\") pod \"console-69c467f7c8-cpfwh\" (UID: \"a4727262-eb15-4453-adf2-0fc6c2358508\") " pod="openshift-console/console-69c467f7c8-cpfwh" Nov 03 01:51:59.122662 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:59.122556 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a4727262-eb15-4453-adf2-0fc6c2358508-service-ca\") pod \"console-69c467f7c8-cpfwh\" (UID: \"a4727262-eb15-4453-adf2-0fc6c2358508\") " pod="openshift-console/console-69c467f7c8-cpfwh" Nov 03 01:51:59.125302 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:59.125275 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a4727262-eb15-4453-adf2-0fc6c2358508-console-serving-cert\") pod \"console-69c467f7c8-cpfwh\" (UID: \"a4727262-eb15-4453-adf2-0fc6c2358508\") " pod="openshift-console/console-69c467f7c8-cpfwh" Nov 03 01:51:59.128077 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:59.128056 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a4727262-eb15-4453-adf2-0fc6c2358508-console-oauth-config\") pod \"console-69c467f7c8-cpfwh\" (UID: \"a4727262-eb15-4453-adf2-0fc6c2358508\") " pod="openshift-console/console-69c467f7c8-cpfwh" Nov 03 01:51:59.167522 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:59.167442 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f524q_7d2438ed-bfc2-4639-b87d-84689a82123e/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:59.188830 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:59.188767 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9kwt\" (UniqueName: \"kubernetes.io/projected/a4727262-eb15-4453-adf2-0fc6c2358508-kube-api-access-d9kwt\") pod \"console-69c467f7c8-cpfwh\" (UID: \"a4727262-eb15-4453-adf2-0fc6c2358508\") " pod="openshift-console/console-69c467f7c8-cpfwh" Nov 03 01:51:59.286250 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:59.286219 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69c467f7c8-cpfwh" Nov 03 01:51:59.672069 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:59.672038 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69c467f7c8-cpfwh"] Nov 03 01:51:59.674554 ip-10-0-140-140 kubenswrapper[2579]: W1103 01:51:59.674519 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4727262_eb15_4453_adf2_0fc6c2358508.slice/crio-59950987f839d6591cdddd6cab36a4cf1540b371be239279dc6c981d90a53e6a WatchSource:0}: Error finding container 59950987f839d6591cdddd6cab36a4cf1540b371be239279dc6c981d90a53e6a: Status 404 returned error can't find the container with id 59950987f839d6591cdddd6cab36a4cf1540b371be239279dc6c981d90a53e6a Nov 03 01:51:59.810225 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:59.809590 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69c467f7c8-cpfwh" event={"ID":"a4727262-eb15-4453-adf2-0fc6c2358508","Type":"ContainerStarted","Data":"6ed06dd8080d8cc71cbc17d832250c0d5dc93fa2cd71732a1350fc35b6a29d49"} Nov 03 01:51:59.810225 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:59.809651 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69c467f7c8-cpfwh" event={"ID":"a4727262-eb15-4453-adf2-0fc6c2358508","Type":"ContainerStarted","Data":"59950987f839d6591cdddd6cab36a4cf1540b371be239279dc6c981d90a53e6a"} Nov 03 01:51:59.929020 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:51:59.928922 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-69c467f7c8-cpfwh" podStartSLOduration=1.92890297 podStartE2EDuration="1.92890297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:51:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:51:59.927502108 +0000 UTC m=+71.930949572" watchObservedRunningTime="2026-11-03 01:51:59.92890297 +0000 UTC m=+71.932350430" Nov 03 01:52:00.815594 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:52:00.815564 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-f5qvs" event={"ID":"6ae1437e-8885-42a5-b092-3862cd404c60","Type":"ContainerStarted","Data":"c71f0d176a818d7da8914df4730130e0d88441711412da65bd83b351e846ef20"} Nov 03 01:52:00.815777 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:52:00.815758 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-f5qvs" Nov 03 01:52:00.934342 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:52:00.934289 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-f5qvs" podStartSLOduration=66.060559433 podStartE2EDuration="1m11.934272375s" totalImagesPullingTime="5.873712942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:50:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:52:00.933176066 +0000 UTC m=+72.936623532" watchObservedRunningTime="2026-11-03 01:52:00.934272375 +0000 UTC m=+72.937719839" Nov 03 01:52:02.688922 ip-10-0-140-140 kubenswrapper[2579]: I1103 01:52:02.688789 2579 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 01:52:02.688916 ip-10-0-140-140 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 01:52:02.701736 ip-10-0-140-140 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 01:52:02.701890 ip-10-0-140-140 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 01:52:02.701937 ip-10-0-140-140 systemd[1]: kubelet.service: Consumed 4.113s CPU time. Nov 03 01:52:02.710989 ip-10-0-140-140 systemd[1]: Starting Kubernetes Kubelet... Nov 03 01:52:02.769507 ip-10-0-140-140 kubenswrapper[5605]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 01:52:02.769507 ip-10-0-140-140 kubenswrapper[5605]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 01:52:02.769507 ip-10-0-140-140 kubenswrapper[5605]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 01:52:02.769507 ip-10-0-140-140 kubenswrapper[5605]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 01:52:02.771443 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771348 5605 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 01:52:02.771443 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771442 5605 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 01:52:02.771443 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771446 5605 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 01:52:02.771540 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771449 5605 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 01:52:02.771540 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771453 5605 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 01:52:02.771540 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771456 5605 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 01:52:02.771540 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771459 5605 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 01:52:02.771540 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771462 5605 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 01:52:02.771540 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771465 5605 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 01:52:02.771540 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771467 5605 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 01:52:02.771540 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771470 5605 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 01:52:02.771540 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771472 5605 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 01:52:02.771540 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771475 5605 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 01:52:02.771540 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771478 5605 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 01:52:02.771540 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771480 5605 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 01:52:02.771540 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771482 5605 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 01:52:02.771540 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771485 5605 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 01:52:02.771540 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771487 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 01:52:02.771540 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771490 5605 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 01:52:02.771540 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771492 5605 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 01:52:02.771540 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771498 5605 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 01:52:02.771540 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771502 5605 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 01:52:02.771540 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771504 5605 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 01:52:02.772026 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771507 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 01:52:02.772026 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771510 5605 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 01:52:02.772026 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771512 5605 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 01:52:02.772026 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771515 5605 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 01:52:02.772026 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771517 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 01:52:02.772026 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771520 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 01:52:02.772026 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771523 5605 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 01:52:02.772026 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771526 5605 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 01:52:02.772026 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771529 5605 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 01:52:02.772026 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771531 5605 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 01:52:02.772026 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771534 5605 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 01:52:02.772026 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771536 5605 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 01:52:02.772026 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771539 5605 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 01:52:02.772026 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771541 5605 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 01:52:02.772026 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771546 5605 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 01:52:02.772026 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771550 5605 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 01:52:02.772026 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771554 5605 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 01:52:02.772026 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771557 5605 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 01:52:02.772026 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771560 5605 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 01:52:02.772510 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771562 5605 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 01:52:02.772510 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771565 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 01:52:02.772510 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771568 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 01:52:02.772510 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771571 5605 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 01:52:02.772510 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771573 5605 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 01:52:02.772510 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771576 5605 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 01:52:02.772510 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771579 5605 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 01:52:02.772510 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771582 5605 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 01:52:02.772510 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771584 5605 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 01:52:02.772510 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771587 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 01:52:02.772510 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771590 5605 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 01:52:02.772510 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771593 5605 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 01:52:02.772510 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771595 5605 feature_gate.go:386] unrecognized feature gate: Example Nov 03 01:52:02.772510 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771598 5605 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 01:52:02.772510 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771602 5605 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 01:52:02.772510 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771604 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 01:52:02.772510 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771607 5605 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 01:52:02.772510 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771609 5605 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 01:52:02.772510 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771612 5605 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 01:52:02.772954 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771616 5605 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 01:52:02.772954 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771619 5605 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 01:52:02.772954 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771621 5605 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 01:52:02.772954 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771624 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 01:52:02.772954 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771628 5605 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 01:52:02.772954 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771633 5605 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 01:52:02.772954 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771637 5605 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 01:52:02.772954 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771640 5605 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 01:52:02.772954 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771643 5605 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 01:52:02.772954 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771645 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 01:52:02.772954 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771648 5605 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 01:52:02.772954 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771651 5605 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 01:52:02.772954 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771653 5605 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 01:52:02.772954 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771656 5605 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 01:52:02.772954 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771658 5605 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 01:52:02.772954 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771661 5605 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 01:52:02.772954 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771663 5605 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 01:52:02.772954 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771666 5605 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 01:52:02.772954 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771669 5605 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 01:52:02.773438 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771671 5605 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 01:52:02.773438 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771673 5605 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 01:52:02.773438 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771676 5605 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 01:52:02.773438 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771678 5605 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 01:52:02.773438 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771681 5605 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 01:52:02.773438 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771683 5605 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 01:52:02.773438 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771686 5605 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 01:52:02.773438 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771688 5605 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 01:52:02.773438 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771691 5605 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 01:52:02.773438 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771694 5605 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 01:52:02.773438 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771697 5605 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 01:52:02.773438 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771700 5605 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 01:52:02.773438 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771703 5605 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 01:52:02.773438 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771706 5605 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 01:52:02.773438 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771709 5605 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 01:52:02.773438 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771711 5605 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 01:52:02.773438 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771714 5605 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 01:52:02.773438 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771717 5605 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 01:52:02.773438 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771720 5605 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 01:52:02.773438 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771722 5605 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 01:52:02.773932 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771725 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 01:52:02.773932 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771727 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 01:52:02.773932 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771730 5605 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 01:52:02.773932 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771732 5605 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 01:52:02.773932 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771734 5605 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 01:52:02.773932 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.771737 5605 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 01:52:02.773932 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772508 5605 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 01:52:02.773932 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772517 5605 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 01:52:02.773932 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772521 5605 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 01:52:02.773932 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772524 5605 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 01:52:02.773932 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772527 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 01:52:02.773932 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772529 5605 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 01:52:02.773932 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772532 5605 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 01:52:02.773932 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772535 5605 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 01:52:02.773932 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772537 5605 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 01:52:02.773932 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772540 5605 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 01:52:02.773932 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772542 5605 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 01:52:02.773932 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772545 5605 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 01:52:02.773932 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772548 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 01:52:02.773932 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772550 5605 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 01:52:02.774453 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772553 5605 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 01:52:02.774453 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772558 5605 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 01:52:02.774453 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772562 5605 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 01:52:02.774453 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772564 5605 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 01:52:02.774453 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772567 5605 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 01:52:02.774453 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772570 5605 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 01:52:02.774453 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772574 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 01:52:02.774453 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772576 5605 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 01:52:02.774453 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772579 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 01:52:02.774453 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772582 5605 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 01:52:02.774453 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772585 5605 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 01:52:02.774453 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772593 5605 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 01:52:02.774453 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772596 5605 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 01:52:02.774453 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772599 5605 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 01:52:02.774453 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772601 5605 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 01:52:02.774453 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772604 5605 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 01:52:02.774453 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772608 5605 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 01:52:02.774453 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772611 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 01:52:02.774453 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772615 5605 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 01:52:02.774921 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772618 5605 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 01:52:02.774921 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772621 5605 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 01:52:02.774921 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772623 5605 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 01:52:02.774921 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772626 5605 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 01:52:02.774921 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772628 5605 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 01:52:02.774921 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772631 5605 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 01:52:02.774921 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772634 5605 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 01:52:02.774921 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772636 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 01:52:02.774921 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772638 5605 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 01:52:02.774921 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772641 5605 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 01:52:02.774921 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772644 5605 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 01:52:02.774921 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772646 5605 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 01:52:02.774921 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772649 5605 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 01:52:02.774921 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772652 5605 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 01:52:02.774921 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772655 5605 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 01:52:02.774921 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772657 5605 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 01:52:02.774921 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772660 5605 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 01:52:02.774921 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772662 5605 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 01:52:02.774921 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772665 5605 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 01:52:02.775400 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772668 5605 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 01:52:02.775400 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772671 5605 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 01:52:02.775400 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772673 5605 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 01:52:02.775400 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772676 5605 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 01:52:02.775400 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772678 5605 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 01:52:02.775400 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772681 5605 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 01:52:02.775400 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772684 5605 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 01:52:02.775400 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772686 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 01:52:02.775400 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772689 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 01:52:02.775400 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772691 5605 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 01:52:02.775400 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772694 5605 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 01:52:02.775400 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772696 5605 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 01:52:02.775400 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772699 5605 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 01:52:02.775400 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772701 5605 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 01:52:02.775400 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772703 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 01:52:02.775400 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772706 5605 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 01:52:02.775400 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772708 5605 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 01:52:02.775400 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772711 5605 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 01:52:02.775400 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772713 5605 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 01:52:02.775923 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772716 5605 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 01:52:02.775923 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772718 5605 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 01:52:02.775923 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772721 5605 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 01:52:02.775923 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772723 5605 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 01:52:02.775923 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772726 5605 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 01:52:02.775923 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772728 5605 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 01:52:02.775923 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772731 5605 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 01:52:02.775923 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772734 5605 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 01:52:02.775923 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772737 5605 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 01:52:02.775923 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772739 5605 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 01:52:02.775923 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772742 5605 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 01:52:02.775923 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772744 5605 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 01:52:02.775923 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772749 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 01:52:02.775923 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772752 5605 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 01:52:02.775923 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772754 5605 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 01:52:02.775923 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772757 5605 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 01:52:02.775923 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772759 5605 feature_gate.go:386] unrecognized feature gate: Example Nov 03 01:52:02.775923 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772762 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 01:52:02.775923 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772764 5605 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 01:52:02.775923 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772767 5605 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 01:52:02.776434 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772770 5605 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 01:52:02.776434 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772772 5605 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 01:52:02.776434 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772776 5605 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 01:52:02.776434 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772778 5605 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 01:52:02.776434 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772781 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 01:52:02.776434 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772783 5605 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 01:52:02.776434 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772786 5605 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 01:52:02.776434 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772788 5605 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 01:52:02.776434 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772791 5605 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 01:52:02.776434 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772793 5605 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 01:52:02.776434 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772796 5605 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 01:52:02.776434 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772798 5605 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 01:52:02.776434 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772801 5605 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 01:52:02.776434 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772803 5605 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 01:52:02.776434 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.772806 5605 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 01:52:02.776434 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773289 5605 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 01:52:02.776434 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773296 5605 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 01:52:02.776434 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773299 5605 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 01:52:02.776434 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773302 5605 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 01:52:02.776434 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773305 5605 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 01:52:02.776931 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773308 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 01:52:02.776931 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773310 5605 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 01:52:02.776931 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773313 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 01:52:02.776931 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773315 5605 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 01:52:02.776931 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773320 5605 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 01:52:02.776931 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773323 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 01:52:02.776931 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773325 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 01:52:02.776931 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773328 5605 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 01:52:02.776931 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773330 5605 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 01:52:02.776931 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773333 5605 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 01:52:02.776931 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773335 5605 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 01:52:02.776931 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773338 5605 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 01:52:02.776931 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773341 5605 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 01:52:02.776931 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773344 5605 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 01:52:02.776931 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773346 5605 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 01:52:02.776931 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773349 5605 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 01:52:02.776931 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773351 5605 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 01:52:02.776931 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773384 5605 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 01:52:02.776931 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773387 5605 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 01:52:02.776931 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773389 5605 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 01:52:02.777552 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773392 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 01:52:02.777552 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773395 5605 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 01:52:02.777552 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773397 5605 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 01:52:02.777552 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773400 5605 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 01:52:02.777552 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773403 5605 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 01:52:02.777552 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773405 5605 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 01:52:02.777552 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773408 5605 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 01:52:02.777552 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773411 5605 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 01:52:02.777552 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773414 5605 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 01:52:02.777552 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773417 5605 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 01:52:02.777552 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773419 5605 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 01:52:02.777552 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773422 5605 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 01:52:02.777552 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773424 5605 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 01:52:02.777552 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773427 5605 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 01:52:02.777552 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773429 5605 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 01:52:02.777552 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773431 5605 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 01:52:02.777552 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773435 5605 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 01:52:02.777552 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773438 5605 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 01:52:02.777552 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773441 5605 feature_gate.go:386] unrecognized feature gate: Example Nov 03 01:52:02.777552 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773444 5605 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 01:52:02.778342 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773447 5605 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 01:52:02.778342 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773450 5605 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 01:52:02.778342 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773453 5605 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 01:52:02.778342 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773455 5605 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 01:52:02.778342 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773458 5605 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 01:52:02.778342 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773461 5605 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 01:52:02.778342 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773463 5605 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 01:52:02.778342 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773466 5605 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 01:52:02.778342 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773468 5605 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 01:52:02.778342 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773472 5605 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 01:52:02.778342 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773476 5605 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 01:52:02.778342 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773479 5605 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 01:52:02.778342 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773481 5605 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 01:52:02.778342 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773484 5605 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 01:52:02.778342 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773487 5605 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 01:52:02.778342 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773489 5605 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 01:52:02.778342 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773492 5605 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 01:52:02.778342 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773495 5605 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 01:52:02.778342 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773497 5605 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 01:52:02.778870 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773500 5605 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 01:52:02.778870 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773502 5605 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 01:52:02.778870 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773505 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 01:52:02.778870 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773507 5605 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 01:52:02.778870 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773510 5605 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 01:52:02.778870 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773512 5605 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 01:52:02.778870 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773515 5605 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 01:52:02.778870 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773517 5605 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 01:52:02.778870 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773520 5605 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 01:52:02.778870 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773525 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 01:52:02.778870 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773528 5605 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 01:52:02.778870 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773530 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 01:52:02.778870 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773533 5605 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 01:52:02.778870 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773535 5605 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 01:52:02.778870 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773538 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 01:52:02.778870 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773540 5605 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 01:52:02.778870 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773543 5605 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 01:52:02.778870 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773545 5605 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 01:52:02.778870 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773548 5605 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 01:52:02.779327 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773551 5605 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 01:52:02.779327 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773553 5605 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 01:52:02.779327 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773555 5605 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 01:52:02.779327 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773558 5605 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 01:52:02.779327 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773561 5605 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 01:52:02.779327 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773563 5605 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 01:52:02.779327 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773565 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 01:52:02.779327 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773568 5605 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 01:52:02.779327 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773571 5605 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 01:52:02.779327 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773575 5605 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 01:52:02.779327 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773578 5605 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 01:52:02.779327 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773581 5605 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 01:52:02.779327 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773583 5605 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 01:52:02.779327 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773586 5605 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 01:52:02.779327 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773588 5605 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 01:52:02.779327 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773591 5605 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 01:52:02.779327 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773593 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 01:52:02.779327 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773596 5605 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 01:52:02.779327 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773598 5605 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 01:52:02.779796 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773602 5605 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 01:52:02.779796 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773604 5605 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 01:52:02.779796 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773607 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 01:52:02.779796 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.773610 5605 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 01:52:02.779796 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773708 5605 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 01:52:02.779796 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773724 5605 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 01:52:02.779796 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773729 5605 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 01:52:02.779796 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773734 5605 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 01:52:02.779796 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773738 5605 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 01:52:02.779796 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773741 5605 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 01:52:02.779796 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773745 5605 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 01:52:02.779796 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773749 5605 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 01:52:02.779796 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773753 5605 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 01:52:02.779796 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773756 5605 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 01:52:02.779796 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773760 5605 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 01:52:02.779796 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773763 5605 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 01:52:02.779796 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773766 5605 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 01:52:02.779796 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773769 5605 flags.go:64] FLAG: --cgroup-root="" Nov 03 01:52:02.779796 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773772 5605 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 01:52:02.779796 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773775 5605 flags.go:64] FLAG: --client-ca-file="" Nov 03 01:52:02.779796 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773778 5605 flags.go:64] FLAG: --cloud-provider="external" Nov 03 01:52:02.779796 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773781 5605 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 01:52:02.779796 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773785 5605 flags.go:64] FLAG: --cluster-domain="" Nov 03 01:52:02.780345 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773788 5605 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 01:52:02.780345 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773791 5605 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 01:52:02.780345 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773794 5605 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 01:52:02.780345 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773797 5605 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 01:52:02.780345 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773801 5605 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 01:52:02.780345 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773804 5605 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 01:52:02.780345 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773807 5605 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 01:52:02.780345 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773810 5605 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 01:52:02.780345 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773813 5605 flags.go:64] FLAG: --contention-profiling="false" Nov 03 01:52:02.780345 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773816 5605 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 01:52:02.780345 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773819 5605 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 01:52:02.780345 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773822 5605 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 01:52:02.780345 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773825 5605 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 01:52:02.780345 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773829 5605 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 01:52:02.780345 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773832 5605 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 01:52:02.780345 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773835 5605 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 01:52:02.780345 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773838 5605 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 01:52:02.780345 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773841 5605 flags.go:64] FLAG: --enable-server="true" Nov 03 01:52:02.780345 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773844 5605 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 01:52:02.780345 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773847 5605 flags.go:64] FLAG: --event-burst="100" Nov 03 01:52:02.780345 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773850 5605 flags.go:64] FLAG: --event-qps="50" Nov 03 01:52:02.780345 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773853 5605 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 01:52:02.780345 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773856 5605 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 01:52:02.780345 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773859 5605 flags.go:64] FLAG: --eviction-hard="" Nov 03 01:52:02.780951 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773863 5605 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 01:52:02.780951 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773866 5605 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 01:52:02.780951 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773869 5605 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 01:52:02.780951 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773872 5605 flags.go:64] FLAG: --eviction-soft="" Nov 03 01:52:02.780951 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773875 5605 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 01:52:02.780951 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773878 5605 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 01:52:02.780951 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773881 5605 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 01:52:02.780951 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773883 5605 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 01:52:02.780951 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773886 5605 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 01:52:02.780951 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773889 5605 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 01:52:02.780951 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773892 5605 flags.go:64] FLAG: --feature-gates="" Nov 03 01:52:02.780951 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773896 5605 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 01:52:02.780951 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773899 5605 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 01:52:02.780951 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773902 5605 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 01:52:02.780951 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773905 5605 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 01:52:02.780951 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773908 5605 flags.go:64] FLAG: --healthz-port="10248" Nov 03 01:52:02.780951 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773911 5605 flags.go:64] FLAG: --help="false" Nov 03 01:52:02.780951 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773913 5605 flags.go:64] FLAG: --hostname-override="ip-10-0-140-140.us-west-2.compute.internal" Nov 03 01:52:02.780951 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773917 5605 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 01:52:02.780951 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773919 5605 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 01:52:02.780951 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773922 5605 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 01:52:02.780951 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773925 5605 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 01:52:02.780951 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773928 5605 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 01:52:02.781506 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773931 5605 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 01:52:02.781506 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773934 5605 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 01:52:02.781506 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773941 5605 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 01:52:02.781506 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773944 5605 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 01:52:02.781506 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773947 5605 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 01:52:02.781506 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773950 5605 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 01:52:02.781506 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773953 5605 flags.go:64] FLAG: --kube-reserved="" Nov 03 01:52:02.781506 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773956 5605 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 01:52:02.781506 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773958 5605 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 01:52:02.781506 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773963 5605 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 01:52:02.781506 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773966 5605 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 01:52:02.781506 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773969 5605 flags.go:64] FLAG: --lock-file="" Nov 03 01:52:02.781506 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773972 5605 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 01:52:02.781506 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773975 5605 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 01:52:02.781506 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773978 5605 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 01:52:02.781506 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773983 5605 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 01:52:02.781506 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773986 5605 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 01:52:02.781506 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773989 5605 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 01:52:02.781506 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773992 5605 flags.go:64] FLAG: --logging-format="text" Nov 03 01:52:02.781506 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773994 5605 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 01:52:02.781506 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.773998 5605 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 01:52:02.781506 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.774000 5605 flags.go:64] FLAG: --manifest-url="" Nov 03 01:52:02.781506 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.774003 5605 flags.go:64] FLAG: --manifest-url-header="" Nov 03 01:52:02.781506 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.774008 5605 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 01:52:02.781506 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.774011 5605 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 01:52:02.782104 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.774015 5605 flags.go:64] FLAG: --max-pods="110" Nov 03 01:52:02.782104 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.774018 5605 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 01:52:02.782104 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.774021 5605 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 01:52:02.782104 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.774023 5605 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 01:52:02.782104 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.774027 5605 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 01:52:02.782104 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.774030 5605 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 01:52:02.782104 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.774033 5605 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 01:52:02.782104 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.774035 5605 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 01:52:02.782104 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.774042 5605 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 01:52:02.782104 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.774047 5605 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 01:52:02.782104 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.774050 5605 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 01:52:02.782104 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.774053 5605 flags.go:64] FLAG: --pod-cidr="" Nov 03 01:52:02.782104 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.774055 5605 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 01:52:02.782104 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.774058 5605 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 01:52:02.782104 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.774061 5605 flags.go:64] FLAG: --pods-per-core="0" Nov 03 01:52:02.782104 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.774064 5605 flags.go:64] FLAG: --port="10250" Nov 03 01:52:02.782104 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.774069 5605 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 01:52:02.782104 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.774072 5605 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0a53d2985e9b38c8e" Nov 03 01:52:02.782104 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.774075 5605 flags.go:64] FLAG: --qos-reserved="" Nov 03 01:52:02.782104 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.774078 5605 flags.go:64] FLAG: --read-only-port="10255" Nov 03 01:52:02.782104 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.774081 5605 flags.go:64] FLAG: --register-node="true" Nov 03 01:52:02.782104 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.774084 5605 flags.go:64] FLAG: --register-with-taints="" Nov 03 01:52:02.782104 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.774088 5605 flags.go:64] FLAG: --registry-burst="10" Nov 03 01:52:02.782104 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.774091 5605 flags.go:64] FLAG: --registry-qps="5" Nov 03 01:52:02.782104 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.774094 5605 flags.go:64] FLAG: --reserved-cpus="" Nov 03 01:52:02.782104 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.774096 5605 flags.go:64] FLAG: --reserved-memory="" Nov 03 01:52:02.782764 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.774100 5605 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 01:52:02.782764 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.774103 5605 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 01:52:02.782764 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.774106 5605 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 01:52:02.782764 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.774109 5605 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 01:52:02.782764 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.774112 5605 flags.go:64] FLAG: --runonce="false" Nov 03 01:52:02.782764 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.774114 5605 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 01:52:02.782764 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.774118 5605 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 01:52:02.782764 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.774120 5605 flags.go:64] FLAG: --seccomp-default="false" Nov 03 01:52:02.782764 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.774123 5605 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 01:52:02.782764 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.774126 5605 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 01:52:02.782764 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.774129 5605 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 01:52:02.782764 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.774132 5605 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 01:52:02.782764 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.774135 5605 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 01:52:02.782764 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.774138 5605 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 01:52:02.782764 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.774141 5605 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 01:52:02.782764 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.774145 5605 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 01:52:02.782764 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.774148 5605 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 01:52:02.782764 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.774151 5605 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 01:52:02.782764 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.774154 5605 flags.go:64] FLAG: --system-cgroups="" Nov 03 01:52:02.782764 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.774157 5605 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 01:52:02.782764 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.774161 5605 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 01:52:02.782764 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.774164 5605 flags.go:64] FLAG: --tls-cert-file="" Nov 03 01:52:02.782764 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.774171 5605 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 01:52:02.782764 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.774175 5605 flags.go:64] FLAG: --tls-min-version="" Nov 03 01:52:02.782764 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.774178 5605 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 01:52:02.783365 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.774181 5605 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 01:52:02.783365 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.774184 5605 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 01:52:02.783365 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.774187 5605 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 01:52:02.783365 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.774190 5605 flags.go:64] FLAG: --v="2" Nov 03 01:52:02.783365 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.774194 5605 flags.go:64] FLAG: --version="false" Nov 03 01:52:02.783365 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.774198 5605 flags.go:64] FLAG: --vmodule="" Nov 03 01:52:02.783365 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.774202 5605 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 01:52:02.783365 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.774205 5605 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 01:52:02.783365 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774331 5605 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 01:52:02.783365 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774335 5605 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 01:52:02.783365 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774338 5605 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 01:52:02.783365 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774341 5605 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 01:52:02.783365 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774343 5605 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 01:52:02.783365 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774346 5605 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 01:52:02.783365 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774349 5605 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 01:52:02.783365 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774351 5605 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 01:52:02.783365 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774367 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 01:52:02.783365 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774370 5605 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 01:52:02.783365 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774373 5605 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 01:52:02.783365 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774375 5605 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 01:52:02.783365 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774378 5605 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 01:52:02.783872 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774381 5605 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 01:52:02.783872 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774385 5605 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 01:52:02.783872 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774389 5605 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 01:52:02.783872 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774392 5605 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 01:52:02.783872 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774395 5605 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 01:52:02.783872 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774398 5605 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 01:52:02.783872 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774400 5605 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 01:52:02.783872 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774403 5605 feature_gate.go:386] unrecognized feature gate: Example Nov 03 01:52:02.783872 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774407 5605 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 01:52:02.783872 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774409 5605 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 01:52:02.783872 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774412 5605 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 01:52:02.783872 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774414 5605 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 01:52:02.783872 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774418 5605 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 01:52:02.783872 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774420 5605 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 01:52:02.783872 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774423 5605 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 01:52:02.783872 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774425 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 01:52:02.783872 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774428 5605 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 01:52:02.783872 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774430 5605 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 01:52:02.783872 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774434 5605 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 01:52:02.784394 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774438 5605 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 01:52:02.784394 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774441 5605 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 01:52:02.784394 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774444 5605 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 01:52:02.784394 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774447 5605 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 01:52:02.784394 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774449 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 01:52:02.784394 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774452 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 01:52:02.784394 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774454 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 01:52:02.784394 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774457 5605 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 01:52:02.784394 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774459 5605 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 01:52:02.784394 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774462 5605 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 01:52:02.784394 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774465 5605 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 01:52:02.784394 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774467 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 01:52:02.784394 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774470 5605 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 01:52:02.784394 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774472 5605 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 01:52:02.784394 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774476 5605 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 01:52:02.784394 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774479 5605 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 01:52:02.784394 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774481 5605 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 01:52:02.784394 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774483 5605 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 01:52:02.784394 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774486 5605 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 01:52:02.784965 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774489 5605 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 01:52:02.784965 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774491 5605 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 01:52:02.784965 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774495 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 01:52:02.784965 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774498 5605 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 01:52:02.784965 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774500 5605 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 01:52:02.784965 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774503 5605 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 01:52:02.784965 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774505 5605 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 01:52:02.784965 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774508 5605 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 01:52:02.784965 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774511 5605 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 01:52:02.784965 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774513 5605 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 01:52:02.784965 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774516 5605 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 01:52:02.784965 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774519 5605 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 01:52:02.784965 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774521 5605 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 01:52:02.784965 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774524 5605 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 01:52:02.784965 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774526 5605 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 01:52:02.784965 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774528 5605 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 01:52:02.784965 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774531 5605 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 01:52:02.784965 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774533 5605 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 01:52:02.784965 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774536 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 01:52:02.784965 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774538 5605 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 01:52:02.785491 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774541 5605 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 01:52:02.785491 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774543 5605 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 01:52:02.785491 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774546 5605 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 01:52:02.785491 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774549 5605 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 01:52:02.785491 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774551 5605 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 01:52:02.785491 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774553 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 01:52:02.785491 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774556 5605 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 01:52:02.785491 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774560 5605 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 01:52:02.785491 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774563 5605 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 01:52:02.785491 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774565 5605 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 01:52:02.785491 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774568 5605 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 01:52:02.785491 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774570 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 01:52:02.785491 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774573 5605 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 01:52:02.785491 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774576 5605 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 01:52:02.785491 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774579 5605 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 01:52:02.785491 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774581 5605 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 01:52:02.785491 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774584 5605 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 01:52:02.785491 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774586 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 01:52:02.785491 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774589 5605 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 01:52:02.785491 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774592 5605 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 01:52:02.786014 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774594 5605 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 01:52:02.786014 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774597 5605 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 01:52:02.786014 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774599 5605 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 01:52:02.786014 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774602 5605 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 01:52:02.786014 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774605 5605 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 01:52:02.786014 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774607 5605 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 01:52:02.786014 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774610 5605 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 01:52:02.786014 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774612 5605 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 01:52:02.786014 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774615 5605 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 01:52:02.786014 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774617 5605 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 01:52:02.786014 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774620 5605 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 01:52:02.786014 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774622 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 01:52:02.786014 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774624 5605 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 01:52:02.786014 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774627 5605 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 01:52:02.786014 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774629 5605 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 01:52:02.786014 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.774654 5605 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 01:52:02.786427 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774689 5605 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 01:52:02.786427 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774693 5605 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 01:52:02.786427 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774696 5605 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 01:52:02.786427 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774700 5605 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 01:52:02.786427 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774704 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 01:52:02.786427 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774707 5605 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 01:52:02.786427 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774710 5605 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 01:52:02.786427 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774713 5605 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 01:52:02.786427 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774716 5605 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 01:52:02.786427 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774718 5605 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 01:52:02.786427 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774722 5605 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 01:52:02.786427 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774725 5605 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 01:52:02.786427 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774728 5605 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 01:52:02.786427 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774730 5605 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 01:52:02.786427 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774733 5605 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 01:52:02.786427 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774735 5605 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 01:52:02.786427 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774738 5605 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 01:52:02.786427 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774741 5605 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 01:52:02.786427 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774744 5605 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 01:52:02.786427 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774746 5605 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 01:52:02.787108 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774749 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 01:52:02.787108 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774751 5605 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 01:52:02.787108 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774754 5605 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 01:52:02.787108 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774757 5605 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 01:52:02.787108 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774759 5605 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 01:52:02.787108 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774762 5605 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 01:52:02.787108 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774764 5605 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 01:52:02.787108 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774766 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 01:52:02.787108 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774769 5605 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 01:52:02.787108 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774772 5605 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 01:52:02.787108 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774774 5605 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 01:52:02.787108 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774777 5605 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 01:52:02.787108 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774779 5605 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 01:52:02.787108 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774782 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 01:52:02.787108 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774784 5605 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 01:52:02.787108 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774787 5605 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 01:52:02.787108 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774791 5605 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 01:52:02.787108 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774793 5605 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 01:52:02.787108 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774796 5605 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 01:52:02.787108 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774798 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 01:52:02.787739 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774801 5605 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 01:52:02.787739 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774803 5605 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 01:52:02.787739 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774807 5605 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 01:52:02.787739 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774809 5605 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 01:52:02.787739 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774812 5605 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 01:52:02.787739 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774815 5605 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 01:52:02.787739 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774818 5605 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 01:52:02.787739 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774821 5605 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 01:52:02.787739 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774823 5605 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 01:52:02.787739 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774826 5605 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 01:52:02.787739 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774829 5605 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 01:52:02.787739 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774832 5605 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 01:52:02.787739 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774834 5605 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 01:52:02.787739 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774837 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 01:52:02.787739 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774839 5605 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 01:52:02.787739 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774842 5605 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 01:52:02.787739 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774844 5605 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 01:52:02.787739 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774847 5605 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 01:52:02.787739 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774849 5605 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 01:52:02.787739 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774851 5605 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 01:52:02.788284 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774854 5605 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 01:52:02.788284 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774856 5605 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 01:52:02.788284 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774859 5605 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 01:52:02.788284 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774862 5605 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 01:52:02.788284 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774864 5605 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 01:52:02.788284 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774867 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 01:52:02.788284 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774869 5605 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 01:52:02.788284 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774871 5605 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 01:52:02.788284 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774875 5605 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 01:52:02.788284 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774878 5605 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 01:52:02.788284 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774880 5605 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 01:52:02.788284 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774883 5605 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 01:52:02.788284 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774887 5605 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 01:52:02.788284 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774891 5605 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 01:52:02.788284 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774895 5605 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 01:52:02.788284 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774899 5605 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 01:52:02.788284 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774901 5605 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 01:52:02.788284 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774904 5605 feature_gate.go:386] unrecognized feature gate: Example Nov 03 01:52:02.788284 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774906 5605 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 01:52:02.788874 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774909 5605 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 01:52:02.788874 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774911 5605 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 01:52:02.788874 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774914 5605 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 01:52:02.788874 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774916 5605 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 01:52:02.788874 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774919 5605 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 01:52:02.788874 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774922 5605 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 01:52:02.788874 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774925 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 01:52:02.788874 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774928 5605 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 01:52:02.788874 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774930 5605 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 01:52:02.788874 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774934 5605 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 01:52:02.788874 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774938 5605 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 01:52:02.788874 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774941 5605 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 01:52:02.788874 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774944 5605 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 01:52:02.788874 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774946 5605 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 01:52:02.788874 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774949 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 01:52:02.788874 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774952 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 01:52:02.788874 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774954 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 01:52:02.788874 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774956 5605 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 01:52:02.788874 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774959 5605 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 01:52:02.789321 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774962 5605 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 01:52:02.789321 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774964 5605 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 01:52:02.789321 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774968 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 01:52:02.789321 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774971 5605 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 01:52:02.789321 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774973 5605 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 01:52:02.789321 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774976 5605 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 01:52:02.789321 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774978 5605 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 01:52:02.789321 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.774981 5605 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 01:52:02.789321 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.779826 5605 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 01:52:02.789321 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.779843 5605 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 01:52:02.789321 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.779866 5605 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 01:52:02.789321 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.779871 5605 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 01:52:02.789321 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.779943 5605 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 01:52:02.789321 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.779948 5605 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 01:52:02.789321 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.779951 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 01:52:02.789321 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.779954 5605 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 01:52:02.789321 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.779957 5605 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 01:52:02.789321 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.779960 5605 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 01:52:02.789321 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.779963 5605 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 01:52:02.789787 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.779966 5605 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 01:52:02.789787 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.779968 5605 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 01:52:02.789787 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.779971 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 01:52:02.789787 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.779973 5605 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 01:52:02.789787 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.779976 5605 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 01:52:02.789787 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.779980 5605 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 01:52:02.789787 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.779984 5605 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 01:52:02.789787 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.779988 5605 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 01:52:02.789787 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780003 5605 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 01:52:02.789787 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780007 5605 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 01:52:02.789787 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780011 5605 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 01:52:02.789787 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780014 5605 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 01:52:02.789787 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780016 5605 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 01:52:02.789787 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780019 5605 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 01:52:02.789787 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780022 5605 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 01:52:02.789787 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780025 5605 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 01:52:02.789787 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780027 5605 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 01:52:02.789787 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780030 5605 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 01:52:02.789787 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780033 5605 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 01:52:02.790228 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780036 5605 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 01:52:02.790228 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780038 5605 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 01:52:02.790228 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780042 5605 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 01:52:02.790228 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780045 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 01:52:02.790228 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780048 5605 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 01:52:02.790228 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780050 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 01:52:02.790228 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780053 5605 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 01:52:02.790228 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780057 5605 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 01:52:02.790228 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780059 5605 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 01:52:02.790228 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780062 5605 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 01:52:02.790228 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780065 5605 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 01:52:02.790228 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780067 5605 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 01:52:02.790228 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780070 5605 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 01:52:02.790228 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780072 5605 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 01:52:02.790228 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780075 5605 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 01:52:02.790228 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780077 5605 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 01:52:02.790228 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780080 5605 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 01:52:02.790228 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780082 5605 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 01:52:02.790228 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780085 5605 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 01:52:02.790228 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780087 5605 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 01:52:02.790707 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780095 5605 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 01:52:02.790707 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780098 5605 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 01:52:02.790707 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780101 5605 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 01:52:02.790707 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780103 5605 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 01:52:02.790707 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780106 5605 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 01:52:02.790707 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780108 5605 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 01:52:02.790707 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780111 5605 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 01:52:02.790707 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780113 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 01:52:02.790707 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780116 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 01:52:02.790707 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780118 5605 feature_gate.go:386] unrecognized feature gate: Example Nov 03 01:52:02.790707 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780120 5605 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 01:52:02.790707 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780123 5605 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 01:52:02.790707 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780125 5605 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 01:52:02.790707 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780128 5605 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 01:52:02.790707 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780130 5605 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 01:52:02.790707 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780133 5605 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 01:52:02.790707 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780136 5605 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 01:52:02.790707 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780138 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 01:52:02.790707 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780141 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 01:52:02.791215 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780143 5605 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 01:52:02.791215 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780146 5605 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 01:52:02.791215 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780148 5605 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 01:52:02.791215 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780151 5605 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 01:52:02.791215 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780153 5605 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 01:52:02.791215 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780156 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 01:52:02.791215 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780158 5605 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 01:52:02.791215 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780161 5605 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 01:52:02.791215 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780163 5605 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 01:52:02.791215 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780166 5605 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 01:52:02.791215 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780168 5605 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 01:52:02.791215 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780178 5605 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 01:52:02.791215 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780181 5605 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 01:52:02.791215 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780184 5605 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 01:52:02.791215 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780194 5605 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 01:52:02.791215 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780197 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 01:52:02.791215 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780199 5605 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 01:52:02.791215 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780202 5605 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 01:52:02.791215 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780204 5605 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 01:52:02.791215 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780207 5605 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 01:52:02.791857 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780209 5605 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 01:52:02.791857 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780212 5605 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 01:52:02.791857 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780214 5605 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 01:52:02.791857 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780217 5605 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 01:52:02.791857 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780219 5605 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 01:52:02.791857 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780221 5605 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 01:52:02.791857 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780224 5605 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 01:52:02.791857 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780227 5605 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 01:52:02.791857 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780230 5605 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 01:52:02.791857 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780233 5605 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 01:52:02.791857 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780235 5605 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 01:52:02.791857 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780238 5605 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 01:52:02.791857 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780240 5605 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 01:52:02.791857 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780243 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 01:52:02.791857 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780245 5605 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 01:52:02.791857 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780248 5605 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 01:52:02.791857 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780250 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 01:52:02.791857 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780253 5605 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 01:52:02.791857 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780255 5605 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 01:52:02.791857 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780257 5605 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 01:52:02.792616 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780260 5605 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 01:52:02.792616 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.780287 5605 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 01:52:02.792616 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780441 5605 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 01:52:02.792616 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780449 5605 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 01:52:02.792616 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780454 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 01:52:02.792616 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780457 5605 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 01:52:02.792616 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780460 5605 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 01:52:02.792616 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780468 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 01:52:02.792616 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780471 5605 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 01:52:02.792616 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780474 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 01:52:02.792616 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780476 5605 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 01:52:02.792616 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780479 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 01:52:02.792616 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780481 5605 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 01:52:02.792616 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780484 5605 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 01:52:02.792616 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780487 5605 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 01:52:02.793195 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780489 5605 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 01:52:02.793195 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780491 5605 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 01:52:02.793195 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780494 5605 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 01:52:02.793195 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780497 5605 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 01:52:02.793195 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780499 5605 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 01:52:02.793195 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780502 5605 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 01:52:02.793195 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780506 5605 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 01:52:02.793195 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780510 5605 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 01:52:02.793195 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780513 5605 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 01:52:02.793195 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780516 5605 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 01:52:02.793195 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780519 5605 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 01:52:02.793195 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780521 5605 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 01:52:02.793195 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780524 5605 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 01:52:02.793195 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780527 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 01:52:02.793195 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780530 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 01:52:02.793195 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780532 5605 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 01:52:02.793195 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780535 5605 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 01:52:02.793195 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780537 5605 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 01:52:02.793195 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780540 5605 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 01:52:02.793913 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780542 5605 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 01:52:02.793913 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780544 5605 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 01:52:02.793913 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780547 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 01:52:02.793913 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780549 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 01:52:02.793913 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780552 5605 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 01:52:02.793913 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780554 5605 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 01:52:02.793913 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780558 5605 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 01:52:02.793913 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780560 5605 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 01:52:02.793913 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780563 5605 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 01:52:02.793913 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780565 5605 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 01:52:02.793913 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780568 5605 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 01:52:02.793913 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780570 5605 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 01:52:02.793913 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780573 5605 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 01:52:02.793913 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780575 5605 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 01:52:02.793913 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780578 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 01:52:02.793913 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780580 5605 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 01:52:02.793913 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780583 5605 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 01:52:02.793913 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780586 5605 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 01:52:02.793913 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780588 5605 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 01:52:02.794379 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780591 5605 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 01:52:02.794379 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780594 5605 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 01:52:02.794379 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780596 5605 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 01:52:02.794379 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780599 5605 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 01:52:02.794379 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780601 5605 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 01:52:02.794379 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780604 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 01:52:02.794379 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780606 5605 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 01:52:02.794379 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780609 5605 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 01:52:02.794379 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780611 5605 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 01:52:02.794379 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780614 5605 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 01:52:02.794379 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780616 5605 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 01:52:02.794379 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780619 5605 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 01:52:02.794379 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780621 5605 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 01:52:02.794379 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780624 5605 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 01:52:02.794379 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780626 5605 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 01:52:02.794379 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780629 5605 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 01:52:02.794379 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780631 5605 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 01:52:02.794379 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780633 5605 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 01:52:02.794379 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780636 5605 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 01:52:02.794857 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780638 5605 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 01:52:02.794857 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780641 5605 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 01:52:02.794857 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780644 5605 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 01:52:02.794857 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780646 5605 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 01:52:02.794857 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780649 5605 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 01:52:02.794857 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780652 5605 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 01:52:02.794857 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780654 5605 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 01:52:02.794857 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780656 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 01:52:02.794857 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780659 5605 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 01:52:02.794857 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780662 5605 feature_gate.go:386] unrecognized feature gate: Example Nov 03 01:52:02.794857 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780664 5605 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 01:52:02.794857 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780667 5605 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 01:52:02.794857 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780669 5605 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 01:52:02.794857 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780672 5605 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 01:52:02.794857 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780675 5605 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 01:52:02.794857 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780677 5605 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 01:52:02.794857 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780680 5605 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 01:52:02.794857 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780682 5605 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 01:52:02.794857 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780685 5605 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 01:52:02.794857 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780687 5605 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 01:52:02.795312 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780690 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 01:52:02.795312 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780692 5605 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 01:52:02.795312 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780695 5605 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 01:52:02.795312 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780697 5605 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 01:52:02.795312 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780700 5605 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 01:52:02.795312 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780702 5605 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 01:52:02.795312 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780704 5605 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 01:52:02.795312 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780707 5605 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 01:52:02.795312 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780709 5605 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 01:52:02.795312 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780712 5605 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 01:52:02.795312 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780714 5605 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 01:52:02.795312 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780717 5605 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 01:52:02.795312 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780719 5605 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 01:52:02.795312 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780721 5605 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 01:52:02.795312 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780724 5605 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 01:52:02.795312 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780727 5605 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 01:52:02.795744 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.780750 5605 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 01:52:02.795744 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780781 5605 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 01:52:02.795744 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780785 5605 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 01:52:02.795744 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780788 5605 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 01:52:02.795744 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780790 5605 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 01:52:02.795744 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780793 5605 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 01:52:02.795744 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780796 5605 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 01:52:02.795744 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780799 5605 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 01:52:02.795744 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780801 5605 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 01:52:02.795744 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780804 5605 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 01:52:02.795744 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780806 5605 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 01:52:02.795744 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780809 5605 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 01:52:02.795744 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780811 5605 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 01:52:02.795744 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780814 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 01:52:02.795744 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780816 5605 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 01:52:02.795744 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780819 5605 feature_gate.go:386] unrecognized feature gate: Example Nov 03 01:52:02.796119 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780821 5605 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 01:52:02.796119 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780824 5605 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 01:52:02.796119 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780826 5605 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 01:52:02.796119 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780829 5605 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 01:52:02.796119 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780831 5605 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 01:52:02.796119 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780833 5605 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 01:52:02.796119 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780836 5605 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 01:52:02.796119 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780838 5605 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 01:52:02.796119 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780841 5605 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 01:52:02.796119 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780843 5605 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 01:52:02.796119 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780846 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 01:52:02.796119 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780848 5605 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 01:52:02.796119 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780851 5605 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 01:52:02.796119 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780853 5605 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 01:52:02.796119 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780856 5605 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 01:52:02.796119 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780859 5605 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 01:52:02.796119 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780861 5605 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 01:52:02.796119 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780864 5605 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 01:52:02.796119 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780866 5605 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 01:52:02.796119 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780869 5605 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 01:52:02.796665 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780871 5605 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 01:52:02.796665 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780874 5605 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 01:52:02.796665 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780876 5605 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 01:52:02.796665 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780879 5605 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 01:52:02.796665 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780882 5605 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 01:52:02.796665 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780884 5605 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 01:52:02.796665 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780886 5605 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 01:52:02.796665 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780890 5605 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 01:52:02.796665 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780894 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 01:52:02.796665 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780897 5605 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 01:52:02.796665 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780900 5605 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 01:52:02.796665 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780902 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 01:52:02.796665 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780905 5605 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 01:52:02.796665 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780907 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 01:52:02.796665 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780910 5605 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 01:52:02.796665 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780912 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 01:52:02.796665 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780914 5605 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 01:52:02.796665 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780917 5605 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 01:52:02.796665 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780919 5605 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 01:52:02.797202 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780922 5605 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 01:52:02.797202 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780924 5605 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 01:52:02.797202 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780927 5605 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 01:52:02.797202 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780934 5605 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 01:52:02.797202 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780936 5605 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 01:52:02.797202 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780939 5605 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 01:52:02.797202 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780942 5605 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 01:52:02.797202 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780946 5605 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 01:52:02.797202 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780949 5605 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 01:52:02.797202 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780953 5605 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 01:52:02.797202 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780956 5605 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 01:52:02.797202 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780959 5605 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 01:52:02.797202 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780962 5605 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 01:52:02.797202 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780965 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 01:52:02.797202 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780968 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 01:52:02.797202 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780971 5605 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 01:52:02.797202 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780973 5605 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 01:52:02.797202 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780976 5605 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 01:52:02.797202 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780978 5605 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 01:52:02.797759 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780981 5605 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 01:52:02.797759 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780983 5605 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 01:52:02.797759 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780986 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 01:52:02.797759 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780988 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 01:52:02.797759 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780991 5605 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 01:52:02.797759 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780994 5605 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 01:52:02.797759 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780996 5605 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 01:52:02.797759 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.780998 5605 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 01:52:02.797759 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.781001 5605 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 01:52:02.797759 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.781004 5605 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 01:52:02.797759 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.781006 5605 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 01:52:02.797759 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.781009 5605 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 01:52:02.797759 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.781011 5605 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 01:52:02.797759 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.781014 5605 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 01:52:02.797759 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.781017 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 01:52:02.797759 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.781019 5605 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 01:52:02.797759 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.781025 5605 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 01:52:02.797759 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.781028 5605 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 01:52:02.797759 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.781030 5605 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 01:52:02.798203 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.781033 5605 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 01:52:02.798203 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.781036 5605 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 01:52:02.798203 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.781038 5605 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 01:52:02.798203 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.781040 5605 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 01:52:02.798203 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.781043 5605 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 01:52:02.798203 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.781046 5605 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 01:52:02.798203 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.781049 5605 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 01:52:02.798203 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.781051 5605 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 01:52:02.798203 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.781054 5605 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 01:52:02.798203 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.781057 5605 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 01:52:02.798203 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.781059 5605 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 01:52:02.798203 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.781062 5605 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 01:52:02.798203 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.781064 5605 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 01:52:02.798203 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:02.781067 5605 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 01:52:02.798203 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.781116 5605 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 01:52:02.798203 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.782079 5605 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 01:52:02.798203 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.782134 5605 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 01:52:02.798203 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.782301 5605 server.go:1014] "Starting client certificate rotation" Nov 03 01:52:02.798701 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.782455 5605 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 01:52:02.798701 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.782505 5605 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 01:45:49 +0000 UTC" deadline="2028-06-19 22:10:54.362587723 +0000 UTC" Nov 03 01:52:02.798701 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.782540 5605 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14276h18m51.580050863s" Nov 03 01:52:02.798701 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.782802 5605 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 01:52:02.798701 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.783864 5605 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 01:52:02.798701 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.785049 5605 log.go:25] "Validated CRI v1 runtime API" Nov 03 01:52:02.798701 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.786907 5605 log.go:25] "Validated CRI v1 image API" Nov 03 01:52:02.798701 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.787794 5605 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 01:52:02.798701 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.789993 5605 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 9a63c23e-b746-4088-9b39-6cf1e227dc8d:/dev/nvme0n1p3 fcac2184-e10a-43b1-82d5-dd214366aa77:/dev/nvme0n1p4] Nov 03 01:52:02.798956 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.790012 5605 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/04fb7c6e6f597b086dc818c0595aee3f95569ab857b1f8ef044b5d76100b4942/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/04fb7c6e6f597b086dc818c0595aee3f95569ab857b1f8ef044b5d76100b4942/userdata/shm major:0 minor:332 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/19bbae04ef2fe7675bd877cca8a8cac40e2e02ce17e8440de5bf5c040ffd0e83/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/19bbae04ef2fe7675bd877cca8a8cac40e2e02ce17e8440de5bf5c040ffd0e83/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1deb7a08cfcf5ea6b79d48b1886cf9b399b6c4fcc5c2f45bee5545d9a477ec65/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1deb7a08cfcf5ea6b79d48b1886cf9b399b6c4fcc5c2f45bee5545d9a477ec65/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/48fdcde00f2d340a6ea75f58aefdbc17764a92e6033bf2530ab04d45395270ec/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/48fdcde00f2d340a6ea75f58aefdbc17764a92e6033bf2530ab04d45395270ec/userdata/shm major:0 minor:325 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4f7e8ae3dd363ba52fbf24a93910e9605ace951fbbdf6e51650ffac0aafe7a77/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4f7e8ae3dd363ba52fbf24a93910e9605ace951fbbdf6e51650ffac0aafe7a77/userdata/shm major:0 minor:316 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/502e7f3026c2bb58bfb48aae412d650bacf07e07855d86ada27a101dd0326349/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/502e7f3026c2bb58bfb48aae412d650bacf07e07855d86ada27a101dd0326349/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/535f8f7d5b0302230f474cf92cfcadc6365f660f9f7339ab151a81e4890cea8e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/535f8f7d5b0302230f474cf92cfcadc6365f660f9f7339ab151a81e4890cea8e/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/59950987f839d6591cdddd6cab36a4cf1540b371be239279dc6c981d90a53e6a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/59950987f839d6591cdddd6cab36a4cf1540b371be239279dc6c981d90a53e6a/userdata/shm major:0 minor:424 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/69597725f815b11f8be350787f47840c1fcd6e2ffc45c95addf48313ec7cd46d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/69597725f815b11f8be350787f47840c1fcd6e2ffc45c95addf48313ec7cd46d/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6c90f04210adfeedaf7fc82f96407539f7eef4c1ced7c1f75c523b11d27cc513/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6c90f04210adfeedaf7fc82f96407539f7eef4c1ced7c1f75c523b11d27cc513/userdata/shm major:0 minor:260 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6d44cceeee48b2e37bdb5bdef30b12a3578a06e24872cbbcb5fa45a898aab4c0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6d44cceeee48b2e37bdb5bdef30b12a3578a06e24872cbbcb5fa45a898aab4c0/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6e7abde09bd8790efbf067d9fa68b368e263d7f206db973770b8389ef07cbbf0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6e7abde09bd8790efbf067d9fa68b368e263d7f206db973770b8389ef07cbbf0/userdata/shm major:0 minor:83 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7f82c91811ed5d544dec301d9fdf527dfc4c44d87b6967fdcaf42c0d21897209/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7f82c91811ed5d544dec301d9fdf527dfc4c44d87b6967fdcaf42c0d21897209/userdata/shm major:0 minor:341 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/988bc0719feedc4d376b3edfbe25039b9faf76ee6d89c0e2e9ba1ea8ad53553a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/988bc0719feedc4d376b3edfbe25039b9faf76ee6d89c0e2e9ba1ea8ad53553a/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9c5979a417e760236a272153330856b889b67f0f680390ecfa5423d5211d9978/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9c5979a417e760236a272153330856b889b67f0f680390ecfa5423d5211d9978/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b72c4f4a29c37e5578ac2dffd291c96fe82c9cf9453fcba1b9525237a59fa29b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b72c4f4a29c37e5578ac2dffd291c96fe82c9cf9453fcba1b9525237a59fa29b/userdata/shm major:0 minor:347 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bc2201422c99cde1c72025b313b97433d58fce7e26f4b82febbb95752ea56185/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bc2201422c99cde1c72025b313b97433d58fce7e26f4b82febbb95752ea56185/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ce9b5c350d4575ec8ed292d16267ce993d863109ff54133fa02f2f49f41c879a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ce9b5c350d4575ec8ed292d16267ce993d863109ff54133fa02f2f49f41c879a/userdata/shm major:0 minor:348 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d8aa584797783a6981d1de93f65e4b22f1e462cdccc7c73acaa94ca5530a26f4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d8aa584797783a6981d1de93f65e4b22f1e462cdccc7c73acaa94ca5530a26f4/userdata/shm major:0 minor:278 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/da8b4e239c1214ccf41568209cfff8db65166c4b47549e0e45e287564408e64e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/da8b4e239c1214ccf41568209cfff8db65166c4b47549e0e45e287564408e64e/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e6b490f1544909557410be518cb9d934c6f2996cede5301488d9934ab887b297/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e6b490f1544909557410be518cb9d934c6f2996cede5301488d9934ab887b297/userdata/shm major:0 minor:304 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f618f2b28711a3231ea6e344c22c588e935e8fa1b8fb39eda474a018c73fc5ba/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f618f2b28711a3231ea6e344c22c588e935e8fa1b8fb39eda474a018c73fc5ba/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f666cadc2b74c94a27dc69fcae7d613fd3a4651e984de98a8076baf26c9abcbc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f666cadc2b74c94a27dc69fcae7d613fd3a4651e984de98a8076baf26c9abcbc/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fb9d1cd812647e2f4378d1305d22e70c4bc844ac53832a289468b8e7e922b5ea/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fb9d1cd812647e2f4378d1305d22e70c4bc844ac53832a289468b8e7e922b5ea/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0801f13c-d14a-4b9e-b45a-a51af48d17c3/volumes/kubernetes.io~projected/kube-api-access-qqp5v:{mountpoint:/var/lib/kubelet/pods/0801f13c-d14a-4b9e-b45a-a51af48d17c3/volumes/kubernetes.io~projected/kube-api-access-qqp5v major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f363d1e-561d-43ae-9d53-379c86b2a558/volumes/kubernetes.io~projected/kube-api-access-dtzct:{mountpoint:/var/lib/kubelet/pods/0f363d1e-561d-43ae-9d53-379c86b2a558/volumes/kubernetes.io~projected/kube-api-access-dtzct major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f363d1e-561d-43ae-9d53-379c86b2a558/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/0f363d1e-561d-43ae-9d53-379c86b2a558/volumes/kubernetes.io~secret/cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/13fa5796-b3e6-4d1c-9759-88b7094d4b37/volumes/kubernetes.io~projected/kube-api-access-bb4tp:{mountpoint:/var/lib/kubelet/pods/13fa5796-b3e6-4d1c-9759-88b7094d4b37/volumes/kubernetes.io~projected/kube-api-access-bb4tp major:0 minor:331 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/13fa5796-b3e6-4d1c-9759-88b7094d4b37/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/13fa5796-b3e6-4d1c-9759-88b7094d4b37/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/13fa5796-b3e6-4d1c-9759-88b7094d4b37/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/13fa5796-b3e6-4d1c-9759-88b7094d4b37/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/28f00eb6-78c2-4d93-bde5-c1ea12601dbd/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/28f00eb6-78c2-4d93-bde5-c1ea12601dbd/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3bc52103-b80b-49a8-9bed-15b196fdef12/volumes/kubernetes.io~projected/kube-api-access-xk6qs:{mountpoint:/var/lib/kubelet/pods/3bc52103-b80b-49a8-9bed-15b196fdef12/volumes/kubernetes.io~projected/kube-api-access-xk6qs major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3bc52103-b80b-49a8-9bed-15b196fdef12/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/3bc52103-b80b-49a8-9bed-15b196fdef12/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d7d07f7-27c2-489a-8e44-fe366584a267/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/3d7d07f7-27c2-489a-8e44-fe366584a267/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:303 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d7d07f7-27c2-489a-8e44-fe366584a267/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/3d7d07f7-27c2-489a-8e44-fe366584a267/volume-subpaths/web-config/alertmanager/9 major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d7d07f7-27c2-489a-8e44-fe366584a267/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/3d7d07f7-27c2-489a-8e44-fe366584a267/volume-subpaths/web-config/config-reloader/6 major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d7d07f7-27c2-489a-8e44-fe366584a267/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/3d7d07f7-27c2-489a-8e44-fe366584a267/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d7d07f7-27c2-489a-8e44-fe366584a267/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/3d7d07f7-27c2-489a-8e44-fe366584a267/volumes/kubernetes.io~empty-dir/config-out major:0 minor:298 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d7d07f7-27c2-489a-8e44-fe366584a267/volumes/kubernetes.io~projected/kube-api-access-nl9gt:{mountpoint:/var/lib/kubelet/pods/3d7d07f7-27c2-489a-8e44-fe366584a267/volumes/kubernetes.io~projected/kube-api-access-nl9gt major:0 minor:308 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d7d07f7-27c2-489a-8e44-fe366584a267/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/3d7d07f7-27c2-489a-8e44-fe366584a267/volumes/kubernetes.io~projected/tls-assets major:0 minor:301 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d7d07f7-27c2-489a-8e44-fe366584a267/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/3d7d07f7-27c2-489a-8e44-fe366584a267/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:303 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d7d07f7-27c2-489a-8e44-fe366584a267/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/3d7d07f7-27c2-489a-8e44-fe366584a267/volumes/kubernetes.io~secret/config-volume major:0 minor:297 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d7d07f7-27c2-489a-8e44-fe366584a267/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/3d7d07f7-27c2-489a-8e44-fe366584a267/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:302 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d7d07f7-27c2-489a-8e44-fe366584a267/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/3d7d07f7-27c2-489a-8e44-fe366584a267/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d7d07f7-27c2-489a-8e44-fe366584a267/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/3d7d07f7-27c2-489a-8e44-fe366584a267/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d7d07f7-27c2-489a-8e44-fe366584a267/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/3d7d07f7-27c2-489a-8e44-fe366584a267/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:300 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d7d07f7-27c2-489a-8e44-fe366584a267/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/3d7d07f7-27c2-489a-8e44-fe366584a267/volumes/kubernetes.io~secret/web-config major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6/volumes/kubernetes.io~projected/kube-api-access-bmhdf:{mountpoint:/var/lib/kubelet/pods/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6/volumes/kubernetes.io~projected/kube-api-access-bmhdf major:0 minor:259 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:258 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:253 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4480cca6-b2e9-4840-ac8c-a88a40ed90f2/volumes/kubernetes.io~projected/kube-api-access-bkn5x:{mountpoint:/var/lib/kubelet/pods/4480cca6-b2e9-4840-ac8c-a88a40ed90f2/volumes/kubernetes.io~projected/kube-api-access-bkn5x major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4480cca6-b2e9-4840-ac8c-a88a40ed90f2/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/4480cca6-b2e9-4840-ac8c-a88a40ed90f2/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:276 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4480cca6-b2e9-4840-ac8c-a88a40ed90f2/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/4480cca6-b2e9-4840-ac8c-a88a40ed90f2/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:272 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/52b546b3-7bf6-4239-91cb-c8aaca1c9d18/volumes/kubernetes.io~projected/kube-api-access-pqvvj:{mountpoint:/var/lib/kubelet/pods/52b546b3-7bf6-4239-91cb-c8aaca1c9d18/volumes/kubernetes.io~projected/kube-api-access-pqvvj major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/52b546b3-7bf6-4239-91cb-c8aaca1c9d18/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/52b546b3-7bf6-4239-91cb-c8aaca1c9d18/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/56117bce-d686-4c24-98c8-c0ff122778ac/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/56117bce-d686-4c24-98c8-c0ff122778ac/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/56117bce-d686-4c24-98c8-c0ff122778ac/volumes/kubernetes.io~projected/kube-api-access-5kgsp:{mountpoint:/var/lib/kubelet/pods/56117bce-d686-4c24-98c8-c0ff122778ac/volumes/kubernetes.io~projected/kube-api-access-5kgsp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/56117bce-d686-4c24-98c8-c0ff122778ac/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/56117bce-d686-4c24-98c8-c0ff122778ac/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/592adf6b-1edd-4f5e-88a2-6ed92895f22a/volumes/kubernetes.io~projected/kube-api-access-bw55q:{mountpoint:/var/lib/kubelet/pods/592adf6b-1edd-4f5e-88a2-6ed92895f22a/volumes/kubernetes.io~projected/kube-api-access-bw55q major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5d514c70-f1a7-462a-bd3e-6f5bdc016c39/volumes/kubernetes.io~projected/kube-api-access-dhl6w:{mountpoint:/var/lib/kubelet/pods/5d514c70-f1a7-462a-bd3e-6f5bdc016c39/volumes/kubernetes.io~projected/kube-api-access-dhl6w major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5d514c70-f1a7-462a-bd3e-6f5bdc016c39/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/5d514c70-f1a7-462a-bd3e-6f5bdc016c39/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5d514c70-f1a7-462a-bd3e-6f5bdc016c39/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/5d514c70-f1a7-462a-bd3e-6f5bdc016c39/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6ae1437e-8885-42a5-b092-3862cd404c60/volumes/kubernetes.io~projected/kube-api-access-sr2jw:{mountpoint:/var/lib/kubelet/pods/6ae1437e-8885-42a5-b092-3862cd404c60/volumes/kubernetes.io~projected/kube-api-access-sr2jw major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c3f175d-932c-42be-9660-e4fd3dceaf49/volumes/kubernetes.io~projected/kube-api-access-pgvhv:{mountpoint:/var/lib/kubelet/pods/7c3f175d-932c-42be-9660-e4fd3dceaf49/volumes/kubernetes.io~projected/kube-api-access-pgvhv major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7d2438ed-bfc2-4639-b87d-84689a82123e/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/7d2438ed-bfc2-4639-b87d-84689a82123e/volumes/kubernetes.io~secret/agent-certs major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/85148983-aa3f-45c2-a062-da2748dd9011/volumes/kubernetes.io~projected/kube-api-access-2qmrm:{mountpoint:/var/lib/kubelet/pods/85148983-aa3f-45c2-a062-da2748dd9011/volumes/kubernetes.io~projected/kube-api-access-2qmrm major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9152cf36-4997-470e-8902-802a96ebc2c7/volumes/kubernetes.io~projected/kube-api-access-bfgdw:{mountpoint:/var/lib/kubelet/pods/9152cf36-4997-470e-8902-802a96ebc2c7/volumes/kubernetes.io~projected/kube-api-access-bfgdw major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/923b90de-5e45-4add-b092-91af16b7dbad/volumes/kubernetes.io~projected/kube-api-access-l75kg:{mountpoint:/var/lib/kubelet/pods/923b90de-5e45-4add-b092-91af16b7dbad/volumes/kubernetes.io~projected/kube-api-access-l75kg major:0 minor:324 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/923b90de-5e45-4add-b092-91af16b7dbad/volumes/kubernetes.io~secret/federate-client-tls:{mountpoint:/var/lib/kubelet/pods/923b90de-5e45-4add-b092-91af16b7dbad/volumes/kubernetes.io~secret/federate-client-tls major:0 minor:321 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/923b90de-5e45-4add-b092-91af16b7dbad/volumes/kubernetes.io~secret/secret-telemeter-client:{mountpoint:/var/lib/kubelet/pods/923b90de-5e45-4add-b092-91af16b7dbad/volumes/kubernetes.io~secret/secret-telemeter-client major:0 minor:322 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/923b90de-5e45-4add-b092-91af16b7dbad/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/923b90de-5e45-4add-b092-91af16b7dbad/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config major:0 minor:320 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/923b90de-5e45-4add-b092-91af16b7dbad/volumes/kubernetes.io~secret/telemeter-client-tls:{mountpoint:/var/lib/kubelet/pods/923b90de-5e45-4add-b092-91af16b7dbad/volumes/kubernetes.io~secret/telemeter-client-tls major:0 minor:323 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b/volumes/kubernetes.io~projected/kube-api-access-7ccfs:{mountpoint:/var/lib/kubelet/pods/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b/volumes/kubernetes.io~projected/kube-api-access-7ccfs major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4727262-eb15-4453-adf2-0fc6c2358508/volumes/kubernetes.io~projected/kube-api-access-d9kwt:{mountpoint:/var/lib/kubelet/pods/a4727262-eb15-4453-adf2-0fc6c2358508/volumes/kubernetes.io~projected/kube-api-access-d9kwt major:0 minor:423 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4727262-eb15-4453-adf2-0fc6c2358508/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/a4727262-eb15-4453-adf2-0fc6c2358508/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:422 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4727262-eb15-4453-adf2-0fc6c2358508/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/a4727262-eb15-4453-adf2-0fc6c2358508/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:418 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b/volumes/kubernetes.io~projected/kube-api-access-l2l49:{mountpoint:/var/lib/kubelet/pods/dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b/volumes/kubernetes.io~projected/kube-api-access-l2l49 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b/volumes/kubernetes.io~secret/metrics-certs major:0 minor:334 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df76407a-ce41-4a4a-b51e-8420b817f97c/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/df76407a-ce41-4a4a-b51e-8420b817f97c/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df76407a-ce41-4a4a-b51e-8420b817f97c/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/df76407a-ce41-4a4a-b51e-8420b817f97c/volumes/kubernetes.io~empty-dir/tmp major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df76407a-ce41-4a4a-b51e-8420b817f97c/volumes/kubernetes.io~projected/kube-api-access-ffwtk:{mountpoint:/var/lib/kubelet/pods/df76407a-ce41-4a4a-b51e-8420b817f97c/volumes/kubernetes.io~projected/kube-api-access-ffwtk major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e32389d5-705f-4202-98ea-6fc6a5ad5d54/volumes/kubernetes.io~projected/kube-api-access-kv9fz:{mountpoint:/var/lib/kubelet/pods/e32389d5-705f-4202-98ea-6fc6a5ad5d54/volumes/kubernetes.io~projected/kube-api-access-kv9fz major:0 minor:315 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e32389d5-705f-4202-98ea-6fc6a5ad5d54/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/e32389d5-705f-4202-98ea-6fc6a5ad5d54/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e32389d5-705f-4202-98ea-6fc6a5ad5d54/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/e32389d5-705f-4202-98ea-6fc6a5ad5d54/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:312 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e32389d5-705f-4202-98ea-6fc6a5ad5d54/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/e32389d5-705f-4202-98ea-6fc6a5ad5d54/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:313 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/dcdefe06d5c5ebc8d6eed46cff52b0f2934a797dc42ad817030e01360d34c00b/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/6eda1fcc85da022a7129eda5522a0a95b9523fb5ebcb721836e6e1159e2ecbf7/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/04d88fabb8ce84deb5c88429f63c8b2cbb22f0cc5758b9095858fe4276b84ed4/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/9f35146d5de01051b3e43e6738d1b4acfaf3706255793cd6bbc673616ccf2c3f/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/fc29c1089b51bb608f5519f28e66b187ccca0abd06889118bed92bb1c7dd8b3f/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/fcd13707255b88865f0d7019823bb783e8b1cf77b24c9293e9c12db2a48c0d4c/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/bb087ba372bef8318626d9ca22c65df4c55ff4cb94bee323dec54e86d7548a26/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/567bc5baa9f3af4e6f3006316934521cf87e68e7c6fe411354d9ab3ab184d134/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/4bfe4126868d5496cd4033b7192aafe614f74d05d218a1a520226a49977a4712/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/containers/storage/overlay/75d6f80a38735ca6d5a2c4c01cd1659974e23e46d7fd4ed2dd86b72c2888cdd2/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/c13d8b47cc6ad5ac6123d3e864f43d56a29228f0a6242aeecba32768d26909a3/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-148:{mountpoint:/var/lib/containers/storage/overlay/a394eb8b772e1bf6fc3d569caf8265f250628df4aa378b7c634617348d264e4d/merged major:0 minor:148 fsType:overlay blockSize:0} overlay_0-153:{mountpoint:/var/lib/containers/storage/overlay/3f02bd9d85ee362b92a1419ff77b0b3fd30587a0671cec1847715e8ecac22f3a/merged major:0 minor:153 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/608f93629b90cd26997e6e5a87317c11e160a69876f068ee346d8d81ddab00a0/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/d4b8998c919a0c55b2b2936aeb573e888fc32ea52d8c9475636bcba8328a9c97/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/ec7df7b473c7a1941b944d615945d91dd313998237049c259589c38781f4102f/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/5cb2a36223274035cd5355dd73a16a61ee15ac7fea0235a8c854deed601fea08/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/53267d7244a3cdfa2ebea7d4019b699873c50d8f9a56607bb226a4f3c435e80a/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/f78c15afcd72882aa876dcc0ba1db540fcaa9de45839763ed8bda483d64ee4b3/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/26a9faac43ce1c5491152e3914eb39f13b180f46f8a7e99ea9ba0390ac7b8811/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/548135bae16ca930b2c9677213282f544cfae55deddc9178768e13f22bb27808/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/b25b77f0cd911a695fdac3ea81e67f957f9d46362bccb52b619e285a253fe9fb/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/d2fb49481764dc2609e07e78c342bc941ba449ca720b8ce855e8c9558f5e5d3f/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/303d3d55a2df1502be32c349363c2341cdbe0c870b415db2ef83c74df4911eca/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-221:{mountpoint:/var/lib/containers/storage/overlay/7241759002cf55389ff05e6e784949ee4e5f73e51df22510e4c27b3369203d69/merged major:0 minor:221 fsType:overlay blockSize:0} overlay_0-223:{mountpoint:/var/lib/containers/storage/overlay/adbf37ef2aa30510cbbe1c9a97571d24f1105be8f28234746a7e474f7bf9cc5f/merged major:0 minor:223 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/11b0799a08b924e674d7dc2f3b27db8b00b48c459c6c70859125f66901968290/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/ce85d2d5e5f4bd70a27b913d73952ad9619e787cc975fdf2fedf09120dd1287f/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/4f129e86572b24c73d256263d2340a79ec6b5d13588ce3fdeb8f7e27677ac537/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-238:{mountpoint:/var/lib/containers/storage/overlay/3ec768a15a9024b27450375716e3d45969898b72d2fa1a55c3809729f454f78d/merged major:0 minor:238 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/9b3138c34806f62a49f72e5cd299d7700968ced8e8bf26d192066aa9cc4c7248/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/a982c26064e8774ccfbe703de721bd12540aea5e50da63ae55545498c494380c/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/610855db4be52d94c24d778abcf49789272bc8f8c80fa8ad09f46bf4c425bc30/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/900ae302ef10f1cb1f1592c6271971d37ab6f6026487e948235a320028aa8d12/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/c2cceca989a25bbc55f3e69986ef019b416907c3895a7b089b43765d1843e2a5/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/15778c54b03ab9b1cd5deef0cddebb7722ca01ab09e7699b02b17fd630a3a070/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/06211a787ed02d4baf0811cbf86f40727fe4f8035dcfa37edc44b3d099123296/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-290:{mountpoint:/var/lib/containers/storage/overlay/a087c4076bdc782b11b269f2ce4c39dbe2a763ce82229a307d7ee95d761e589c/merged major:0 minor:290 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/6bd284ed659cb1e5bdc7d9e8a70fea28cf64190da1f1188bf1f25abbdd900405/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-318:{mountpoint:/var/lib/containers/storage/overlay/2ee05f5af87c145e4f081aeb3133baca8f0c1f345ed4aa1645866663ebca2c28/merged major:0 minor:318 fsType:overlay blockSize:0} overlay_0-327:{mountpoint:/var/lib/containers/storage/overlay/52e04fd9259f6bc145c35b28f2cd2e0aa4028e4fff09535fbbdaa1549361a602/merged major:0 minor:327 fsType:overlay blockSize:0} overlay_0-337:{mountpoint:/var/lib/containers/storage/overlay/13be57ddc20e4cd94c61c02bc530804760506dd5fd41af8242ca3842667455f0/merged major:0 minor:337 fsType:overlay blockSize:0} overlay_0-339:{mountpoint:/var/lib/containers/storage/overlay/657d88d62755b28f0c20026ff32ca66705cd42b3137d252d617fe1bc8cc9247b/merged major:0 minor:339 fsType:overlay blockSize:0} overlay_0-351:{mountpoint:/var/lib/containers/storage/overlay/7578acf8a0e83bdc4870ede37cee3ad540be13a74f1deac9fc60373e9489e87c/merged major:0 minor:351 fsType:overlay blockSize:0} overlay_0-353:{mountpoint:/var/lib/containers/storage/overlay/5a6afa21a43074fa39790357cf5e3b3ca0df5d3356259889863d8e0706f47016/merged major:0 minor:353 fsType:overlay blockSize:0} overlay_0-359:{mountpoint:/var/lib/containers/storage/overlay/7f0c0f4c65978f41a19fd1868067423f3642a4204c56c500e527754cd184b019/merged major:0 minor:359 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/641bef21fe38759949e126e5d8cace28a0448af9bd23aaf2665e401c46b31a7c/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/52cd620585b9381cd0d19f157d85a0810b0533c4b94de36c6bb2ba3f67efd480/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-367:{mountpoint:/var/lib/containers/storage/overlay/32f857fd2079822c9b19f72267add6397aee25a894412234e31413b1ffe226eb/merged major:0 minor:367 fsType:overlay blockSize:0} overlay_0-369:{mountpoint:/var/lib/containers/storage/overlay/2e2a0f524d9235207d3aaff29f83cbb8475c300040bf2ec6c6384e9551d264a5/merged major:0 minor:369 fsType:overlay blockSize:0} overlay_0-375:{mountpoint:/var/lib/containers/storage/overlay/b963d5f0b80f3eb23d6b956aba27bf95cec18ff6244d7d5cd8c1c848298c4586/merged major:0 minor:375 fsType:overlay blockSize:0} overlay_0-380:{mountpoint:/var/lib/containers/storage/overlay/52ab3e5a8afa868492499499dc249337e19a0a67a74dbe4cc9e2db699822b020/merged major:0 minor:380 fsType:overlay blockSize:0} overlay_0-382:{mountpoint:/var/lib/containers/storage/overlay/43b33f41f9912f94c7bb274dad38a8e249fa3ac1f052a8bf620d76ec8477539c/merged major:0 minor:382 fsType:overlay blockSize:0} overlay_0-391:{mountpoint:/var/lib/containers/storage/overlay/bad668596430090dba01136cc08c36d98993407b4fcb8d1949c9993f0793250c/merged major:0 minor:391 fsType:overlay blockSize:0} overlay_0-396:{mountpoint:/var/lib/containers/storage/overlay/7303f20e045463fe7deebc337a5662febfa2c18d4b68cff78b4001898ee0ddf6/merged major:0 minor:396 fsType:overlay blockSize:0} overlay_0-402:{mountpoint:/var/lib/containers/storage/overlay/f42d0d587b85a109f47e4b559de5c75af08fe78106ab92cf219dd50fef5cb525/merged major:0 minor:402 fsType:overlay blockSize:0} overlay_0-407:{mountpoint:/var/lib/containers/storage/overlay/645ee2c35e4553a8dbca59fdf8b5d81be38ca7a5ac8e54eee8cfe9375aa54a46/merged major:0 minor:407 fsType:overlay blockSize:0} overlay_0-412:{mountpoint:/var/lib/containers/storage/overlay/c1ead120f23d9789e5747630e798f205748f4fd95ddbef4be85808f3f414f688/merged major:0 minor:412 fsType:overlay blockSize:0} overlay_0-426:{mountpoint:/var/lib/containers/storage/overlay/267960e2e9a113b6174fc1f7fc36d60aba1a342ead09b6da8fce8182e0eb193f/merged major:0 minor:426 fsType:overlay blockSize:0} overlay_0-428:{mountpoint:/var/lib/containers/storage/overlay/ce99298c8a3c1b92ba301968c9c20e5781cb0978dbc62d04d2ba30b51ff777a5/merged major:0 minor:428 fsType:overlay blockSize:0} overlay_0-430:{mountpoint:/var/lib/containers/storage/overlay/e738e2dd4469e29e235a31e109c33629ef0ba42bff4173491b8c875658a29833/merged major:0 minor:430 fsType:overlay blockSize:0} overlay_0-436:{mountpoint:/var/lib/containers/storage/overlay/0b46e5558ddaa55143c2370bb762821587b253436d77fa3c367ff7aa0b19fe27/merged major:0 minor:436 fsType:overlay blockSize:0} overlay_0-64:{mountpoint:/var/lib/containers/storage/overlay/8f439199f0d2a96efad6058ef2fc46a5db52d577a2239383d439b19790ed4fb3/merged major:0 minor:64 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/7cd70e5d0278a2c33505bf2ab879c427a35d240e6c1382efdc0d3f51d098a1b1/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/29af7ea21b2047989ddf9d78e2d5d027da5f17d4576f5ba18ebb91ed2b5870fa/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/44f59ba52fb5dc7447dba3f08a135449673fec1531b1cd122dc19486c776ca48/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/2b4226bc4ff61b9453eb968303d738e1975666f12400f0bcd4a58dd49ba82b25/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/4a0f889cb516acc38e2961da88d2c5493ad6d3f0e96b91ca433da2bac6116710/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/897c8cb216d92ec269d91bfdf057573dee99cf41dc5baa0c86819456fe027bec/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/4d6e3d8fca0ff358d01a363e412a84314078d50480a279228ffa6eebb0ba5c80/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/0407f0d338b455a58b6ec6ab19054a030308fdd0c1698b05890161b02f93fce8/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/da670f3bc15d5fcac2d8a40c74281aea9d9e6065e5b4c2b3f606e54048c69354/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 03 01:52:02.799225 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.798813 5605 manager.go:217] Machine: {Timestamp:2026-11-03 01:52:02.798321514 +0000 UTC m=+0.057519658 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3207986 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec241a7425d58df988e7452d3ec16766 SystemUUID:ec241a74-25d5-8df9-88e7-452d3ec16766 BootID:ebcc842b-08f1-43b3-ae33-2284918fea4f Filesystems:[{Device:overlay_0-351 DeviceMajor:0 DeviceMinor:351 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-436 DeviceMajor:0 DeviceMinor:436 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/df76407a-ce41-4a4a-b51e-8420b817f97c/volumes/kubernetes.io~projected/kube-api-access-ffwtk DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/da8b4e239c1214ccf41568209cfff8db65166c4b47549e0e45e287564408e64e/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:258 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-327 DeviceMajor:0 DeviceMinor:327 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-412 DeviceMajor:0 DeviceMinor:412 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a4727262-eb15-4453-adf2-0fc6c2358508/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:418 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/56117bce-d686-4c24-98c8-c0ff122778ac/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:49 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6/volumes/kubernetes.io~projected/kube-api-access-bmhdf DeviceMajor:0 DeviceMinor:259 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-380 DeviceMajor:0 DeviceMinor:380 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-148 DeviceMajor:0 DeviceMinor:148 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-318 DeviceMajor:0 DeviceMinor:318 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b/volumes/kubernetes.io~projected/kube-api-access-7ccfs DeviceMajor:0 DeviceMinor:52 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7c3f175d-932c-42be-9660-e4fd3dceaf49/volumes/kubernetes.io~projected/kube-api-access-pgvhv DeviceMajor:0 DeviceMinor:54 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/df76407a-ce41-4a4a-b51e-8420b817f97c/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:48 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/502e7f3026c2bb58bfb48aae412d650bacf07e07855d86ada27a101dd0326349/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4480cca6-b2e9-4840-ac8c-a88a40ed90f2/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:272 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3d7d07f7-27c2-489a-8e44-fe366584a267/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:299 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6d44cceeee48b2e37bdb5bdef30b12a3578a06e24872cbbcb5fa45a898aab4c0/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-153 DeviceMajor:0 DeviceMinor:153 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bc2201422c99cde1c72025b313b97433d58fce7e26f4b82febbb95752ea56185/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-337 DeviceMajor:0 DeviceMinor:337 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3d7d07f7-27c2-489a-8e44-fe366584a267/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:298 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4f7e8ae3dd363ba52fbf24a93910e9605ace951fbbdf6e51650ffac0aafe7a77/userdata/shm DeviceMajor:0 DeviceMinor:316 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/48fdcde00f2d340a6ea75f58aefdbc17764a92e6033bf2530ab04d45395270ec/userdata/shm DeviceMajor:0 DeviceMinor:325 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-396 DeviceMajor:0 DeviceMinor:396 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-402 DeviceMajor:0 DeviceMinor:402 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-430 DeviceMajor:0 DeviceMinor:430 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6ae1437e-8885-42a5-b092-3862cd404c60/volumes/kubernetes.io~projected/kube-api-access-sr2jw DeviceMajor:0 DeviceMinor:336 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-391 DeviceMajor:0 DeviceMinor:391 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0f363d1e-561d-43ae-9d53-379c86b2a558/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3d7d07f7-27c2-489a-8e44-fe366584a267/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:303 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/13fa5796-b3e6-4d1c-9759-88b7094d4b37/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:329 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3d7d07f7-27c2-489a-8e44-fe366584a267/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:303 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/69597725f815b11f8be350787f47840c1fcd6e2ffc45c95addf48313ec7cd46d/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4480cca6-b2e9-4840-ac8c-a88a40ed90f2/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:276 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b72c4f4a29c37e5578ac2dffd291c96fe82c9cf9453fcba1b9525237a59fa29b/userdata/shm DeviceMajor:0 DeviceMinor:347 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-359 DeviceMajor:0 DeviceMinor:359 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-428 DeviceMajor:0 DeviceMinor:428 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3bc52103-b80b-49a8-9bed-15b196fdef12/volumes/kubernetes.io~projected/kube-api-access-xk6qs DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e32389d5-705f-4202-98ea-6fc6a5ad5d54/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:313 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e32389d5-705f-4202-98ea-6fc6a5ad5d54/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:314 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-367 DeviceMajor:0 DeviceMinor:367 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/52b546b3-7bf6-4239-91cb-c8aaca1c9d18/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/923b90de-5e45-4add-b092-91af16b7dbad/volumes/kubernetes.io~secret/federate-client-tls DeviceMajor:0 DeviceMinor:321 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/923b90de-5e45-4add-b092-91af16b7dbad/volumes/kubernetes.io~secret/telemeter-client-tls DeviceMajor:0 DeviceMinor:323 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6e7abde09bd8790efbf067d9fa68b368e263d7f206db973770b8389ef07cbbf0/userdata/shm DeviceMajor:0 DeviceMinor:83 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5d514c70-f1a7-462a-bd3e-6f5bdc016c39/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:229 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/923b90de-5e45-4add-b092-91af16b7dbad/volumes/kubernetes.io~projected/kube-api-access-l75kg DeviceMajor:0 DeviceMinor:324 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7f82c91811ed5d544dec301d9fdf527dfc4c44d87b6967fdcaf42c0d21897209/userdata/shm DeviceMajor:0 DeviceMinor:341 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-407 DeviceMajor:0 DeviceMinor:407 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3d7d07f7-27c2-489a-8e44-fe366584a267/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:296 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/923b90de-5e45-4add-b092-91af16b7dbad/volumes/kubernetes.io~secret/secret-telemeter-client DeviceMajor:0 DeviceMinor:322 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-382 DeviceMajor:0 DeviceMinor:382 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f618f2b28711a3231ea6e344c22c588e935e8fa1b8fb39eda474a018c73fc5ba/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-221 DeviceMajor:0 DeviceMinor:221 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/df76407a-ce41-4a4a-b51e-8420b817f97c/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:47 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3d7d07f7-27c2-489a-8e44-fe366584a267/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:295 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3d7d07f7-27c2-489a-8e44-fe366584a267/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:302 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b/volumes/kubernetes.io~projected/kube-api-access-l2l49 DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fb9d1cd812647e2f4378d1305d22e70c4bc844ac53832a289468b8e7e922b5ea/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-426 DeviceMajor:0 DeviceMinor:426 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-141 DeviceMajor:0 DeviceMinor:141 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3d7d07f7-27c2-489a-8e44-fe366584a267/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:300 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e6b490f1544909557410be518cb9d934c6f2996cede5301488d9934ab887b297/userdata/shm DeviceMajor:0 DeviceMinor:304 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3d7d07f7-27c2-489a-8e44-fe366584a267/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:301 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/535f8f7d5b0302230f474cf92cfcadc6365f660f9f7339ab151a81e4890cea8e/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/56117bce-d686-4c24-98c8-c0ff122778ac/volumes/kubernetes.io~projected/kube-api-access-5kgsp DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/52b546b3-7bf6-4239-91cb-c8aaca1c9d18/volumes/kubernetes.io~projected/kube-api-access-pqvvj DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/13fa5796-b3e6-4d1c-9759-88b7094d4b37/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:330 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/04fb7c6e6f597b086dc818c0595aee3f95569ab857b1f8ef044b5d76100b4942/userdata/shm DeviceMajor:0 DeviceMinor:332 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/59950987f839d6591cdddd6cab36a4cf1540b371be239279dc6c981d90a53e6a/userdata/shm DeviceMajor:0 DeviceMinor:424 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/85148983-aa3f-45c2-a062-da2748dd9011/volumes/kubernetes.io~projected/kube-api-access-2qmrm DeviceMajor:0 DeviceMinor:53 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:253 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6c90f04210adfeedaf7fc82f96407539f7eef4c1ced7c1f75c523b11d27cc513/userdata/shm DeviceMajor:0 DeviceMinor:260 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-290 DeviceMajor:0 DeviceMinor:290 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a4727262-eb15-4453-adf2-0fc6c2358508/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:422 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-369 DeviceMajor:0 DeviceMinor:369 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a4727262-eb15-4453-adf2-0fc6c2358508/volumes/kubernetes.io~projected/kube-api-access-d9kwt DeviceMajor:0 DeviceMinor:423 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0f363d1e-561d-43ae-9d53-379c86b2a558/volumes/kubernetes.io~projected/kube-api-access-dtzct DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3d7d07f7-27c2-489a-8e44-fe366584a267/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:296 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3d7d07f7-27c2-489a-8e44-fe366584a267/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:296 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9c5979a417e760236a272153330856b889b67f0f680390ecfa5423d5211d9978/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3d7d07f7-27c2-489a-8e44-fe366584a267/volumes/kubernetes.io~projected/kube-api-access-nl9gt DeviceMajor:0 DeviceMinor:308 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ce9b5c350d4575ec8ed292d16267ce993d863109ff54133fa02f2f49f41c879a/userdata/shm DeviceMajor:0 DeviceMinor:348 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-353 DeviceMajor:0 DeviceMinor:353 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-375 DeviceMajor:0 DeviceMinor:375 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/988bc0719feedc4d376b3edfbe25039b9faf76ee6d89c0e2e9ba1ea8ad53553a/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f666cadc2b74c94a27dc69fcae7d613fd3a4651e984de98a8076baf26c9abcbc/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7d2438ed-bfc2-4639-b87d-84689a82123e/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:50 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5d514c70-f1a7-462a-bd3e-6f5bdc016c39/volumes/kubernetes.io~projected/kube-api-access-dhl6w DeviceMajor:0 DeviceMinor:230 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9152cf36-4997-470e-8902-802a96ebc2c7/volumes/kubernetes.io~projected/kube-api-access-bfgdw DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e32389d5-705f-4202-98ea-6fc6a5ad5d54/volumes/kubernetes.io~projected/kube-api-access-kv9fz DeviceMajor:0 DeviceMinor:315 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/13fa5796-b3e6-4d1c-9759-88b7094d4b37/volumes/kubernetes.io~projected/kube-api-access-bb4tp DeviceMajor:0 DeviceMinor:331 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/28f00eb6-78c2-4d93-bde5-c1ea12601dbd/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:335 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3d7d07f7-27c2-489a-8e44-fe366584a267/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:297 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0801f13c-d14a-4b9e-b45a-a51af48d17c3/volumes/kubernetes.io~projected/kube-api-access-qqp5v DeviceMajor:0 DeviceMinor:51 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/592adf6b-1edd-4f5e-88a2-6ed92895f22a/volumes/kubernetes.io~projected/kube-api-access-bw55q DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-64 DeviceMajor:0 DeviceMinor:64 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/56117bce-d686-4c24-98c8-c0ff122778ac/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/3bc52103-b80b-49a8-9bed-15b196fdef12/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5d514c70-f1a7-462a-bd3e-6f5bdc016c39/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:225 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3d7d07f7-27c2-489a-8e44-fe366584a267/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:296 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-238 DeviceMajor:0 DeviceMinor:238 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e32389d5-705f-4202-98ea-6fc6a5ad5d54/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:312 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/923b90de-5e45-4add-b092-91af16b7dbad/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:320 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1deb7a08cfcf5ea6b79d48b1886cf9b399b6c4fcc5c2f45bee5545d9a477ec65/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-223 DeviceMajor:0 DeviceMinor:223 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:334 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-339 DeviceMajor:0 DeviceMinor:339 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/19bbae04ef2fe7675bd877cca8a8cac40e2e02ce17e8440de5bf5c040ffd0e83/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4480cca6-b2e9-4840-ac8c-a88a40ed90f2/volumes/kubernetes.io~projected/kube-api-access-bkn5x DeviceMajor:0 DeviceMinor:277 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d8aa584797783a6981d1de93f65e4b22f1e462cdccc7c73acaa94ca5530a26f4/userdata/shm DeviceMajor:0 DeviceMinor:278 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:04fb7c6e6f597b0 MacAddress:3e:76:aa:6b:cf:3c Speed:10000 Mtu:8901} {Name:1deb7a08cfcf5ea MacAddress:46:12:95:ad:f3:f5 Speed:10000 Mtu:8901} {Name:48fdcde00f2d340 MacAddress:e6:3c:d7:5a:9b:be Speed:10000 Mtu:8901} {Name:4f7e8ae3dd363ba MacAddress:7a:e5:e5:20:27:a2 Speed:10000 Mtu:8901} {Name:502e7f3026c2bb5 MacAddress:5a:af:db:2b:c2:47 Speed:10000 Mtu:8901} {Name:59950987f839d65 MacAddress:06:a3:2f:2f:a7:4f Speed:10000 Mtu:8901} {Name:6c90f04210adfee MacAddress:2a:c7:35:9e:b5:cd Speed:10000 Mtu:8901} {Name:7f82c91811ed5d5 MacAddress:26:45:a0:ae:46:21 Speed:10000 Mtu:8901} {Name:9c5979a417e7602 MacAddress:ce:ac:ac:a7:84:d3 Speed:10000 Mtu:8901} {Name:b72c4f4a29c37e5 MacAddress:8e:de:2c:a9:b3:ee Speed:10000 Mtu:8901} {Name:bc2201422c99cde MacAddress:aa:85:6d:e8:6d:78 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:e1:74:bd:49 Speed:0 Mtu:9001} {Name:br-int MacAddress:d6:9d:cd:1f:4d:76 Speed:0 Mtu:8901} {Name:ce9b5c350d4575e MacAddress:f2:f8:4e:e1:06:47 Speed:10000 Mtu:8901} {Name:e6b490f15449095 MacAddress:d6:7f:78:be:52:d4 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:e1:74:bd:49 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:e6:1f:2e:54:30:77 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:92:41:f2:20:eb:eb Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 01:52:02.799447 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.799229 5605 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 01:52:02.799447 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.799272 5605 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 01:52:02.799652 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.799628 5605 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 01:52:02.799771 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.799652 5605 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-140.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 01:52:02.799847 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.799774 5605 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 01:52:02.799847 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.799782 5605 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 01:52:02.799847 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.799792 5605 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 01:52:02.799847 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.799811 5605 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 01:52:02.799947 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.799893 5605 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 01:52:02.800072 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.800062 5605 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 01:52:02.800175 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.800166 5605 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 01:52:02.800209 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.800201 5605 kubelet.go:493] "Attempting to sync node with API server" Nov 03 01:52:02.800236 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.800217 5605 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 01:52:02.800236 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.800233 5605 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 01:52:02.800286 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.800241 5605 kubelet.go:405] "Adding apiserver pod source" Nov 03 01:52:02.800286 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.800249 5605 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 01:52:02.801186 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.801170 5605 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 01:52:02.801186 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.801185 5605 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 01:52:02.801720 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.801700 5605 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 01:52:02.805656 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.805641 5605 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 01:52:02.805726 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.805655 5605 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 01:52:02.805772 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.805755 5605 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 01:52:02.806314 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.806302 5605 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 01:52:02.806350 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.806330 5605 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 01:52:02.806442 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.806433 5605 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 01:52:02.806481 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.806447 5605 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 01:52:02.806481 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.806453 5605 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 01:52:02.806481 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.806458 5605 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 01:52:02.806481 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.806464 5605 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 01:52:02.806481 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.806469 5605 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 01:52:02.806481 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.806475 5605 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 01:52:02.806481 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.806480 5605 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 01:52:02.806751 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.806487 5605 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 01:52:02.806751 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.806493 5605 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 01:52:02.806751 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.806501 5605 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 01:52:02.806751 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.806509 5605 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 01:52:02.806751 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.806541 5605 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 01:52:02.806751 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.806546 5605 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 01:52:02.810474 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.810447 5605 server.go:1263] "Started kubelet" Nov 03 01:52:02.810566 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.810504 5605 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 01:52:02.810663 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.810591 5605 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 01:52:02.810783 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.810693 5605 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 01:52:02.811012 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.810975 5605 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 01:52:02.810994 ip-10-0-140-140 systemd[1]: Started Kubernetes Kubelet. Nov 03 01:52:02.811843 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.811826 5605 server.go:317] "Adding debug handlers to kubelet server" Nov 03 01:52:02.813837 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.813822 5605 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 01:52:02.814036 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.814018 5605 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 01:52:02.814161 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.814144 5605 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 01:45:50 +0000 UTC" deadline="2028-04-01 07:01:45.37188067 +0000 UTC" Nov 03 01:52:02.814249 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.814164 5605 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12365h9m42.557720277s" Nov 03 01:52:02.814249 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.814199 5605 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 01:52:02.814249 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.814205 5605 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 01:52:02.814485 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.814296 5605 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 01:52:02.814600 ip-10-0-140-140 kubenswrapper[5605]: E1103 01:52:02.814541 5605 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-140.us-west-2.compute.internal\" not found" Nov 03 01:52:02.815682 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.815596 5605 factory.go:55] Registering systemd factory Nov 03 01:52:02.815682 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.815632 5605 factory.go:223] Registration of the systemd container factory successfully Nov 03 01:52:02.816714 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.816636 5605 factory.go:153] Registering CRI-O factory Nov 03 01:52:02.816714 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.816653 5605 factory.go:223] Registration of the crio container factory successfully Nov 03 01:52:02.817028 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.816950 5605 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 01:52:02.817028 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.816979 5605 factory.go:103] Registering Raw factory Nov 03 01:52:02.817028 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.816995 5605 manager.go:1199] Started watching for new ooms in manager Nov 03 01:52:02.817266 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.817222 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3bc52103-b80b-49a8-9bed-15b196fdef12" volumeName="kubernetes.io/secret/3bc52103-b80b-49a8-9bed-15b196fdef12-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 03 01:52:02.817266 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.817259 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56117bce-d686-4c24-98c8-c0ff122778ac" volumeName="kubernetes.io/projected/56117bce-d686-4c24-98c8-c0ff122778ac-kube-api-access-5kgsp" seLinuxMountContext="" Nov 03 01:52:02.818972 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.817272 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="987bbde2-6a6a-44fc-b5e4-b0197bcecd8b" volumeName="kubernetes.io/projected/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-kube-api-access-7ccfs" seLinuxMountContext="" Nov 03 01:52:02.818972 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.817285 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13fa5796-b3e6-4d1c-9759-88b7094d4b37" volumeName="kubernetes.io/configmap/13fa5796-b3e6-4d1c-9759-88b7094d4b37-service-ca" seLinuxMountContext="" Nov 03 01:52:02.818972 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.817305 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13fa5796-b3e6-4d1c-9759-88b7094d4b37" volumeName="kubernetes.io/empty-dir/13fa5796-b3e6-4d1c-9759-88b7094d4b37-tmp" seLinuxMountContext="" Nov 03 01:52:02.818972 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.817316 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6" volumeName="kubernetes.io/empty-dir/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-tmp" seLinuxMountContext="" Nov 03 01:52:02.818972 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.817327 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="52b546b3-7bf6-4239-91cb-c8aaca1c9d18" volumeName="kubernetes.io/projected/52b546b3-7bf6-4239-91cb-c8aaca1c9d18-kube-api-access-pqvvj" seLinuxMountContext="" Nov 03 01:52:02.818972 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.817338 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="923b90de-5e45-4add-b092-91af16b7dbad" volumeName="kubernetes.io/projected/923b90de-5e45-4add-b092-91af16b7dbad-kube-api-access-l75kg" seLinuxMountContext="" Nov 03 01:52:02.818972 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.817351 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0801f13c-d14a-4b9e-b45a-a51af48d17c3" volumeName="kubernetes.io/projected/0801f13c-d14a-4b9e-b45a-a51af48d17c3-kube-api-access-qqp5v" seLinuxMountContext="" Nov 03 01:52:02.818972 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.817376 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d7d07f7-27c2-489a-8e44-fe366584a267" volumeName="kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-web-config" seLinuxMountContext="" Nov 03 01:52:02.818972 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.817388 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f363d1e-561d-43ae-9d53-379c86b2a558" volumeName="kubernetes.io/secret/0f363d1e-561d-43ae-9d53-379c86b2a558-cert" seLinuxMountContext="" Nov 03 01:52:02.818972 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.817399 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6" volumeName="kubernetes.io/projected/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-kube-api-access-bmhdf" seLinuxMountContext="" Nov 03 01:52:02.818972 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.817411 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d2438ed-bfc2-4639-b87d-84689a82123e" volumeName="kubernetes.io/configmap/7d2438ed-bfc2-4639-b87d-84689a82123e-konnectivity-ca" seLinuxMountContext="" Nov 03 01:52:02.818972 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.817431 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4727262-eb15-4453-adf2-0fc6c2358508" volumeName="kubernetes.io/configmap/a4727262-eb15-4453-adf2-0fc6c2358508-trusted-ca-bundle" seLinuxMountContext="" Nov 03 01:52:02.818972 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.817447 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e32389d5-705f-4202-98ea-6fc6a5ad5d54" volumeName="kubernetes.io/projected/e32389d5-705f-4202-98ea-6fc6a5ad5d54-kube-api-access-kv9fz" seLinuxMountContext="" Nov 03 01:52:02.818972 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.817459 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13fa5796-b3e6-4d1c-9759-88b7094d4b37" volumeName="kubernetes.io/configmap/13fa5796-b3e6-4d1c-9759-88b7094d4b37-console-config" seLinuxMountContext="" Nov 03 01:52:02.818972 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.817470 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d7d07f7-27c2-489a-8e44-fe366584a267" volumeName="kubernetes.io/empty-dir/3d7d07f7-27c2-489a-8e44-fe366584a267-alertmanager-main-db" seLinuxMountContext="" Nov 03 01:52:02.818972 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.817480 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="52b546b3-7bf6-4239-91cb-c8aaca1c9d18" volumeName="kubernetes.io/configmap/52b546b3-7bf6-4239-91cb-c8aaca1c9d18-config-volume" seLinuxMountContext="" Nov 03 01:52:02.818972 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.817492 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c3f175d-932c-42be-9660-e4fd3dceaf49" volumeName="kubernetes.io/empty-dir/7c3f175d-932c-42be-9660-e4fd3dceaf49-tmp-dir" seLinuxMountContext="" Nov 03 01:52:02.818972 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.817505 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9152cf36-4997-470e-8902-802a96ebc2c7" volumeName="kubernetes.io/projected/9152cf36-4997-470e-8902-802a96ebc2c7-kube-api-access-bfgdw" seLinuxMountContext="" Nov 03 01:52:02.818972 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.817523 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="987bbde2-6a6a-44fc-b5e4-b0197bcecd8b" volumeName="kubernetes.io/configmap/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-multus-daemon-config" seLinuxMountContext="" Nov 03 01:52:02.818972 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.817535 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df76407a-ce41-4a4a-b51e-8420b817f97c" volumeName="kubernetes.io/empty-dir/df76407a-ce41-4a4a-b51e-8420b817f97c-tmp" seLinuxMountContext="" Nov 03 01:52:02.818972 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.817547 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df76407a-ce41-4a4a-b51e-8420b817f97c" volumeName="kubernetes.io/projected/df76407a-ce41-4a4a-b51e-8420b817f97c-kube-api-access-ffwtk" seLinuxMountContext="" Nov 03 01:52:02.818972 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.817559 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d7d07f7-27c2-489a-8e44-fe366584a267" volumeName="kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Nov 03 01:52:02.818972 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.817573 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f363d1e-561d-43ae-9d53-379c86b2a558" volumeName="kubernetes.io/projected/0f363d1e-561d-43ae-9d53-379c86b2a558-kube-api-access-dtzct" seLinuxMountContext="" Nov 03 01:52:02.818972 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.817587 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d7d07f7-27c2-489a-8e44-fe366584a267" volumeName="kubernetes.io/configmap/3d7d07f7-27c2-489a-8e44-fe366584a267-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Nov 03 01:52:02.818972 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.817600 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d7d07f7-27c2-489a-8e44-fe366584a267" volumeName="kubernetes.io/projected/3d7d07f7-27c2-489a-8e44-fe366584a267-tls-assets" seLinuxMountContext="" Nov 03 01:52:02.818972 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.817610 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="987bbde2-6a6a-44fc-b5e4-b0197bcecd8b" volumeName="kubernetes.io/configmap/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-cni-binary-copy" seLinuxMountContext="" Nov 03 01:52:02.818972 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.817626 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13fa5796-b3e6-4d1c-9759-88b7094d4b37" volumeName="kubernetes.io/configmap/13fa5796-b3e6-4d1c-9759-88b7094d4b37-oauth-serving-cert" seLinuxMountContext="" Nov 03 01:52:02.818972 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.817637 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6" volumeName="kubernetes.io/secret/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-console-serving-cert" seLinuxMountContext="" Nov 03 01:52:02.818972 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.817649 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56117bce-d686-4c24-98c8-c0ff122778ac" volumeName="kubernetes.io/secret/56117bce-d686-4c24-98c8-c0ff122778ac-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 01:52:02.818972 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.817676 5605 manager.go:319] Starting recovery of all containers Nov 03 01:52:02.818972 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.817676 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d514c70-f1a7-462a-bd3e-6f5bdc016c39" volumeName="kubernetes.io/configmap/5d514c70-f1a7-462a-bd3e-6f5bdc016c39-metrics-client-ca" seLinuxMountContext="" Nov 03 01:52:02.818972 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.818407 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c3f175d-932c-42be-9660-e4fd3dceaf49" volumeName="kubernetes.io/projected/7c3f175d-932c-42be-9660-e4fd3dceaf49-kube-api-access-pgvhv" seLinuxMountContext="" Nov 03 01:52:02.818972 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.818461 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4727262-eb15-4453-adf2-0fc6c2358508" volumeName="kubernetes.io/secret/a4727262-eb15-4453-adf2-0fc6c2358508-console-serving-cert" seLinuxMountContext="" Nov 03 01:52:02.818972 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.818475 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e32389d5-705f-4202-98ea-6fc6a5ad5d54" volumeName="kubernetes.io/secret/e32389d5-705f-4202-98ea-6fc6a5ad5d54-client-ca-bundle" seLinuxMountContext="" Nov 03 01:52:02.818972 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.818496 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="28f00eb6-78c2-4d93-bde5-c1ea12601dbd" volumeName="kubernetes.io/secret/28f00eb6-78c2-4d93-bde5-c1ea12601dbd-original-pull-secret" seLinuxMountContext="" Nov 03 01:52:02.818972 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.818507 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6" volumeName="kubernetes.io/configmap/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-service-ca" seLinuxMountContext="" Nov 03 01:52:02.818972 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.818518 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4480cca6-b2e9-4840-ac8c-a88a40ed90f2" volumeName="kubernetes.io/secret/4480cca6-b2e9-4840-ac8c-a88a40ed90f2-node-exporter-tls" seLinuxMountContext="" Nov 03 01:52:02.818972 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.818539 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="52b546b3-7bf6-4239-91cb-c8aaca1c9d18" volumeName="kubernetes.io/empty-dir/52b546b3-7bf6-4239-91cb-c8aaca1c9d18-tmp-dir" seLinuxMountContext="" Nov 03 01:52:02.818972 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.818550 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ae1437e-8885-42a5-b092-3862cd404c60" volumeName="kubernetes.io/projected/6ae1437e-8885-42a5-b092-3862cd404c60-kube-api-access-sr2jw" seLinuxMountContext="" Nov 03 01:52:02.818972 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.818562 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85148983-aa3f-45c2-a062-da2748dd9011" volumeName="kubernetes.io/projected/85148983-aa3f-45c2-a062-da2748dd9011-kube-api-access-2qmrm" seLinuxMountContext="" Nov 03 01:52:02.818972 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.818590 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9152cf36-4997-470e-8902-802a96ebc2c7" volumeName="kubernetes.io/configmap/9152cf36-4997-470e-8902-802a96ebc2c7-cni-binary-copy" seLinuxMountContext="" Nov 03 01:52:02.818972 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.818602 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e32389d5-705f-4202-98ea-6fc6a5ad5d54" volumeName="kubernetes.io/configmap/e32389d5-705f-4202-98ea-6fc6a5ad5d54-metrics-server-audit-profiles" seLinuxMountContext="" Nov 03 01:52:02.818972 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.818613 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13fa5796-b3e6-4d1c-9759-88b7094d4b37" volumeName="kubernetes.io/secret/13fa5796-b3e6-4d1c-9759-88b7094d4b37-console-oauth-config" seLinuxMountContext="" Nov 03 01:52:02.818972 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.818632 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d7d07f7-27c2-489a-8e44-fe366584a267" volumeName="kubernetes.io/configmap/3d7d07f7-27c2-489a-8e44-fe366584a267-metrics-client-ca" seLinuxMountContext="" Nov 03 01:52:02.818972 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.818643 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4480cca6-b2e9-4840-ac8c-a88a40ed90f2" volumeName="kubernetes.io/projected/4480cca6-b2e9-4840-ac8c-a88a40ed90f2-kube-api-access-bkn5x" seLinuxMountContext="" Nov 03 01:52:02.818972 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.818656 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4480cca6-b2e9-4840-ac8c-a88a40ed90f2" volumeName="kubernetes.io/secret/4480cca6-b2e9-4840-ac8c-a88a40ed90f2-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 01:52:02.818972 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.818675 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d514c70-f1a7-462a-bd3e-6f5bdc016c39" volumeName="kubernetes.io/projected/5d514c70-f1a7-462a-bd3e-6f5bdc016c39-kube-api-access-dhl6w" seLinuxMountContext="" Nov 03 01:52:02.818972 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.818696 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9152cf36-4997-470e-8902-802a96ebc2c7" volumeName="kubernetes.io/configmap/9152cf36-4997-470e-8902-802a96ebc2c7-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 01:52:02.818972 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.818709 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="923b90de-5e45-4add-b092-91af16b7dbad" volumeName="kubernetes.io/secret/923b90de-5e45-4add-b092-91af16b7dbad-secret-telemeter-client" seLinuxMountContext="" Nov 03 01:52:02.818972 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.818721 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4727262-eb15-4453-adf2-0fc6c2358508" volumeName="kubernetes.io/configmap/a4727262-eb15-4453-adf2-0fc6c2358508-oauth-serving-cert" seLinuxMountContext="" Nov 03 01:52:02.818972 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.818731 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d7d07f7-27c2-489a-8e44-fe366584a267" volumeName="kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Nov 03 01:52:02.818972 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.818742 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d7d07f7-27c2-489a-8e44-fe366584a267" volumeName="kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Nov 03 01:52:02.818972 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.818755 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13fa5796-b3e6-4d1c-9759-88b7094d4b37" volumeName="kubernetes.io/configmap/13fa5796-b3e6-4d1c-9759-88b7094d4b37-trusted-ca-bundle" seLinuxMountContext="" Nov 03 01:52:02.818972 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.818766 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13fa5796-b3e6-4d1c-9759-88b7094d4b37" volumeName="kubernetes.io/projected/13fa5796-b3e6-4d1c-9759-88b7094d4b37-kube-api-access-bb4tp" seLinuxMountContext="" Nov 03 01:52:02.818972 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.818778 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6" volumeName="kubernetes.io/configmap/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-console-config" seLinuxMountContext="" Nov 03 01:52:02.818972 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.818800 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4480cca6-b2e9-4840-ac8c-a88a40ed90f2" volumeName="kubernetes.io/empty-dir/4480cca6-b2e9-4840-ac8c-a88a40ed90f2-node-exporter-textfile" seLinuxMountContext="" Nov 03 01:52:02.818972 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.818812 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85148983-aa3f-45c2-a062-da2748dd9011" volumeName="kubernetes.io/configmap/85148983-aa3f-45c2-a062-da2748dd9011-iptables-alerter-script" seLinuxMountContext="" Nov 03 01:52:02.818972 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.818822 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="923b90de-5e45-4add-b092-91af16b7dbad" volumeName="kubernetes.io/configmap/923b90de-5e45-4add-b092-91af16b7dbad-serving-certs-ca-bundle" seLinuxMountContext="" Nov 03 01:52:02.818972 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.818833 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3bc52103-b80b-49a8-9bed-15b196fdef12" volumeName="kubernetes.io/empty-dir/3bc52103-b80b-49a8-9bed-15b196fdef12-data-volume" seLinuxMountContext="" Nov 03 01:52:02.818972 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.818844 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d7d07f7-27c2-489a-8e44-fe366584a267" volumeName="kubernetes.io/empty-dir/3d7d07f7-27c2-489a-8e44-fe366584a267-config-out" seLinuxMountContext="" Nov 03 01:52:02.818972 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.818854 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d7d07f7-27c2-489a-8e44-fe366584a267" volumeName="kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-cluster-tls-config" seLinuxMountContext="" Nov 03 01:52:02.818972 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.818867 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d2438ed-bfc2-4639-b87d-84689a82123e" volumeName="kubernetes.io/secret/7d2438ed-bfc2-4639-b87d-84689a82123e-agent-certs" seLinuxMountContext="" Nov 03 01:52:02.822380 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.818879 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9152cf36-4997-470e-8902-802a96ebc2c7" volumeName="kubernetes.io/configmap/9152cf36-4997-470e-8902-802a96ebc2c7-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 01:52:02.822380 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.822232 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="923b90de-5e45-4add-b092-91af16b7dbad" volumeName="kubernetes.io/secret/923b90de-5e45-4add-b092-91af16b7dbad-federate-client-tls" seLinuxMountContext="" Nov 03 01:52:02.822380 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.822257 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="923b90de-5e45-4add-b092-91af16b7dbad" volumeName="kubernetes.io/secret/923b90de-5e45-4add-b092-91af16b7dbad-telemeter-client-tls" seLinuxMountContext="" Nov 03 01:52:02.822380 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.822267 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4727262-eb15-4453-adf2-0fc6c2358508" volumeName="kubernetes.io/configmap/a4727262-eb15-4453-adf2-0fc6c2358508-service-ca" seLinuxMountContext="" Nov 03 01:52:02.822380 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.822280 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6" volumeName="kubernetes.io/secret/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-console-oauth-config" seLinuxMountContext="" Nov 03 01:52:02.822380 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.822289 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4727262-eb15-4453-adf2-0fc6c2358508" volumeName="kubernetes.io/configmap/a4727262-eb15-4453-adf2-0fc6c2358508-console-config" seLinuxMountContext="" Nov 03 01:52:02.822380 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.822302 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4727262-eb15-4453-adf2-0fc6c2358508" volumeName="kubernetes.io/projected/a4727262-eb15-4453-adf2-0fc6c2358508-kube-api-access-d9kwt" seLinuxMountContext="" Nov 03 01:52:02.822380 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.822311 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4727262-eb15-4453-adf2-0fc6c2358508" volumeName="kubernetes.io/secret/a4727262-eb15-4453-adf2-0fc6c2358508-console-oauth-config" seLinuxMountContext="" Nov 03 01:52:02.822380 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.822319 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df76407a-ce41-4a4a-b51e-8420b817f97c" volumeName="kubernetes.io/empty-dir/df76407a-ce41-4a4a-b51e-8420b817f97c-etc-tuned" seLinuxMountContext="" Nov 03 01:52:02.822380 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.822330 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e32389d5-705f-4202-98ea-6fc6a5ad5d54" volumeName="kubernetes.io/secret/e32389d5-705f-4202-98ea-6fc6a5ad5d54-secret-metrics-server-tls" seLinuxMountContext="" Nov 03 01:52:02.822380 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.822338 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3bc52103-b80b-49a8-9bed-15b196fdef12" volumeName="kubernetes.io/configmap/3bc52103-b80b-49a8-9bed-15b196fdef12-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 03 01:52:02.822380 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.822350 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4480cca6-b2e9-4840-ac8c-a88a40ed90f2" volumeName="kubernetes.io/configmap/4480cca6-b2e9-4840-ac8c-a88a40ed90f2-metrics-client-ca" seLinuxMountContext="" Nov 03 01:52:02.822941 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.822408 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="52b546b3-7bf6-4239-91cb-c8aaca1c9d18" volumeName="kubernetes.io/secret/52b546b3-7bf6-4239-91cb-c8aaca1c9d18-metrics-tls" seLinuxMountContext="" Nov 03 01:52:02.822941 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.822423 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="592adf6b-1edd-4f5e-88a2-6ed92895f22a" volumeName="kubernetes.io/projected/592adf6b-1edd-4f5e-88a2-6ed92895f22a-kube-api-access-bw55q" seLinuxMountContext="" Nov 03 01:52:02.822941 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.822453 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4727262-eb15-4453-adf2-0fc6c2358508" volumeName="kubernetes.io/empty-dir/a4727262-eb15-4453-adf2-0fc6c2358508-tmp" seLinuxMountContext="" Nov 03 01:52:02.822941 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.822466 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56117bce-d686-4c24-98c8-c0ff122778ac" volumeName="kubernetes.io/configmap/56117bce-d686-4c24-98c8-c0ff122778ac-env-overrides" seLinuxMountContext="" Nov 03 01:52:02.822941 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.822484 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d7d07f7-27c2-489a-8e44-fe366584a267" volumeName="kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-secret-alertmanager-main-tls" seLinuxMountContext="" Nov 03 01:52:02.822941 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.822500 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d7d07f7-27c2-489a-8e44-fe366584a267" volumeName="kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-config-volume" seLinuxMountContext="" Nov 03 01:52:02.822941 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.822512 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13fa5796-b3e6-4d1c-9759-88b7094d4b37" volumeName="kubernetes.io/secret/13fa5796-b3e6-4d1c-9759-88b7094d4b37-console-serving-cert" seLinuxMountContext="" Nov 03 01:52:02.822941 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.822529 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56117bce-d686-4c24-98c8-c0ff122778ac" volumeName="kubernetes.io/configmap/56117bce-d686-4c24-98c8-c0ff122778ac-ovnkube-script-lib" seLinuxMountContext="" Nov 03 01:52:02.822941 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.822542 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="923b90de-5e45-4add-b092-91af16b7dbad" volumeName="kubernetes.io/configmap/923b90de-5e45-4add-b092-91af16b7dbad-metrics-client-ca" seLinuxMountContext="" Nov 03 01:52:02.822941 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.822614 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b" volumeName="kubernetes.io/projected/dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b-kube-api-access-l2l49" seLinuxMountContext="" Nov 03 01:52:02.822941 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.822639 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e32389d5-705f-4202-98ea-6fc6a5ad5d54" volumeName="kubernetes.io/empty-dir/e32389d5-705f-4202-98ea-6fc6a5ad5d54-audit-log" seLinuxMountContext="" Nov 03 01:52:02.822941 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.822685 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3bc52103-b80b-49a8-9bed-15b196fdef12" volumeName="kubernetes.io/projected/3bc52103-b80b-49a8-9bed-15b196fdef12-kube-api-access-xk6qs" seLinuxMountContext="" Nov 03 01:52:02.822941 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.822704 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d514c70-f1a7-462a-bd3e-6f5bdc016c39" volumeName="kubernetes.io/secret/5d514c70-f1a7-462a-bd3e-6f5bdc016c39-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 01:52:02.822941 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.822717 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d514c70-f1a7-462a-bd3e-6f5bdc016c39" volumeName="kubernetes.io/secret/5d514c70-f1a7-462a-bd3e-6f5bdc016c39-prometheus-operator-tls" seLinuxMountContext="" Nov 03 01:52:02.822941 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.822745 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="923b90de-5e45-4add-b092-91af16b7dbad" volumeName="kubernetes.io/secret/923b90de-5e45-4add-b092-91af16b7dbad-secret-telemeter-client-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 01:52:02.822941 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.822758 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b" volumeName="kubernetes.io/secret/dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b-metrics-certs" seLinuxMountContext="" Nov 03 01:52:02.822941 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.822782 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e32389d5-705f-4202-98ea-6fc6a5ad5d54" volumeName="kubernetes.io/configmap/e32389d5-705f-4202-98ea-6fc6a5ad5d54-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Nov 03 01:52:02.822941 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.822813 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d7d07f7-27c2-489a-8e44-fe366584a267" volumeName="kubernetes.io/projected/3d7d07f7-27c2-489a-8e44-fe366584a267-kube-api-access-nl9gt" seLinuxMountContext="" Nov 03 01:52:02.822941 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.822837 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6" volumeName="kubernetes.io/configmap/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-oauth-serving-cert" seLinuxMountContext="" Nov 03 01:52:02.822941 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.822853 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4480cca6-b2e9-4840-ac8c-a88a40ed90f2" volumeName="kubernetes.io/configmap/4480cca6-b2e9-4840-ac8c-a88a40ed90f2-node-exporter-accelerators-collector-config" seLinuxMountContext="" Nov 03 01:52:02.822941 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.822878 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56117bce-d686-4c24-98c8-c0ff122778ac" volumeName="kubernetes.io/configmap/56117bce-d686-4c24-98c8-c0ff122778ac-ovnkube-config" seLinuxMountContext="" Nov 03 01:52:02.822941 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.822890 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="592adf6b-1edd-4f5e-88a2-6ed92895f22a" volumeName="kubernetes.io/configmap/592adf6b-1edd-4f5e-88a2-6ed92895f22a-serviceca" seLinuxMountContext="" Nov 03 01:52:02.822941 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.822907 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="923b90de-5e45-4add-b092-91af16b7dbad" volumeName="kubernetes.io/configmap/923b90de-5e45-4add-b092-91af16b7dbad-telemeter-trusted-ca-bundle" seLinuxMountContext="" Nov 03 01:52:02.822941 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.822931 5605 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e32389d5-705f-4202-98ea-6fc6a5ad5d54" volumeName="kubernetes.io/secret/e32389d5-705f-4202-98ea-6fc6a5ad5d54-secret-metrics-server-client-certs" seLinuxMountContext="" Nov 03 01:52:02.822941 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.822943 5605 reconstruct.go:97] "Volume reconstruction finished" Nov 03 01:52:02.822941 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.822950 5605 reconciler.go:29] "Reconciler: start to sync state" Nov 03 01:52:02.840935 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.840881 5605 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 01:52:02.842503 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.842478 5605 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 01:52:02.842503 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.842505 5605 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 01:52:02.842661 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.842527 5605 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 01:52:02.842661 ip-10-0-140-140 kubenswrapper[5605]: E1103 01:52:02.842566 5605 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 01:52:02.854075 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.854032 5605 generic.go:356] "Generic (PLEG): container finished" podID="4480cca6-b2e9-4840-ac8c-a88a40ed90f2" containerID="6de7e466197c3677347a4b99ad4c74967961b793939e0e414ccec863c44b0ce0" exitCode=0 Nov 03 01:52:02.860031 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.860005 5605 generic.go:356] "Generic (PLEG): container finished" podID="9152cf36-4997-470e-8902-802a96ebc2c7" containerID="2cde1738af27fdb2d6b0d1a8a95c5a7ece144ca35e2b7549038be52669e42954" exitCode=0 Nov 03 01:52:02.860031 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.860029 5605 generic.go:356] "Generic (PLEG): container finished" podID="9152cf36-4997-470e-8902-802a96ebc2c7" containerID="ace6e3a8d61e46c606efc566bae17f1879a9fb1930a5ccb382963042173a04c5" exitCode=0 Nov 03 01:52:02.860187 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.860082 5605 generic.go:356] "Generic (PLEG): container finished" podID="9152cf36-4997-470e-8902-802a96ebc2c7" containerID="ee19309bbbeb838da62c2138b0fd04e48db6c4e4c2e7463b36f475442b9abf6b" exitCode=0 Nov 03 01:52:02.860187 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.860094 5605 generic.go:356] "Generic (PLEG): container finished" podID="9152cf36-4997-470e-8902-802a96ebc2c7" containerID="ef3a3ad4dcd0b7b6726fd15c67345b4bd8d407e35a646a94eff195a8ee59092d" exitCode=0 Nov 03 01:52:02.860187 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.860105 5605 generic.go:356] "Generic (PLEG): container finished" podID="9152cf36-4997-470e-8902-802a96ebc2c7" containerID="7f39f434749ee4bec89a3ec1c468c53f1dbe225a0811178a618924663c8df8f1" exitCode=0 Nov 03 01:52:02.860187 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.860114 5605 generic.go:356] "Generic (PLEG): container finished" podID="9152cf36-4997-470e-8902-802a96ebc2c7" containerID="a1ad295aeb842338664a10f33a39f2ad700aee049a86a69790085fc0e2e1316a" exitCode=0 Nov 03 01:52:02.871511 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.871481 5605 generic.go:356] "Generic (PLEG): container finished" podID="3d7d07f7-27c2-489a-8e44-fe366584a267" containerID="0d9d5fbaa013bb1c1bc86e40f772c92f6173c111758e1557c9ffb7578af08cf3" exitCode=0 Nov 03 01:52:02.908034 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.907998 5605 generic.go:356] "Generic (PLEG): container finished" podID="3b5ab86e21798d22090c574595774d10" containerID="0d8c0999791f3327b46bc8895969e4f93e864da713eedc74f06cd76721c12a72" exitCode=0 Nov 03 01:52:02.914609 ip-10-0-140-140 kubenswrapper[5605]: E1103 01:52:02.914586 5605 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-140.us-west-2.compute.internal\" not found" Nov 03 01:52:02.926252 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.926226 5605 manager.go:324] Recovery completed Nov 03 01:52:02.942719 ip-10-0-140-140 kubenswrapper[5605]: E1103 01:52:02.942646 5605 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 03 01:52:02.947259 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.947242 5605 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 01:52:02.949142 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.949126 5605 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-140.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 01:52:02.949196 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.949155 5605 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-140.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 01:52:02.949196 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.949165 5605 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-140.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 01:52:02.952402 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.952385 5605 cpu_manager.go:226] "Starting" policy="none" Nov 03 01:52:02.952402 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.952398 5605 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 01:52:02.952512 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.952434 5605 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 01:52:02.952512 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.952450 5605 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 01:52:02.952593 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.952576 5605 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 01:52:02.952668 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.952646 5605 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 01:52:02.952697 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.952669 5605 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 01:52:02.952697 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.952674 5605 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 01:52:02.952697 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.952680 5605 policy_none.go:50] "Start" Nov 03 01:52:02.952697 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.952689 5605 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 01:52:02.952797 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.952700 5605 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 01:52:02.952864 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.952854 5605 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 01:52:02.952892 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.952865 5605 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 01:52:02.952892 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.952870 5605 policy_none.go:44] "Start" Nov 03 01:52:02.953316 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.953296 5605 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-140.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 01:52:02.953403 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.953325 5605 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-140.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 01:52:02.953403 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.953336 5605 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-140.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 01:52:02.957262 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.957239 5605 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 01:52:02.959715 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.959695 5605 manager.go:341] "Starting Device Plugin manager" Nov 03 01:52:02.959794 ip-10-0-140-140 kubenswrapper[5605]: E1103 01:52:02.959719 5605 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 01:52:02.959794 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.959731 5605 server.go:87] "Starting device plugin registration server" Nov 03 01:52:02.960081 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.960069 5605 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 01:52:02.960121 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.960085 5605 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 01:52:02.960213 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.960194 5605 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 01:52:02.960419 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.960381 5605 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 01:52:02.960500 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.960427 5605 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 01:52:02.960500 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.960434 5605 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 01:52:02.961526 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.961413 5605 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T01:52:02.960416296Z","UUID":"003f2438-bd78-45bc-9e76-0a94dd5875f2","Handler":null,"Name":"","Endpoint":""} Nov 03 01:52:02.961943 ip-10-0-140-140 kubenswrapper[5605]: E1103 01:52:02.961919 5605 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 01:52:02.966179 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.966157 5605 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 01:52:02.966278 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.966190 5605 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 01:52:02.967853 ip-10-0-140-140 kubenswrapper[5605]: E1103 01:52:02.967829 5605 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-140.us-west-2.compute.internal\" not found" Nov 03 01:52:02.998527 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.998493 5605 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 01:52:02.998527 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.998498 5605 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 01:52:02.998716 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.998566 5605 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 01:52:02.999156 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:02.999131 5605 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 01:52:03.060651 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.060425 5605 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 01:52:03.061623 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.061603 5605 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-140.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 01:52:03.061765 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.061635 5605 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-140.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 01:52:03.061765 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.061646 5605 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-140.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 01:52:03.061765 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.061689 5605 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-140.us-west-2.compute.internal" Nov 03 01:52:03.091863 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.091835 5605 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 01:52:03.143735 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.143684 5605 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-140.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-140.us-west-2.compute.internal"] Nov 03 01:52:03.144299 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.144255 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-140.us-west-2.compute.internal" event={"ID":"d1cefe6117be002b0e139e82f22d4917","Type":"ContainerStarted","Data":"4324339d1bde2f5a0ca57f44f779b47a0fe0177f77f6d2827d9bb41f35384908"} Nov 03 01:52:03.144469 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.144303 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-140.us-west-2.compute.internal" event={"ID":"d1cefe6117be002b0e139e82f22d4917","Type":"ContainerStarted","Data":"535f8f7d5b0302230f474cf92cfcadc6365f660f9f7339ab151a81e4890cea8e"} Nov 03 01:52:03.144469 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.144381 5605 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-140.us-west-2.compute.internal" Nov 03 01:52:03.144469 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.144447 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-140.us-west-2.compute.internal" event={"ID":"3b5ab86e21798d22090c574595774d10","Type":"ContainerStarted","Data":"25cf6dcfecb0fc1b8717378e9fd9fb8a071982c9d5c4602699f25be38c3df528"} Nov 03 01:52:03.144582 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.144470 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-140.us-west-2.compute.internal" event={"ID":"3b5ab86e21798d22090c574595774d10","Type":"ContainerDied","Data":"0d8c0999791f3327b46bc8895969e4f93e864da713eedc74f06cd76721c12a72"} Nov 03 01:52:03.144582 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.144490 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-140.us-west-2.compute.internal" event={"ID":"3b5ab86e21798d22090c574595774d10","Type":"ContainerStarted","Data":"988bc0719feedc4d376b3edfbe25039b9faf76ee6d89c0e2e9ba1ea8ad53553a"} Nov 03 01:52:03.144582 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.144498 5605 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-140.us-west-2.compute.internal" Nov 03 01:52:03.166579 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.164841 5605 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-140-140.us-west-2.compute.internal" Nov 03 01:52:03.166579 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.164915 5605 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-140.us-west-2.compute.internal" Nov 03 01:52:03.201413 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.201335 5605 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 01:52:03.201413 ip-10-0-140-140 kubenswrapper[5605]: E1103 01:52:03.201396 5605 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-140.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-140.us-west-2.compute.internal" Nov 03 01:52:03.202009 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.201993 5605 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 01:52:03.202044 ip-10-0-140-140 kubenswrapper[5605]: E1103 01:52:03.202034 5605 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-140.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-140.us-west-2.compute.internal" Nov 03 01:52:03.300765 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.300715 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3b5ab86e21798d22090c574595774d10-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-140.us-west-2.compute.internal\" (UID: \"3b5ab86e21798d22090c574595774d10\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-140.us-west-2.compute.internal" Nov 03 01:52:03.300907 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.300773 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b5ab86e21798d22090c574595774d10-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-140.us-west-2.compute.internal\" (UID: \"3b5ab86e21798d22090c574595774d10\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-140.us-west-2.compute.internal" Nov 03 01:52:03.300907 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.300832 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d1cefe6117be002b0e139e82f22d4917-config\") pod \"kube-apiserver-proxy-ip-10-0-140-140.us-west-2.compute.internal\" (UID: \"d1cefe6117be002b0e139e82f22d4917\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-140.us-west-2.compute.internal" Nov 03 01:52:03.401411 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.401375 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3b5ab86e21798d22090c574595774d10-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-140.us-west-2.compute.internal\" (UID: \"3b5ab86e21798d22090c574595774d10\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-140.us-west-2.compute.internal" Nov 03 01:52:03.401551 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.401422 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b5ab86e21798d22090c574595774d10-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-140.us-west-2.compute.internal\" (UID: \"3b5ab86e21798d22090c574595774d10\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-140.us-west-2.compute.internal" Nov 03 01:52:03.401551 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.401443 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d1cefe6117be002b0e139e82f22d4917-config\") pod \"kube-apiserver-proxy-ip-10-0-140-140.us-west-2.compute.internal\" (UID: \"d1cefe6117be002b0e139e82f22d4917\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-140.us-west-2.compute.internal" Nov 03 01:52:03.401551 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.401450 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b5ab86e21798d22090c574595774d10-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-140.us-west-2.compute.internal\" (UID: \"3b5ab86e21798d22090c574595774d10\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-140.us-west-2.compute.internal" Nov 03 01:52:03.401551 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.401482 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d1cefe6117be002b0e139e82f22d4917-config\") pod \"kube-apiserver-proxy-ip-10-0-140-140.us-west-2.compute.internal\" (UID: \"d1cefe6117be002b0e139e82f22d4917\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-140.us-west-2.compute.internal" Nov 03 01:52:03.401551 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.401423 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3b5ab86e21798d22090c574595774d10-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-140.us-west-2.compute.internal\" (UID: \"3b5ab86e21798d22090c574595774d10\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-140.us-west-2.compute.internal" Nov 03 01:52:03.800919 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.800885 5605 apiserver.go:52] "Watching apiserver" Nov 03 01:52:03.914897 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.914869 5605 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-140.us-west-2.compute.internal" Nov 03 01:52:03.915065 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.914940 5605 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-140.us-west-2.compute.internal" Nov 03 01:52:03.931454 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.929309 5605 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 01:52:03.931454 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.930151 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2q7kh","openshift-console/console-676fb595d5-lmt4w","openshift-insights/insights-runtime-extractor-dmrb8","openshift-monitoring/telemeter-client-699d68669f-9jg2q","openshift-console/console-69c467f7c8-cpfwh","openshift-monitoring/alertmanager-main-0","openshift-dns/node-resolver-7zdvv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-140.us-west-2.compute.internal","openshift-multus/network-metrics-daemon-vzfhc","openshift-network-diagnostics/network-check-target-f5qvs","kube-system/global-pull-secret-syncer-kkngd","openshift-network-operator/iptables-alerter-vmjc5","openshift-image-registry/node-ca-gjs9f","openshift-multus/multus-trr75","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdpjb","openshift-console/console-59fbbdf587-6nm8q","openshift-monitoring/prometheus-operator-9cd89b74c-jmj4g","openshift-ovn-kubernetes/ovnkube-node-x452r","openshift-ingress-canary/ingress-canary-2d49s","kube-system/konnectivity-agent-f524q","kube-system/kube-apiserver-proxy-ip-10-0-140-140.us-west-2.compute.internal","openshift-dns/dns-default-t2dwr","openshift-monitoring/metrics-server-b984f6b6b-g6f2h","openshift-multus/multus-additional-cni-plugins-gwlr2","openshift-cluster-node-tuning-operator/tuned-qm8zc"] Nov 03 01:52:03.975585 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.975551 5605 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 01:52:03.975736 ip-10-0-140-140 kubenswrapper[5605]: E1103 01:52:03.975612 5605 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-140.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-140.us-west-2.compute.internal" Nov 03 01:52:03.977716 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.977561 5605 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 01:52:03.977716 ip-10-0-140-140 kubenswrapper[5605]: E1103 01:52:03.977609 5605 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-140.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-140.us-west-2.compute.internal" Nov 03 01:52:03.979972 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.979950 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 01:52:03.980057 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.980004 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 01:52:03.980262 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.980241 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 01:52:03.980317 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.980262 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 01:52:03.980317 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.980292 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-2mmqt\"" Nov 03 01:52:03.980317 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.980289 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 01:52:03.980317 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.980298 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hgghc\"" Nov 03 01:52:03.980608 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.980592 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 01:52:03.980809 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.980795 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5qdp4\"" Nov 03 01:52:03.980899 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.980874 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jwmm7\"" Nov 03 01:52:03.980975 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.980912 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 01:52:03.981097 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.980982 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 01:52:03.981097 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.981028 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 01:52:03.981097 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.981071 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 01:52:03.981097 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.981086 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 01:52:03.981259 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.981173 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 01:52:03.981519 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.981505 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 01:52:03.981608 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.981594 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-zq5dk\"" Nov 03 01:52:03.982144 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.982124 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 01:52:03.982445 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.982427 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bqcwb\"" Nov 03 01:52:03.982498 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.982452 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 01:52:03.983074 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.983061 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-btxgv\"" Nov 03 01:52:03.983301 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.983285 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 01:52:03.984157 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.984144 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 01:52:03.984461 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.984440 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 01:52:03.984579 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.984564 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-wpc48\"" Nov 03 01:52:03.984630 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.984580 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-zxsvp\"" Nov 03 01:52:03.984774 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.984706 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 01:52:03.984774 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.984760 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 01:52:03.984906 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.984893 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 01:52:03.984943 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.984928 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 01:52:03.985136 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.985118 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 01:52:03.987427 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.987316 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 01:52:03.987427 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.987318 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-v5ks7\"" Nov 03 01:52:03.987427 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.987407 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 01:52:03.987558 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.987481 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 01:52:03.987558 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.987543 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 01:52:03.987681 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.987663 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pc8bh\"" Nov 03 01:52:03.987718 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:03.987675 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-f7c55\"" Nov 03 01:52:04.012783 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.012744 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 01:52:04.012783 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.012744 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 03 01:52:04.012783 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.012748 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 01:52:04.013008 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.012823 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 01:52:04.013008 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.013002 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 03 01:52:04.013099 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.013012 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 03 01:52:04.013099 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.013021 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 03 01:52:04.013099 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.013030 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 03 01:52:04.013099 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.013048 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 01:52:04.013099 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.013056 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 01:52:04.013099 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.013079 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 01:52:04.013099 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.013086 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bsx6s\"" Nov 03 01:52:04.013099 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.013089 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 01:52:04.013420 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.013105 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-rvjqm\"" Nov 03 01:52:04.013420 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.013158 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 01:52:04.013420 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.013247 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 01:52:04.013420 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.013273 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 03 01:52:04.013420 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.013378 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 03 01:52:04.013420 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.013402 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 03 01:52:04.013712 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.013450 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 01:52:04.013712 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.013462 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7cgvv\"" Nov 03 01:52:04.013712 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.013664 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 03 01:52:04.013712 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.013686 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 01:52:04.013855 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.013726 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-t7b59\"" Nov 03 01:52:04.013855 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.013728 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 03 01:52:04.013855 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.013749 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 03 01:52:04.013855 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.013768 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-8rp9w\"" Nov 03 01:52:04.013855 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.013780 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 03 01:52:04.013855 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.013785 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 03 01:52:04.014033 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.013901 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-s796x\"" Nov 03 01:52:04.014033 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.013910 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 01:52:04.014033 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.013937 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 01:52:04.014033 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.013952 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 01:52:04.014033 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.013904 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 03 01:52:04.014033 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.013983 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 03 01:52:04.014033 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.013910 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 01:52:04.014033 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.014033 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 03 01:52:04.014290 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.014084 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-x6kj8\"" Nov 03 01:52:04.014290 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.014098 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jwnvd\"" Nov 03 01:52:04.014290 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.014107 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 03 01:52:04.014290 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.014154 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 03 01:52:04.019428 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.019400 5605 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 01:52:04.026127 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.026110 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 01:52:04.026208 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.026138 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 01:52:04.026208 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.026160 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 01:52:04.031782 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.031763 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 03 01:52:04.031867 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.031803 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 03 01:52:04.032461 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.032445 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 03 01:52:04.080386 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.080301 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 03 01:52:04.080386 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.080320 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 03 01:52:04.080386 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.080311 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 01:52:04.080386 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.080302 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 03 01:52:04.080386 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.080384 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 03 01:52:04.080690 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.080499 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 03 01:52:04.080690 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.080506 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 03 01:52:04.080690 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.080516 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-25h2j\"" Nov 03 01:52:04.080690 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.080539 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 03 01:52:04.080690 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.080603 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-5gb0tdoj89a63\"" Nov 03 01:52:04.081665 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.081651 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 03 01:52:04.106429 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.106401 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/df76407a-ce41-4a4a-b51e-8420b817f97c-etc-sysconfig\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:52:04.106566 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.106434 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9152cf36-4997-470e-8902-802a96ebc2c7-cnibin\") pod \"multus-additional-cni-plugins-gwlr2\" (UID: \"9152cf36-4997-470e-8902-802a96ebc2c7\") " pod="openshift-multus/multus-additional-cni-plugins-gwlr2" Nov 03 01:52:04.106566 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.106457 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-run-ovn\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:52:04.106566 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.106486 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-console-serving-cert\") pod \"console-59fbbdf587-6nm8q\" (UID: \"3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6\") " pod="openshift-console/console-59fbbdf587-6nm8q" Nov 03 01:52:04.106566 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.106524 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d514c70-f1a7-462a-bd3e-6f5bdc016c39-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-jmj4g\" (UID: \"5d514c70-f1a7-462a-bd3e-6f5bdc016c39\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jmj4g" Nov 03 01:52:04.106566 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.106551 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:04.106805 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.106698 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-run-systemd\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:52:04.106805 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.106728 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0801f13c-d14a-4b9e-b45a-a51af48d17c3-socket-dir\") pod \"aws-ebs-csi-driver-node-xdpjb\" (UID: \"0801f13c-d14a-4b9e-b45a-a51af48d17c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdpjb" Nov 03 01:52:04.106805 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.106728 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-console-serving-cert\") pod \"console-59fbbdf587-6nm8q\" (UID: \"3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6\") " pod="openshift-console/console-59fbbdf587-6nm8q" Nov 03 01:52:04.106805 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.106745 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:04.106805 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.106748 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d514c70-f1a7-462a-bd3e-6f5bdc016c39-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-jmj4g\" (UID: \"5d514c70-f1a7-462a-bd3e-6f5bdc016c39\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jmj4g" Nov 03 01:52:04.106805 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.106752 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/56117bce-d686-4c24-98c8-c0ff122778ac-ovn-node-metrics-cert\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:52:04.106805 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.106791 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-host-run-k8s-cni-cncf-io\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:52:04.107103 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.106922 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/56117bce-d686-4c24-98c8-c0ff122778ac-ovn-node-metrics-cert\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:52:04.107103 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.106919 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-web-config\") pod \"alertmanager-main-0\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:04.107103 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.106950 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-cnibin\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:52:04.107103 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.106979 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-host-var-lib-cni-bin\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:52:04.107103 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.106998 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/13fa5796-b3e6-4d1c-9759-88b7094d4b37-tmp\") pod \"console-676fb595d5-lmt4w\" (UID: \"13fa5796-b3e6-4d1c-9759-88b7094d4b37\") " pod="openshift-console/console-676fb595d5-lmt4w" Nov 03 01:52:04.107103 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.107014 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qmrm\" (UniqueName: \"kubernetes.io/projected/85148983-aa3f-45c2-a062-da2748dd9011-kube-api-access-2qmrm\") pod \"iptables-alerter-vmjc5\" (UID: \"85148983-aa3f-45c2-a062-da2748dd9011\") " pod="openshift-network-operator/iptables-alerter-vmjc5" Nov 03 01:52:04.107103 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.107029 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-service-ca\") pod \"console-59fbbdf587-6nm8q\" (UID: \"3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6\") " pod="openshift-console/console-59fbbdf587-6nm8q" Nov 03 01:52:04.107103 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.107070 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/923b90de-5e45-4add-b092-91af16b7dbad-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-699d68669f-9jg2q\" (UID: \"923b90de-5e45-4add-b092-91af16b7dbad\") " pod="openshift-monitoring/telemeter-client-699d68669f-9jg2q" Nov 03 01:52:04.107103 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.107090 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/923b90de-5e45-4add-b092-91af16b7dbad-metrics-client-ca\") pod \"telemeter-client-699d68669f-9jg2q\" (UID: \"923b90de-5e45-4add-b092-91af16b7dbad\") " pod="openshift-monitoring/telemeter-client-699d68669f-9jg2q" Nov 03 01:52:04.107103 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.107105 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a4727262-eb15-4453-adf2-0fc6c2358508-console-oauth-config\") pod \"console-69c467f7c8-cpfwh\" (UID: \"a4727262-eb15-4453-adf2-0fc6c2358508\") " pod="openshift-console/console-69c467f7c8-cpfwh" Nov 03 01:52:04.107599 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.107120 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-host-cni-netd\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:52:04.107599 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.107118 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-web-config\") pod \"alertmanager-main-0\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:04.107599 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.107145 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffwtk\" (UniqueName: \"kubernetes.io/projected/df76407a-ce41-4a4a-b51e-8420b817f97c-kube-api-access-ffwtk\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:52:04.107599 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.107238 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-service-ca\") pod \"console-59fbbdf587-6nm8q\" (UID: \"3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6\") " pod="openshift-console/console-59fbbdf587-6nm8q" Nov 03 01:52:04.107599 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.107268 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e32389d5-705f-4202-98ea-6fc6a5ad5d54-secret-metrics-server-tls\") pod \"metrics-server-b984f6b6b-g6f2h\" (UID: \"e32389d5-705f-4202-98ea-6fc6a5ad5d54\") " pod="openshift-monitoring/metrics-server-b984f6b6b-g6f2h" Nov 03 01:52:04.107599 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.107284 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/df76407a-ce41-4a4a-b51e-8420b817f97c-host\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:52:04.107599 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.107300 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9152cf36-4997-470e-8902-802a96ebc2c7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gwlr2\" (UID: \"9152cf36-4997-470e-8902-802a96ebc2c7\") " pod="openshift-multus/multus-additional-cni-plugins-gwlr2" Nov 03 01:52:04.107599 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.107319 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-host-run-ovn-kubernetes\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:52:04.107599 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.107337 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-oauth-serving-cert\") pod \"console-59fbbdf587-6nm8q\" (UID: \"3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6\") " pod="openshift-console/console-59fbbdf587-6nm8q" Nov 03 01:52:04.107599 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.107351 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9152cf36-4997-470e-8902-802a96ebc2c7-cni-binary-copy\") pod \"multus-additional-cni-plugins-gwlr2\" (UID: \"9152cf36-4997-470e-8902-802a96ebc2c7\") " pod="openshift-multus/multus-additional-cni-plugins-gwlr2" Nov 03 01:52:04.107599 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.107392 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3d7d07f7-27c2-489a-8e44-fe366584a267-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:04.107599 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.107406 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e32389d5-705f-4202-98ea-6fc6a5ad5d54-secret-metrics-server-client-certs\") pod \"metrics-server-b984f6b6b-g6f2h\" (UID: \"e32389d5-705f-4202-98ea-6fc6a5ad5d54\") " pod="openshift-monitoring/metrics-server-b984f6b6b-g6f2h" Nov 03 01:52:04.107599 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.107420 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/df76407a-ce41-4a4a-b51e-8420b817f97c-etc-sysctl-d\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:52:04.107599 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.107505 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/df76407a-ce41-4a4a-b51e-8420b817f97c-var-lib-kubelet\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:52:04.107599 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.107512 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e32389d5-705f-4202-98ea-6fc6a5ad5d54-secret-metrics-server-tls\") pod \"metrics-server-b984f6b6b-g6f2h\" (UID: \"e32389d5-705f-4202-98ea-6fc6a5ad5d54\") " pod="openshift-monitoring/metrics-server-b984f6b6b-g6f2h" Nov 03 01:52:04.107599 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.107526 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-multus-daemon-config\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:52:04.107599 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.107541 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/52b546b3-7bf6-4239-91cb-c8aaca1c9d18-config-volume\") pod \"dns-default-t2dwr\" (UID: \"52b546b3-7bf6-4239-91cb-c8aaca1c9d18\") " pod="openshift-dns/dns-default-t2dwr" Nov 03 01:52:04.107599 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.107559 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9152cf36-4997-470e-8902-802a96ebc2c7-system-cni-dir\") pod \"multus-additional-cni-plugins-gwlr2\" (UID: \"9152cf36-4997-470e-8902-802a96ebc2c7\") " pod="openshift-multus/multus-additional-cni-plugins-gwlr2" Nov 03 01:52:04.107599 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.107584 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4480cca6-b2e9-4840-ac8c-a88a40ed90f2-node-exporter-tls\") pod \"node-exporter-2q7kh\" (UID: \"4480cca6-b2e9-4840-ac8c-a88a40ed90f2\") " pod="openshift-monitoring/node-exporter-2q7kh" Nov 03 01:52:04.107599 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.107606 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:52:04.108490 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.107631 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4480cca6-b2e9-4840-ac8c-a88a40ed90f2-node-exporter-textfile\") pod \"node-exporter-2q7kh\" (UID: \"4480cca6-b2e9-4840-ac8c-a88a40ed90f2\") " pod="openshift-monitoring/node-exporter-2q7kh" Nov 03 01:52:04.108490 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.107645 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4480cca6-b2e9-4840-ac8c-a88a40ed90f2-node-exporter-wtmp\") pod \"node-exporter-2q7kh\" (UID: \"4480cca6-b2e9-4840-ac8c-a88a40ed90f2\") " pod="openshift-monitoring/node-exporter-2q7kh" Nov 03 01:52:04.108490 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.107661 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7d2438ed-bfc2-4639-b87d-84689a82123e-konnectivity-ca\") pod \"konnectivity-agent-f524q\" (UID: \"7d2438ed-bfc2-4639-b87d-84689a82123e\") " pod="kube-system/konnectivity-agent-f524q" Nov 03 01:52:04.108490 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.107682 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/28f00eb6-78c2-4d93-bde5-c1ea12601dbd-kubelet-config\") pod \"global-pull-secret-syncer-kkngd\" (UID: \"28f00eb6-78c2-4d93-bde5-c1ea12601dbd\") " pod="kube-system/global-pull-secret-syncer-kkngd" Nov 03 01:52:04.108490 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.107690 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a4727262-eb15-4453-adf2-0fc6c2358508-console-oauth-config\") pod \"console-69c467f7c8-cpfwh\" (UID: \"a4727262-eb15-4453-adf2-0fc6c2358508\") " pod="openshift-console/console-69c467f7c8-cpfwh" Nov 03 01:52:04.108490 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.107705 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-var-lib-openvswitch\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:52:04.108490 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.107719 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-run-openvswitch\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:52:04.108490 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.107857 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9152cf36-4997-470e-8902-802a96ebc2c7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gwlr2\" (UID: \"9152cf36-4997-470e-8902-802a96ebc2c7\") " pod="openshift-multus/multus-additional-cni-plugins-gwlr2" Nov 03 01:52:04.108490 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.107981 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-oauth-serving-cert\") pod \"console-59fbbdf587-6nm8q\" (UID: \"3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6\") " pod="openshift-console/console-59fbbdf587-6nm8q" Nov 03 01:52:04.108490 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.108089 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9152cf36-4997-470e-8902-802a96ebc2c7-cni-binary-copy\") pod \"multus-additional-cni-plugins-gwlr2\" (UID: \"9152cf36-4997-470e-8902-802a96ebc2c7\") " pod="openshift-multus/multus-additional-cni-plugins-gwlr2" Nov 03 01:52:04.108490 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.108125 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3d7d07f7-27c2-489a-8e44-fe366584a267-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:04.108490 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.108216 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/13fa5796-b3e6-4d1c-9759-88b7094d4b37-tmp\") pod \"console-676fb595d5-lmt4w\" (UID: \"13fa5796-b3e6-4d1c-9759-88b7094d4b37\") " pod="openshift-console/console-676fb595d5-lmt4w" Nov 03 01:52:04.108490 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.108270 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e32389d5-705f-4202-98ea-6fc6a5ad5d54-secret-metrics-server-client-certs\") pod \"metrics-server-b984f6b6b-g6f2h\" (UID: \"e32389d5-705f-4202-98ea-6fc6a5ad5d54\") " pod="openshift-monitoring/metrics-server-b984f6b6b-g6f2h" Nov 03 01:52:04.108490 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.108407 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-multus-daemon-config\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:52:04.108490 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.108428 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/923b90de-5e45-4add-b092-91af16b7dbad-metrics-client-ca\") pod \"telemeter-client-699d68669f-9jg2q\" (UID: \"923b90de-5e45-4add-b092-91af16b7dbad\") " pod="openshift-monitoring/telemeter-client-699d68669f-9jg2q" Nov 03 01:52:04.108490 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.108467 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e32389d5-705f-4202-98ea-6fc6a5ad5d54-metrics-server-audit-profiles\") pod \"metrics-server-b984f6b6b-g6f2h\" (UID: \"e32389d5-705f-4202-98ea-6fc6a5ad5d54\") " pod="openshift-monitoring/metrics-server-b984f6b6b-g6f2h" Nov 03 01:52:04.108490 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.108497 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmhdf\" (UniqueName: \"kubernetes.io/projected/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-kube-api-access-bmhdf\") pod \"console-59fbbdf587-6nm8q\" (UID: \"3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6\") " pod="openshift-console/console-59fbbdf587-6nm8q" Nov 03 01:52:04.109274 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.108533 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/13fa5796-b3e6-4d1c-9759-88b7094d4b37-console-serving-cert\") pod \"console-676fb595d5-lmt4w\" (UID: \"13fa5796-b3e6-4d1c-9759-88b7094d4b37\") " pod="openshift-console/console-676fb595d5-lmt4w" Nov 03 01:52:04.109274 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.108559 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/13fa5796-b3e6-4d1c-9759-88b7094d4b37-console-config\") pod \"console-676fb595d5-lmt4w\" (UID: \"13fa5796-b3e6-4d1c-9759-88b7094d4b37\") " pod="openshift-console/console-676fb595d5-lmt4w" Nov 03 01:52:04.109274 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.108583 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4480cca6-b2e9-4840-ac8c-a88a40ed90f2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2q7kh\" (UID: \"4480cca6-b2e9-4840-ac8c-a88a40ed90f2\") " pod="openshift-monitoring/node-exporter-2q7kh" Nov 03 01:52:04.109274 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.108588 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/923b90de-5e45-4add-b092-91af16b7dbad-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-699d68669f-9jg2q\" (UID: \"923b90de-5e45-4add-b092-91af16b7dbad\") " pod="openshift-monitoring/telemeter-client-699d68669f-9jg2q" Nov 03 01:52:04.109274 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.108611 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/592adf6b-1edd-4f5e-88a2-6ed92895f22a-serviceca\") pod \"node-ca-gjs9f\" (UID: \"592adf6b-1edd-4f5e-88a2-6ed92895f22a\") " pod="openshift-image-registry/node-ca-gjs9f" Nov 03 01:52:04.109274 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.108634 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4480cca6-b2e9-4840-ac8c-a88a40ed90f2-node-exporter-textfile\") pod \"node-exporter-2q7kh\" (UID: \"4480cca6-b2e9-4840-ac8c-a88a40ed90f2\") " pod="openshift-monitoring/node-exporter-2q7kh" Nov 03 01:52:04.109274 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.108682 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/df76407a-ce41-4a4a-b51e-8420b817f97c-run\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:52:04.109274 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.108710 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/df76407a-ce41-4a4a-b51e-8420b817f97c-lib-modules\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:52:04.109274 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.108735 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7c3f175d-932c-42be-9660-e4fd3dceaf49-hosts-file\") pod \"node-resolver-7zdvv\" (UID: \"7c3f175d-932c-42be-9660-e4fd3dceaf49\") " pod="openshift-dns/node-resolver-7zdvv" Nov 03 01:52:04.109274 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.108782 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/52b546b3-7bf6-4239-91cb-c8aaca1c9d18-config-volume\") pod \"dns-default-t2dwr\" (UID: \"52b546b3-7bf6-4239-91cb-c8aaca1c9d18\") " pod="openshift-dns/dns-default-t2dwr" Nov 03 01:52:04.109274 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.108834 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4480cca6-b2e9-4840-ac8c-a88a40ed90f2-node-exporter-tls\") pod \"node-exporter-2q7kh\" (UID: \"4480cca6-b2e9-4840-ac8c-a88a40ed90f2\") " pod="openshift-monitoring/node-exporter-2q7kh" Nov 03 01:52:04.109274 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.108900 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2l49\" (UniqueName: \"kubernetes.io/projected/dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b-kube-api-access-l2l49\") pod \"network-metrics-daemon-vzfhc\" (UID: \"dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b\") " pod="openshift-multus/network-metrics-daemon-vzfhc" Nov 03 01:52:04.109274 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.108938 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7d2438ed-bfc2-4639-b87d-84689a82123e-konnectivity-ca\") pod \"konnectivity-agent-f524q\" (UID: \"7d2438ed-bfc2-4639-b87d-84689a82123e\") " pod="kube-system/konnectivity-agent-f524q" Nov 03 01:52:04.109274 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.108961 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4480cca6-b2e9-4840-ac8c-a88a40ed90f2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2q7kh\" (UID: \"4480cca6-b2e9-4840-ac8c-a88a40ed90f2\") " pod="openshift-monitoring/node-exporter-2q7kh" Nov 03 01:52:04.109274 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.109003 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/df76407a-ce41-4a4a-b51e-8420b817f97c-etc-tuned\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:52:04.109274 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.109018 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/592adf6b-1edd-4f5e-88a2-6ed92895f22a-serviceca\") pod \"node-ca-gjs9f\" (UID: \"592adf6b-1edd-4f5e-88a2-6ed92895f22a\") " pod="openshift-image-registry/node-ca-gjs9f" Nov 03 01:52:04.109274 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.109036 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e32389d5-705f-4202-98ea-6fc6a5ad5d54-metrics-server-audit-profiles\") pod \"metrics-server-b984f6b6b-g6f2h\" (UID: \"e32389d5-705f-4202-98ea-6fc6a5ad5d54\") " pod="openshift-monitoring/metrics-server-b984f6b6b-g6f2h" Nov 03 01:52:04.109274 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.109106 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/13fa5796-b3e6-4d1c-9759-88b7094d4b37-console-serving-cert\") pod \"console-676fb595d5-lmt4w\" (UID: \"13fa5796-b3e6-4d1c-9759-88b7094d4b37\") " pod="openshift-console/console-676fb595d5-lmt4w" Nov 03 01:52:04.109274 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.109122 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-etc-kubernetes\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:52:04.109274 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.109131 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/df76407a-ce41-4a4a-b51e-8420b817f97c-etc-tuned\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:52:04.109274 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.109174 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3bc52103-b80b-49a8-9bed-15b196fdef12-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dmrb8\" (UID: \"3bc52103-b80b-49a8-9bed-15b196fdef12\") " pod="openshift-insights/insights-runtime-extractor-dmrb8" Nov 03 01:52:04.109274 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.109223 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk6qs\" (UniqueName: \"kubernetes.io/projected/3bc52103-b80b-49a8-9bed-15b196fdef12-kube-api-access-xk6qs\") pod \"insights-runtime-extractor-dmrb8\" (UID: \"3bc52103-b80b-49a8-9bed-15b196fdef12\") " pod="openshift-insights/insights-runtime-extractor-dmrb8" Nov 03 01:52:04.109981 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.109293 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/923b90de-5e45-4add-b092-91af16b7dbad-serving-certs-ca-bundle\") pod \"telemeter-client-699d68669f-9jg2q\" (UID: \"923b90de-5e45-4add-b092-91af16b7dbad\") " pod="openshift-monitoring/telemeter-client-699d68669f-9jg2q" Nov 03 01:52:04.109981 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.109323 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4727262-eb15-4453-adf2-0fc6c2358508-tmp\") pod \"console-69c467f7c8-cpfwh\" (UID: \"a4727262-eb15-4453-adf2-0fc6c2358508\") " pod="openshift-console/console-69c467f7c8-cpfwh" Nov 03 01:52:04.109981 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.109341 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl9gt\" (UniqueName: \"kubernetes.io/projected/3d7d07f7-27c2-489a-8e44-fe366584a267-kube-api-access-nl9gt\") pod \"alertmanager-main-0\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:04.109981 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.109378 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3bc52103-b80b-49a8-9bed-15b196fdef12-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dmrb8\" (UID: \"3bc52103-b80b-49a8-9bed-15b196fdef12\") " pod="openshift-insights/insights-runtime-extractor-dmrb8" Nov 03 01:52:04.109981 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.109394 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/13fa5796-b3e6-4d1c-9759-88b7094d4b37-console-config\") pod \"console-676fb595d5-lmt4w\" (UID: \"13fa5796-b3e6-4d1c-9759-88b7094d4b37\") " pod="openshift-console/console-676fb595d5-lmt4w" Nov 03 01:52:04.109981 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.109398 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3bc52103-b80b-49a8-9bed-15b196fdef12-crio-socket\") pod \"insights-runtime-extractor-dmrb8\" (UID: \"3bc52103-b80b-49a8-9bed-15b196fdef12\") " pod="openshift-insights/insights-runtime-extractor-dmrb8" Nov 03 01:52:04.109981 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.109416 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4727262-eb15-4453-adf2-0fc6c2358508-tmp\") pod \"console-69c467f7c8-cpfwh\" (UID: \"a4727262-eb15-4453-adf2-0fc6c2358508\") " pod="openshift-console/console-69c467f7c8-cpfwh" Nov 03 01:52:04.109981 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.109508 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-console-config\") pod \"console-59fbbdf587-6nm8q\" (UID: \"3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6\") " pod="openshift-console/console-59fbbdf587-6nm8q" Nov 03 01:52:04.109981 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.109542 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/923b90de-5e45-4add-b092-91af16b7dbad-serving-certs-ca-bundle\") pod \"telemeter-client-699d68669f-9jg2q\" (UID: \"923b90de-5e45-4add-b092-91af16b7dbad\") " pod="openshift-monitoring/telemeter-client-699d68669f-9jg2q" Nov 03 01:52:04.109981 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.109553 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5d514c70-f1a7-462a-bd3e-6f5bdc016c39-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-jmj4g\" (UID: \"5d514c70-f1a7-462a-bd3e-6f5bdc016c39\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jmj4g" Nov 03 01:52:04.109981 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.109589 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4480cca6-b2e9-4840-ac8c-a88a40ed90f2-sys\") pod \"node-exporter-2q7kh\" (UID: \"4480cca6-b2e9-4840-ac8c-a88a40ed90f2\") " pod="openshift-monitoring/node-exporter-2q7kh" Nov 03 01:52:04.109981 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.109633 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/28f00eb6-78c2-4d93-bde5-c1ea12601dbd-original-pull-secret\") pod \"global-pull-secret-syncer-kkngd\" (UID: \"28f00eb6-78c2-4d93-bde5-c1ea12601dbd\") " pod="kube-system/global-pull-secret-syncer-kkngd" Nov 03 01:52:04.109981 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.109754 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-host-cni-bin\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:52:04.109981 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.109771 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5d514c70-f1a7-462a-bd3e-6f5bdc016c39-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-jmj4g\" (UID: \"5d514c70-f1a7-462a-bd3e-6f5bdc016c39\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jmj4g" Nov 03 01:52:04.109981 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.109803 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-cni-binary-copy\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:52:04.109981 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.109865 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-multus-socket-dir-parent\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:52:04.109981 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.109943 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-host-var-lib-kubelet\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:52:04.109981 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.109976 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-hostroot\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:52:04.109981 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.109992 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-cni-binary-copy\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:52:04.110701 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.109991 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/28f00eb6-78c2-4d93-bde5-c1ea12601dbd-original-pull-secret\") pod \"global-pull-secret-syncer-kkngd\" (UID: \"28f00eb6-78c2-4d93-bde5-c1ea12601dbd\") " pod="kube-system/global-pull-secret-syncer-kkngd" Nov 03 01:52:04.110701 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.110005 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c3f175d-932c-42be-9660-e4fd3dceaf49-tmp-dir\") pod \"node-resolver-7zdvv\" (UID: \"7c3f175d-932c-42be-9660-e4fd3dceaf49\") " pod="openshift-dns/node-resolver-7zdvv" Nov 03 01:52:04.110701 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.110061 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c3f175d-932c-42be-9660-e4fd3dceaf49-tmp-dir\") pod \"node-resolver-7zdvv\" (UID: \"7c3f175d-932c-42be-9660-e4fd3dceaf49\") " pod="openshift-dns/node-resolver-7zdvv" Nov 03 01:52:04.110701 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.110078 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-console-config\") pod \"console-59fbbdf587-6nm8q\" (UID: \"3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6\") " pod="openshift-console/console-59fbbdf587-6nm8q" Nov 03 01:52:04.110701 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.110100 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqvvj\" (UniqueName: \"kubernetes.io/projected/52b546b3-7bf6-4239-91cb-c8aaca1c9d18-kube-api-access-pqvvj\") pod \"dns-default-t2dwr\" (UID: \"52b546b3-7bf6-4239-91cb-c8aaca1c9d18\") " pod="openshift-dns/dns-default-t2dwr" Nov 03 01:52:04.110701 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.110126 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/923b90de-5e45-4add-b092-91af16b7dbad-telemeter-trusted-ca-bundle\") pod \"telemeter-client-699d68669f-9jg2q\" (UID: \"923b90de-5e45-4add-b092-91af16b7dbad\") " pod="openshift-monitoring/telemeter-client-699d68669f-9jg2q" Nov 03 01:52:04.110701 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.110208 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/56117bce-d686-4c24-98c8-c0ff122778ac-env-overrides\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:52:04.110701 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.110398 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-config-volume\") pod \"alertmanager-main-0\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:04.110701 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.110404 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/56117bce-d686-4c24-98c8-c0ff122778ac-env-overrides\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:52:04.110701 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.110439 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a4727262-eb15-4453-adf2-0fc6c2358508-console-config\") pod \"console-69c467f7c8-cpfwh\" (UID: \"a4727262-eb15-4453-adf2-0fc6c2358508\") " pod="openshift-console/console-69c467f7c8-cpfwh" Nov 03 01:52:04.110701 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.110459 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/923b90de-5e45-4add-b092-91af16b7dbad-telemeter-trusted-ca-bundle\") pod \"telemeter-client-699d68669f-9jg2q\" (UID: \"923b90de-5e45-4add-b092-91af16b7dbad\") " pod="openshift-monitoring/telemeter-client-699d68669f-9jg2q" Nov 03 01:52:04.110701 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.110542 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7d2438ed-bfc2-4639-b87d-84689a82123e-agent-certs\") pod \"konnectivity-agent-f524q\" (UID: \"7d2438ed-bfc2-4639-b87d-84689a82123e\") " pod="kube-system/konnectivity-agent-f524q" Nov 03 01:52:04.110701 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.110564 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-config-volume\") pod \"alertmanager-main-0\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:04.110701 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.110587 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqp5v\" (UniqueName: \"kubernetes.io/projected/0801f13c-d14a-4b9e-b45a-a51af48d17c3-kube-api-access-qqp5v\") pod \"aws-ebs-csi-driver-node-xdpjb\" (UID: \"0801f13c-d14a-4b9e-b45a-a51af48d17c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdpjb" Nov 03 01:52:04.110701 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.110610 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-host-slash\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:52:04.110701 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.110654 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/df76407a-ce41-4a4a-b51e-8420b817f97c-etc-modprobe-d\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:52:04.110701 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.110683 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/df76407a-ce41-4a4a-b51e-8420b817f97c-etc-systemd\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:52:04.111397 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.110717 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/52b546b3-7bf6-4239-91cb-c8aaca1c9d18-metrics-tls\") pod \"dns-default-t2dwr\" (UID: \"52b546b3-7bf6-4239-91cb-c8aaca1c9d18\") " pod="openshift-dns/dns-default-t2dwr" Nov 03 01:52:04.111397 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.110724 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a4727262-eb15-4453-adf2-0fc6c2358508-console-config\") pod \"console-69c467f7c8-cpfwh\" (UID: \"a4727262-eb15-4453-adf2-0fc6c2358508\") " pod="openshift-console/console-69c467f7c8-cpfwh" Nov 03 01:52:04.111397 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.110741 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:04.111397 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.110767 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb4tp\" (UniqueName: \"kubernetes.io/projected/13fa5796-b3e6-4d1c-9759-88b7094d4b37-kube-api-access-bb4tp\") pod \"console-676fb595d5-lmt4w\" (UID: \"13fa5796-b3e6-4d1c-9759-88b7094d4b37\") " pod="openshift-console/console-676fb595d5-lmt4w" Nov 03 01:52:04.111397 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.110794 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/56117bce-d686-4c24-98c8-c0ff122778ac-ovnkube-script-lib\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:52:04.111397 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.110821 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw55q\" (UniqueName: \"kubernetes.io/projected/592adf6b-1edd-4f5e-88a2-6ed92895f22a-kube-api-access-bw55q\") pod \"node-ca-gjs9f\" (UID: \"592adf6b-1edd-4f5e-88a2-6ed92895f22a\") " pod="openshift-image-registry/node-ca-gjs9f" Nov 03 01:52:04.111397 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.110852 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-system-cni-dir\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:52:04.111397 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.110865 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7d2438ed-bfc2-4639-b87d-84689a82123e-agent-certs\") pod \"konnectivity-agent-f524q\" (UID: \"7d2438ed-bfc2-4639-b87d-84689a82123e\") " pod="kube-system/konnectivity-agent-f524q" Nov 03 01:52:04.111397 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.110875 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-host-run-netns\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:52:04.111397 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.110892 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:04.111397 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.110898 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-host-var-lib-cni-multus\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:52:04.111397 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.110936 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/52b546b3-7bf6-4239-91cb-c8aaca1c9d18-metrics-tls\") pod \"dns-default-t2dwr\" (UID: \"52b546b3-7bf6-4239-91cb-c8aaca1c9d18\") " pod="openshift-dns/dns-default-t2dwr" Nov 03 01:52:04.111397 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.110942 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfgdw\" (UniqueName: \"kubernetes.io/projected/9152cf36-4997-470e-8902-802a96ebc2c7-kube-api-access-bfgdw\") pod \"multus-additional-cni-plugins-gwlr2\" (UID: \"9152cf36-4997-470e-8902-802a96ebc2c7\") " pod="openshift-multus/multus-additional-cni-plugins-gwlr2" Nov 03 01:52:04.111397 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.110969 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtzct\" (UniqueName: \"kubernetes.io/projected/0f363d1e-561d-43ae-9d53-379c86b2a558-kube-api-access-dtzct\") pod \"ingress-canary-2d49s\" (UID: \"0f363d1e-561d-43ae-9d53-379c86b2a558\") " pod="openshift-ingress-canary/ingress-canary-2d49s" Nov 03 01:52:04.111397 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.111025 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/56117bce-d686-4c24-98c8-c0ff122778ac-ovnkube-script-lib\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:52:04.111397 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.111070 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3d7d07f7-27c2-489a-8e44-fe366584a267-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:04.111397 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.111143 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/13fa5796-b3e6-4d1c-9759-88b7094d4b37-console-oauth-config\") pod \"console-676fb595d5-lmt4w\" (UID: \"13fa5796-b3e6-4d1c-9759-88b7094d4b37\") " pod="openshift-console/console-676fb595d5-lmt4w" Nov 03 01:52:04.111397 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.111171 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4480cca6-b2e9-4840-ac8c-a88a40ed90f2-root\") pod \"node-exporter-2q7kh\" (UID: \"4480cca6-b2e9-4840-ac8c-a88a40ed90f2\") " pod="openshift-monitoring/node-exporter-2q7kh" Nov 03 01:52:04.111397 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.111196 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ccfs\" (UniqueName: \"kubernetes.io/projected/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-kube-api-access-7ccfs\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:52:04.111397 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.111218 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b-metrics-certs\") pod \"network-metrics-daemon-vzfhc\" (UID: \"dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b\") " pod="openshift-multus/network-metrics-daemon-vzfhc" Nov 03 01:52:04.111397 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.111232 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3d7d07f7-27c2-489a-8e44-fe366584a267-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:04.111397 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.111253 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/923b90de-5e45-4add-b092-91af16b7dbad-telemeter-client-tls\") pod \"telemeter-client-699d68669f-9jg2q\" (UID: \"923b90de-5e45-4add-b092-91af16b7dbad\") " pod="openshift-monitoring/telemeter-client-699d68669f-9jg2q" Nov 03 01:52:04.111397 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.111273 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/923b90de-5e45-4add-b092-91af16b7dbad-federate-client-tls\") pod \"telemeter-client-699d68669f-9jg2q\" (UID: \"923b90de-5e45-4add-b092-91af16b7dbad\") " pod="openshift-monitoring/telemeter-client-699d68669f-9jg2q" Nov 03 01:52:04.111397 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.111304 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/13fa5796-b3e6-4d1c-9759-88b7094d4b37-console-oauth-config\") pod \"console-676fb595d5-lmt4w\" (UID: \"13fa5796-b3e6-4d1c-9759-88b7094d4b37\") " pod="openshift-console/console-676fb595d5-lmt4w" Nov 03 01:52:04.111397 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.111325 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-host-run-netns\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:52:04.111397 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.111370 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-node-log\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:52:04.111397 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.111406 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kgsp\" (UniqueName: \"kubernetes.io/projected/56117bce-d686-4c24-98c8-c0ff122778ac-kube-api-access-5kgsp\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:52:04.112284 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.111447 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/13fa5796-b3e6-4d1c-9759-88b7094d4b37-service-ca\") pod \"console-676fb595d5-lmt4w\" (UID: \"13fa5796-b3e6-4d1c-9759-88b7094d4b37\") " pod="openshift-console/console-676fb595d5-lmt4w" Nov 03 01:52:04.112284 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.111455 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b-metrics-certs\") pod \"network-metrics-daemon-vzfhc\" (UID: \"dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b\") " pod="openshift-multus/network-metrics-daemon-vzfhc" Nov 03 01:52:04.112284 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.111510 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/923b90de-5e45-4add-b092-91af16b7dbad-telemeter-client-tls\") pod \"telemeter-client-699d68669f-9jg2q\" (UID: \"923b90de-5e45-4add-b092-91af16b7dbad\") " pod="openshift-monitoring/telemeter-client-699d68669f-9jg2q" Nov 03 01:52:04.112284 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.111517 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/923b90de-5e45-4add-b092-91af16b7dbad-federate-client-tls\") pod \"telemeter-client-699d68669f-9jg2q\" (UID: \"923b90de-5e45-4add-b092-91af16b7dbad\") " pod="openshift-monitoring/telemeter-client-699d68669f-9jg2q" Nov 03 01:52:04.112284 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.111545 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l75kg\" (UniqueName: \"kubernetes.io/projected/923b90de-5e45-4add-b092-91af16b7dbad-kube-api-access-l75kg\") pod \"telemeter-client-699d68669f-9jg2q\" (UID: \"923b90de-5e45-4add-b092-91af16b7dbad\") " pod="openshift-monitoring/telemeter-client-699d68669f-9jg2q" Nov 03 01:52:04.112284 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.111592 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-etc-openvswitch\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:52:04.112284 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.111608 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e32389d5-705f-4202-98ea-6fc6a5ad5d54-client-ca-bundle\") pod \"metrics-server-b984f6b6b-g6f2h\" (UID: \"e32389d5-705f-4202-98ea-6fc6a5ad5d54\") " pod="openshift-monitoring/metrics-server-b984f6b6b-g6f2h" Nov 03 01:52:04.112284 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.111623 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/df76407a-ce41-4a4a-b51e-8420b817f97c-etc-kubernetes\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:52:04.112284 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.111663 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/df76407a-ce41-4a4a-b51e-8420b817f97c-etc-sysctl-conf\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:52:04.112284 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.111677 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/df76407a-ce41-4a4a-b51e-8420b817f97c-sys\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:52:04.112284 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.111714 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/13fa5796-b3e6-4d1c-9759-88b7094d4b37-service-ca\") pod \"console-676fb595d5-lmt4w\" (UID: \"13fa5796-b3e6-4d1c-9759-88b7094d4b37\") " pod="openshift-console/console-676fb595d5-lmt4w" Nov 03 01:52:04.112284 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.111791 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9152cf36-4997-470e-8902-802a96ebc2c7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gwlr2\" (UID: \"9152cf36-4997-470e-8902-802a96ebc2c7\") " pod="openshift-multus/multus-additional-cni-plugins-gwlr2" Nov 03 01:52:04.112284 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.111831 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/85148983-aa3f-45c2-a062-da2748dd9011-iptables-alerter-script\") pod \"iptables-alerter-vmjc5\" (UID: \"85148983-aa3f-45c2-a062-da2748dd9011\") " pod="openshift-network-operator/iptables-alerter-vmjc5" Nov 03 01:52:04.112284 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.111845 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e32389d5-705f-4202-98ea-6fc6a5ad5d54-client-ca-bundle\") pod \"metrics-server-b984f6b6b-g6f2h\" (UID: \"e32389d5-705f-4202-98ea-6fc6a5ad5d54\") " pod="openshift-monitoring/metrics-server-b984f6b6b-g6f2h" Nov 03 01:52:04.112284 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.111863 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-tmp\") pod \"console-59fbbdf587-6nm8q\" (UID: \"3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6\") " pod="openshift-console/console-59fbbdf587-6nm8q" Nov 03 01:52:04.112284 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.111950 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-systemd-units\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:52:04.112284 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.111955 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9152cf36-4997-470e-8902-802a96ebc2c7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gwlr2\" (UID: \"9152cf36-4997-470e-8902-802a96ebc2c7\") " pod="openshift-multus/multus-additional-cni-plugins-gwlr2" Nov 03 01:52:04.112284 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.111967 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgvhv\" (UniqueName: \"kubernetes.io/projected/7c3f175d-932c-42be-9660-e4fd3dceaf49-kube-api-access-pgvhv\") pod \"node-resolver-7zdvv\" (UID: \"7c3f175d-932c-42be-9660-e4fd3dceaf49\") " pod="openshift-dns/node-resolver-7zdvv" Nov 03 01:52:04.112284 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.111971 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-tmp\") pod \"console-59fbbdf587-6nm8q\" (UID: \"3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6\") " pod="openshift-console/console-59fbbdf587-6nm8q" Nov 03 01:52:04.112284 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.111999 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4480cca6-b2e9-4840-ac8c-a88a40ed90f2-node-exporter-accelerators-collector-config\") pod \"node-exporter-2q7kh\" (UID: \"4480cca6-b2e9-4840-ac8c-a88a40ed90f2\") " pod="openshift-monitoring/node-exporter-2q7kh" Nov 03 01:52:04.112284 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.112011 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/85148983-aa3f-45c2-a062-da2748dd9011-iptables-alerter-script\") pod \"iptables-alerter-vmjc5\" (UID: \"85148983-aa3f-45c2-a062-da2748dd9011\") " pod="openshift-network-operator/iptables-alerter-vmjc5" Nov 03 01:52:04.112284 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.112036 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9kwt\" (UniqueName: \"kubernetes.io/projected/a4727262-eb15-4453-adf2-0fc6c2358508-kube-api-access-d9kwt\") pod \"console-69c467f7c8-cpfwh\" (UID: \"a4727262-eb15-4453-adf2-0fc6c2358508\") " pod="openshift-console/console-69c467f7c8-cpfwh" Nov 03 01:52:04.112284 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.112056 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9152cf36-4997-470e-8902-802a96ebc2c7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gwlr2\" (UID: \"9152cf36-4997-470e-8902-802a96ebc2c7\") " pod="openshift-multus/multus-additional-cni-plugins-gwlr2" Nov 03 01:52:04.112284 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.112074 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0801f13c-d14a-4b9e-b45a-a51af48d17c3-registration-dir\") pod \"aws-ebs-csi-driver-node-xdpjb\" (UID: \"0801f13c-d14a-4b9e-b45a-a51af48d17c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdpjb" Nov 03 01:52:04.112284 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.112093 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-log-socket\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:52:04.112284 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.112160 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4480cca6-b2e9-4840-ac8c-a88a40ed90f2-node-exporter-accelerators-collector-config\") pod \"node-exporter-2q7kh\" (UID: \"4480cca6-b2e9-4840-ac8c-a88a40ed90f2\") " pod="openshift-monitoring/node-exporter-2q7kh" Nov 03 01:52:04.112284 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.112230 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e32389d5-705f-4202-98ea-6fc6a5ad5d54-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-b984f6b6b-g6f2h\" (UID: \"e32389d5-705f-4202-98ea-6fc6a5ad5d54\") " pod="openshift-monitoring/metrics-server-b984f6b6b-g6f2h" Nov 03 01:52:04.112284 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.112261 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3bc52103-b80b-49a8-9bed-15b196fdef12-data-volume\") pod \"insights-runtime-extractor-dmrb8\" (UID: \"3bc52103-b80b-49a8-9bed-15b196fdef12\") " pod="openshift-insights/insights-runtime-extractor-dmrb8" Nov 03 01:52:04.112284 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.112281 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-multus-conf-dir\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:52:04.113443 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.112373 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3bc52103-b80b-49a8-9bed-15b196fdef12-data-volume\") pod \"insights-runtime-extractor-dmrb8\" (UID: \"3bc52103-b80b-49a8-9bed-15b196fdef12\") " pod="openshift-insights/insights-runtime-extractor-dmrb8" Nov 03 01:52:04.113443 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.112389 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52b546b3-7bf6-4239-91cb-c8aaca1c9d18-tmp-dir\") pod \"dns-default-t2dwr\" (UID: \"52b546b3-7bf6-4239-91cb-c8aaca1c9d18\") " pod="openshift-dns/dns-default-t2dwr" Nov 03 01:52:04.113443 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.112419 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d7d07f7-27c2-489a-8e44-fe366584a267-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:04.113443 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.112434 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4480cca6-b2e9-4840-ac8c-a88a40ed90f2-metrics-client-ca\") pod \"node-exporter-2q7kh\" (UID: \"4480cca6-b2e9-4840-ac8c-a88a40ed90f2\") " pod="openshift-monitoring/node-exporter-2q7kh" Nov 03 01:52:04.113443 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.112451 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkn5x\" (UniqueName: \"kubernetes.io/projected/4480cca6-b2e9-4840-ac8c-a88a40ed90f2-kube-api-access-bkn5x\") pod \"node-exporter-2q7kh\" (UID: \"4480cca6-b2e9-4840-ac8c-a88a40ed90f2\") " pod="openshift-monitoring/node-exporter-2q7kh" Nov 03 01:52:04.113443 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.112465 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/56117bce-d686-4c24-98c8-c0ff122778ac-ovnkube-config\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:52:04.113443 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.112466 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e32389d5-705f-4202-98ea-6fc6a5ad5d54-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-b984f6b6b-g6f2h\" (UID: \"e32389d5-705f-4202-98ea-6fc6a5ad5d54\") " pod="openshift-monitoring/metrics-server-b984f6b6b-g6f2h" Nov 03 01:52:04.113443 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.112484 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/592adf6b-1edd-4f5e-88a2-6ed92895f22a-host\") pod \"node-ca-gjs9f\" (UID: \"592adf6b-1edd-4f5e-88a2-6ed92895f22a\") " pod="openshift-image-registry/node-ca-gjs9f" Nov 03 01:52:04.113443 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.112488 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52b546b3-7bf6-4239-91cb-c8aaca1c9d18-tmp-dir\") pod \"dns-default-t2dwr\" (UID: \"52b546b3-7bf6-4239-91cb-c8aaca1c9d18\") " pod="openshift-dns/dns-default-t2dwr" Nov 03 01:52:04.113443 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.112508 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr2jw\" (UniqueName: \"kubernetes.io/projected/6ae1437e-8885-42a5-b092-3862cd404c60-kube-api-access-sr2jw\") pod \"network-check-target-f5qvs\" (UID: \"6ae1437e-8885-42a5-b092-3862cd404c60\") " pod="openshift-network-diagnostics/network-check-target-f5qvs" Nov 03 01:52:04.113443 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.112550 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:04.113443 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.112651 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4480cca6-b2e9-4840-ac8c-a88a40ed90f2-metrics-client-ca\") pod \"node-exporter-2q7kh\" (UID: \"4480cca6-b2e9-4840-ac8c-a88a40ed90f2\") " pod="openshift-monitoring/node-exporter-2q7kh" Nov 03 01:52:04.113443 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.112664 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0801f13c-d14a-4b9e-b45a-a51af48d17c3-device-dir\") pod \"aws-ebs-csi-driver-node-xdpjb\" (UID: \"0801f13c-d14a-4b9e-b45a-a51af48d17c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdpjb" Nov 03 01:52:04.113443 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.112685 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-console-oauth-config\") pod \"console-59fbbdf587-6nm8q\" (UID: \"3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6\") " pod="openshift-console/console-59fbbdf587-6nm8q" Nov 03 01:52:04.113443 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.112702 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/923b90de-5e45-4add-b092-91af16b7dbad-secret-telemeter-client\") pod \"telemeter-client-699d68669f-9jg2q\" (UID: \"923b90de-5e45-4add-b092-91af16b7dbad\") " pod="openshift-monitoring/telemeter-client-699d68669f-9jg2q" Nov 03 01:52:04.113443 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.112718 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a4727262-eb15-4453-adf2-0fc6c2358508-oauth-serving-cert\") pod \"console-69c467f7c8-cpfwh\" (UID: \"a4727262-eb15-4453-adf2-0fc6c2358508\") " pod="openshift-console/console-69c467f7c8-cpfwh" Nov 03 01:52:04.113443 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.112733 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv9fz\" (UniqueName: \"kubernetes.io/projected/e32389d5-705f-4202-98ea-6fc6a5ad5d54-kube-api-access-kv9fz\") pod \"metrics-server-b984f6b6b-g6f2h\" (UID: \"e32389d5-705f-4202-98ea-6fc6a5ad5d54\") " pod="openshift-monitoring/metrics-server-b984f6b6b-g6f2h" Nov 03 01:52:04.113443 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.112743 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d7d07f7-27c2-489a-8e44-fe366584a267-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:04.113443 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.112758 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:04.113443 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.112748 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3d7d07f7-27c2-489a-8e44-fe366584a267-config-out\") pod \"alertmanager-main-0\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:04.113443 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.112784 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/56117bce-d686-4c24-98c8-c0ff122778ac-ovnkube-config\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:52:04.113443 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.112793 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a4727262-eb15-4453-adf2-0fc6c2358508-console-serving-cert\") pod \"console-69c467f7c8-cpfwh\" (UID: \"a4727262-eb15-4453-adf2-0fc6c2358508\") " pod="openshift-console/console-69c467f7c8-cpfwh" Nov 03 01:52:04.113443 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.112797 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3d7d07f7-27c2-489a-8e44-fe366584a267-config-out\") pod \"alertmanager-main-0\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:04.113443 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.112897 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-console-oauth-config\") pod \"console-59fbbdf587-6nm8q\" (UID: \"3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6\") " pod="openshift-console/console-59fbbdf587-6nm8q" Nov 03 01:52:04.113443 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.112961 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a4727262-eb15-4453-adf2-0fc6c2358508-service-ca\") pod \"console-69c467f7c8-cpfwh\" (UID: \"a4727262-eb15-4453-adf2-0fc6c2358508\") " pod="openshift-console/console-69c467f7c8-cpfwh" Nov 03 01:52:04.113443 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.112978 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0801f13c-d14a-4b9e-b45a-a51af48d17c3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xdpjb\" (UID: \"0801f13c-d14a-4b9e-b45a-a51af48d17c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdpjb" Nov 03 01:52:04.113443 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.112988 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/923b90de-5e45-4add-b092-91af16b7dbad-secret-telemeter-client\") pod \"telemeter-client-699d68669f-9jg2q\" (UID: \"923b90de-5e45-4add-b092-91af16b7dbad\") " pod="openshift-monitoring/telemeter-client-699d68669f-9jg2q" Nov 03 01:52:04.113443 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.112995 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-host-kubelet\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:52:04.113443 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.112961 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a4727262-eb15-4453-adf2-0fc6c2358508-console-serving-cert\") pod \"console-69c467f7c8-cpfwh\" (UID: \"a4727262-eb15-4453-adf2-0fc6c2358508\") " pod="openshift-console/console-69c467f7c8-cpfwh" Nov 03 01:52:04.113443 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.113034 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:04.113443 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.113050 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d7d07f7-27c2-489a-8e44-fe366584a267-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:04.113443 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.113070 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a4727262-eb15-4453-adf2-0fc6c2358508-oauth-serving-cert\") pod \"console-69c467f7c8-cpfwh\" (UID: \"a4727262-eb15-4453-adf2-0fc6c2358508\") " pod="openshift-console/console-69c467f7c8-cpfwh" Nov 03 01:52:04.113443 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.113082 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13fa5796-b3e6-4d1c-9759-88b7094d4b37-trusted-ca-bundle\") pod \"console-676fb595d5-lmt4w\" (UID: \"13fa5796-b3e6-4d1c-9759-88b7094d4b37\") " pod="openshift-console/console-676fb595d5-lmt4w" Nov 03 01:52:04.113443 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.113109 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0f363d1e-561d-43ae-9d53-379c86b2a558-cert\") pod \"ingress-canary-2d49s\" (UID: \"0f363d1e-561d-43ae-9d53-379c86b2a558\") " pod="openshift-ingress-canary/ingress-canary-2d49s" Nov 03 01:52:04.113443 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.113173 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d7d07f7-27c2-489a-8e44-fe366584a267-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:04.113443 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.113175 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a4727262-eb15-4453-adf2-0fc6c2358508-service-ca\") pod \"console-69c467f7c8-cpfwh\" (UID: \"a4727262-eb15-4453-adf2-0fc6c2358508\") " pod="openshift-console/console-69c467f7c8-cpfwh" Nov 03 01:52:04.113443 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.113194 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df76407a-ce41-4a4a-b51e-8420b817f97c-tmp\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:52:04.113443 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.113209 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/13fa5796-b3e6-4d1c-9759-88b7094d4b37-oauth-serving-cert\") pod \"console-676fb595d5-lmt4w\" (UID: \"13fa5796-b3e6-4d1c-9759-88b7094d4b37\") " pod="openshift-console/console-676fb595d5-lmt4w" Nov 03 01:52:04.113443 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.113217 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:04.113443 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.113224 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3bc52103-b80b-49a8-9bed-15b196fdef12-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dmrb8\" (UID: \"3bc52103-b80b-49a8-9bed-15b196fdef12\") " pod="openshift-insights/insights-runtime-extractor-dmrb8" Nov 03 01:52:04.113443 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.113241 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4727262-eb15-4453-adf2-0fc6c2358508-trusted-ca-bundle\") pod \"console-69c467f7c8-cpfwh\" (UID: \"a4727262-eb15-4453-adf2-0fc6c2358508\") " pod="openshift-console/console-69c467f7c8-cpfwh" Nov 03 01:52:04.113443 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.113257 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0801f13c-d14a-4b9e-b45a-a51af48d17c3-sys-fs\") pod \"aws-ebs-csi-driver-node-xdpjb\" (UID: \"0801f13c-d14a-4b9e-b45a-a51af48d17c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdpjb" Nov 03 01:52:04.113443 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.113264 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df76407a-ce41-4a4a-b51e-8420b817f97c-tmp\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:52:04.113443 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.113273 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-host-run-multus-certs\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:52:04.113443 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.113298 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5d514c70-f1a7-462a-bd3e-6f5bdc016c39-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-jmj4g\" (UID: \"5d514c70-f1a7-462a-bd3e-6f5bdc016c39\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jmj4g" Nov 03 01:52:04.113443 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.113303 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0f363d1e-561d-43ae-9d53-379c86b2a558-cert\") pod \"ingress-canary-2d49s\" (UID: \"0f363d1e-561d-43ae-9d53-379c86b2a558\") " pod="openshift-ingress-canary/ingress-canary-2d49s" Nov 03 01:52:04.113443 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.113352 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/13fa5796-b3e6-4d1c-9759-88b7094d4b37-oauth-serving-cert\") pod \"console-676fb595d5-lmt4w\" (UID: \"13fa5796-b3e6-4d1c-9759-88b7094d4b37\") " pod="openshift-console/console-676fb595d5-lmt4w" Nov 03 01:52:04.113443 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.113373 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13fa5796-b3e6-4d1c-9759-88b7094d4b37-trusted-ca-bundle\") pod \"console-676fb595d5-lmt4w\" (UID: \"13fa5796-b3e6-4d1c-9759-88b7094d4b37\") " pod="openshift-console/console-676fb595d5-lmt4w" Nov 03 01:52:04.113443 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.113337 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0801f13c-d14a-4b9e-b45a-a51af48d17c3-etc-selinux\") pod \"aws-ebs-csi-driver-node-xdpjb\" (UID: \"0801f13c-d14a-4b9e-b45a-a51af48d17c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdpjb" Nov 03 01:52:04.113443 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.113455 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-multus-cni-dir\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:52:04.114851 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.113483 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3bc52103-b80b-49a8-9bed-15b196fdef12-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dmrb8\" (UID: \"3bc52103-b80b-49a8-9bed-15b196fdef12\") " pod="openshift-insights/insights-runtime-extractor-dmrb8" Nov 03 01:52:04.114851 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.113492 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/85148983-aa3f-45c2-a062-da2748dd9011-host-slash\") pod \"iptables-alerter-vmjc5\" (UID: \"85148983-aa3f-45c2-a062-da2748dd9011\") " pod="openshift-network-operator/iptables-alerter-vmjc5" Nov 03 01:52:04.114851 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.113497 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5d514c70-f1a7-462a-bd3e-6f5bdc016c39-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-jmj4g\" (UID: \"5d514c70-f1a7-462a-bd3e-6f5bdc016c39\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jmj4g" Nov 03 01:52:04.114851 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.113527 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhl6w\" (UniqueName: \"kubernetes.io/projected/5d514c70-f1a7-462a-bd3e-6f5bdc016c39-kube-api-access-dhl6w\") pod \"prometheus-operator-9cd89b74c-jmj4g\" (UID: \"5d514c70-f1a7-462a-bd3e-6f5bdc016c39\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jmj4g" Nov 03 01:52:04.114851 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.113565 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:04.114851 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.113671 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/28f00eb6-78c2-4d93-bde5-c1ea12601dbd-dbus\") pod \"global-pull-secret-syncer-kkngd\" (UID: \"28f00eb6-78c2-4d93-bde5-c1ea12601dbd\") " pod="kube-system/global-pull-secret-syncer-kkngd" Nov 03 01:52:04.114851 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.113699 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e32389d5-705f-4202-98ea-6fc6a5ad5d54-audit-log\") pod \"metrics-server-b984f6b6b-g6f2h\" (UID: \"e32389d5-705f-4202-98ea-6fc6a5ad5d54\") " pod="openshift-monitoring/metrics-server-b984f6b6b-g6f2h" Nov 03 01:52:04.114851 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.113796 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:04.114851 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.113799 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e32389d5-705f-4202-98ea-6fc6a5ad5d54-audit-log\") pod \"metrics-server-b984f6b6b-g6f2h\" (UID: \"e32389d5-705f-4202-98ea-6fc6a5ad5d54\") " pod="openshift-monitoring/metrics-server-b984f6b6b-g6f2h" Nov 03 01:52:04.114851 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.113936 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4727262-eb15-4453-adf2-0fc6c2358508-trusted-ca-bundle\") pod \"console-69c467f7c8-cpfwh\" (UID: \"a4727262-eb15-4453-adf2-0fc6c2358508\") " pod="openshift-console/console-69c467f7c8-cpfwh" Nov 03 01:52:04.161558 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.161340 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2l49\" (UniqueName: \"kubernetes.io/projected/dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b-kube-api-access-l2l49\") pod \"network-metrics-daemon-vzfhc\" (UID: \"dd9d0a7f-6414-4490-9449-f0aa1bd2ac3b\") " pod="openshift-multus/network-metrics-daemon-vzfhc" Nov 03 01:52:04.161558 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.161500 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk6qs\" (UniqueName: \"kubernetes.io/projected/3bc52103-b80b-49a8-9bed-15b196fdef12-kube-api-access-xk6qs\") pod \"insights-runtime-extractor-dmrb8\" (UID: \"3bc52103-b80b-49a8-9bed-15b196fdef12\") " pod="openshift-insights/insights-runtime-extractor-dmrb8" Nov 03 01:52:04.161558 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.161534 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qmrm\" (UniqueName: \"kubernetes.io/projected/85148983-aa3f-45c2-a062-da2748dd9011-kube-api-access-2qmrm\") pod \"iptables-alerter-vmjc5\" (UID: \"85148983-aa3f-45c2-a062-da2748dd9011\") " pod="openshift-network-operator/iptables-alerter-vmjc5" Nov 03 01:52:04.161558 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.161534 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffwtk\" (UniqueName: \"kubernetes.io/projected/df76407a-ce41-4a4a-b51e-8420b817f97c-kube-api-access-ffwtk\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:52:04.161831 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.161633 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmhdf\" (UniqueName: \"kubernetes.io/projected/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-kube-api-access-bmhdf\") pod \"console-59fbbdf587-6nm8q\" (UID: \"3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6\") " pod="openshift-console/console-59fbbdf587-6nm8q" Nov 03 01:52:04.161831 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.161717 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqvvj\" (UniqueName: \"kubernetes.io/projected/52b546b3-7bf6-4239-91cb-c8aaca1c9d18-kube-api-access-pqvvj\") pod \"dns-default-t2dwr\" (UID: \"52b546b3-7bf6-4239-91cb-c8aaca1c9d18\") " pod="openshift-dns/dns-default-t2dwr" Nov 03 01:52:04.161831 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.161783 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl9gt\" (UniqueName: \"kubernetes.io/projected/3d7d07f7-27c2-489a-8e44-fe366584a267-kube-api-access-nl9gt\") pod \"alertmanager-main-0\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:04.163889 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.163869 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqp5v\" (UniqueName: \"kubernetes.io/projected/0801f13c-d14a-4b9e-b45a-a51af48d17c3-kube-api-access-qqp5v\") pod \"aws-ebs-csi-driver-node-xdpjb\" (UID: \"0801f13c-d14a-4b9e-b45a-a51af48d17c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdpjb" Nov 03 01:52:04.164304 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.164283 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ccfs\" (UniqueName: \"kubernetes.io/projected/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-kube-api-access-7ccfs\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:52:04.164773 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.164741 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw55q\" (UniqueName: \"kubernetes.io/projected/592adf6b-1edd-4f5e-88a2-6ed92895f22a-kube-api-access-bw55q\") pod \"node-ca-gjs9f\" (UID: \"592adf6b-1edd-4f5e-88a2-6ed92895f22a\") " pod="openshift-image-registry/node-ca-gjs9f" Nov 03 01:52:04.165610 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.164956 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgvhv\" (UniqueName: \"kubernetes.io/projected/7c3f175d-932c-42be-9660-e4fd3dceaf49-kube-api-access-pgvhv\") pod \"node-resolver-7zdvv\" (UID: \"7c3f175d-932c-42be-9660-e4fd3dceaf49\") " pod="openshift-dns/node-resolver-7zdvv" Nov 03 01:52:04.165610 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.165100 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkn5x\" (UniqueName: \"kubernetes.io/projected/4480cca6-b2e9-4840-ac8c-a88a40ed90f2-kube-api-access-bkn5x\") pod \"node-exporter-2q7kh\" (UID: \"4480cca6-b2e9-4840-ac8c-a88a40ed90f2\") " pod="openshift-monitoring/node-exporter-2q7kh" Nov 03 01:52:04.165610 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.165224 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfgdw\" (UniqueName: \"kubernetes.io/projected/9152cf36-4997-470e-8902-802a96ebc2c7-kube-api-access-bfgdw\") pod \"multus-additional-cni-plugins-gwlr2\" (UID: \"9152cf36-4997-470e-8902-802a96ebc2c7\") " pod="openshift-multus/multus-additional-cni-plugins-gwlr2" Nov 03 01:52:04.165610 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.165275 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb4tp\" (UniqueName: \"kubernetes.io/projected/13fa5796-b3e6-4d1c-9759-88b7094d4b37-kube-api-access-bb4tp\") pod \"console-676fb595d5-lmt4w\" (UID: \"13fa5796-b3e6-4d1c-9759-88b7094d4b37\") " pod="openshift-console/console-676fb595d5-lmt4w" Nov 03 01:52:04.165610 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.165582 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l75kg\" (UniqueName: \"kubernetes.io/projected/923b90de-5e45-4add-b092-91af16b7dbad-kube-api-access-l75kg\") pod \"telemeter-client-699d68669f-9jg2q\" (UID: \"923b90de-5e45-4add-b092-91af16b7dbad\") " pod="openshift-monitoring/telemeter-client-699d68669f-9jg2q" Nov 03 01:52:04.165815 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.165661 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtzct\" (UniqueName: \"kubernetes.io/projected/0f363d1e-561d-43ae-9d53-379c86b2a558-kube-api-access-dtzct\") pod \"ingress-canary-2d49s\" (UID: \"0f363d1e-561d-43ae-9d53-379c86b2a558\") " pod="openshift-ingress-canary/ingress-canary-2d49s" Nov 03 01:52:04.165815 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.165724 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9kwt\" (UniqueName: \"kubernetes.io/projected/a4727262-eb15-4453-adf2-0fc6c2358508-kube-api-access-d9kwt\") pod \"console-69c467f7c8-cpfwh\" (UID: \"a4727262-eb15-4453-adf2-0fc6c2358508\") " pod="openshift-console/console-69c467f7c8-cpfwh" Nov 03 01:52:04.165815 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.165736 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kgsp\" (UniqueName: \"kubernetes.io/projected/56117bce-d686-4c24-98c8-c0ff122778ac-kube-api-access-5kgsp\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:52:04.165815 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.165763 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr2jw\" (UniqueName: \"kubernetes.io/projected/6ae1437e-8885-42a5-b092-3862cd404c60-kube-api-access-sr2jw\") pod \"network-check-target-f5qvs\" (UID: \"6ae1437e-8885-42a5-b092-3862cd404c60\") " pod="openshift-network-diagnostics/network-check-target-f5qvs" Nov 03 01:52:04.165933 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.165833 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv9fz\" (UniqueName: \"kubernetes.io/projected/e32389d5-705f-4202-98ea-6fc6a5ad5d54-kube-api-access-kv9fz\") pod \"metrics-server-b984f6b6b-g6f2h\" (UID: \"e32389d5-705f-4202-98ea-6fc6a5ad5d54\") " pod="openshift-monitoring/metrics-server-b984f6b6b-g6f2h" Nov 03 01:52:04.165976 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.165960 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhl6w\" (UniqueName: \"kubernetes.io/projected/5d514c70-f1a7-462a-bd3e-6f5bdc016c39-kube-api-access-dhl6w\") pod \"prometheus-operator-9cd89b74c-jmj4g\" (UID: \"5d514c70-f1a7-462a-bd3e-6f5bdc016c39\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jmj4g" Nov 03 01:52:04.214536 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.214506 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-host-run-netns\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:52:04.214536 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.214534 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-host-var-lib-cni-multus\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:52:04.214716 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.214551 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4480cca6-b2e9-4840-ac8c-a88a40ed90f2-root\") pod \"node-exporter-2q7kh\" (UID: \"4480cca6-b2e9-4840-ac8c-a88a40ed90f2\") " pod="openshift-monitoring/node-exporter-2q7kh" Nov 03 01:52:04.214716 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.214566 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-host-run-netns\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:52:04.214716 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.214580 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-node-log\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:52:04.214716 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.214596 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-etc-openvswitch\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:52:04.214716 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.214619 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/df76407a-ce41-4a4a-b51e-8420b817f97c-etc-kubernetes\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:52:04.214716 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.214627 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-host-var-lib-cni-multus\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:52:04.214716 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.214631 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-host-run-netns\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:52:04.214716 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.214631 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4480cca6-b2e9-4840-ac8c-a88a40ed90f2-root\") pod \"node-exporter-2q7kh\" (UID: \"4480cca6-b2e9-4840-ac8c-a88a40ed90f2\") " pod="openshift-monitoring/node-exporter-2q7kh" Nov 03 01:52:04.214716 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.214650 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-host-run-netns\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:52:04.214716 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.214659 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-node-log\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:52:04.214716 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.214671 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-etc-openvswitch\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:52:04.214716 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.214677 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/df76407a-ce41-4a4a-b51e-8420b817f97c-etc-kubernetes\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:52:04.214716 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.214685 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/df76407a-ce41-4a4a-b51e-8420b817f97c-etc-sysctl-conf\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:52:04.214716 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.214714 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/df76407a-ce41-4a4a-b51e-8420b817f97c-sys\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:52:04.215165 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.214735 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-systemd-units\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:52:04.215165 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.214753 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9152cf36-4997-470e-8902-802a96ebc2c7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gwlr2\" (UID: \"9152cf36-4997-470e-8902-802a96ebc2c7\") " pod="openshift-multus/multus-additional-cni-plugins-gwlr2" Nov 03 01:52:04.215165 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.214769 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0801f13c-d14a-4b9e-b45a-a51af48d17c3-registration-dir\") pod \"aws-ebs-csi-driver-node-xdpjb\" (UID: \"0801f13c-d14a-4b9e-b45a-a51af48d17c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdpjb" Nov 03 01:52:04.215165 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.214770 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-systemd-units\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:52:04.215165 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.214775 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/df76407a-ce41-4a4a-b51e-8420b817f97c-sys\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:52:04.215165 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.214789 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-log-socket\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:52:04.215165 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.214810 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-multus-conf-dir\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:52:04.215165 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.214813 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/df76407a-ce41-4a4a-b51e-8420b817f97c-etc-sysctl-conf\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:52:04.215165 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.214819 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0801f13c-d14a-4b9e-b45a-a51af48d17c3-registration-dir\") pod \"aws-ebs-csi-driver-node-xdpjb\" (UID: \"0801f13c-d14a-4b9e-b45a-a51af48d17c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdpjb" Nov 03 01:52:04.215165 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.214818 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9152cf36-4997-470e-8902-802a96ebc2c7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gwlr2\" (UID: \"9152cf36-4997-470e-8902-802a96ebc2c7\") " pod="openshift-multus/multus-additional-cni-plugins-gwlr2" Nov 03 01:52:04.215165 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.214827 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/592adf6b-1edd-4f5e-88a2-6ed92895f22a-host\") pod \"node-ca-gjs9f\" (UID: \"592adf6b-1edd-4f5e-88a2-6ed92895f22a\") " pod="openshift-image-registry/node-ca-gjs9f" Nov 03 01:52:04.215165 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.214853 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/592adf6b-1edd-4f5e-88a2-6ed92895f22a-host\") pod \"node-ca-gjs9f\" (UID: \"592adf6b-1edd-4f5e-88a2-6ed92895f22a\") " pod="openshift-image-registry/node-ca-gjs9f" Nov 03 01:52:04.215165 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.214855 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-multus-conf-dir\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:52:04.215165 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.214852 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0801f13c-d14a-4b9e-b45a-a51af48d17c3-device-dir\") pod \"aws-ebs-csi-driver-node-xdpjb\" (UID: \"0801f13c-d14a-4b9e-b45a-a51af48d17c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdpjb" Nov 03 01:52:04.215165 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.214850 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-log-socket\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:52:04.215165 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.214889 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0801f13c-d14a-4b9e-b45a-a51af48d17c3-device-dir\") pod \"aws-ebs-csi-driver-node-xdpjb\" (UID: \"0801f13c-d14a-4b9e-b45a-a51af48d17c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdpjb" Nov 03 01:52:04.215165 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.214891 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0801f13c-d14a-4b9e-b45a-a51af48d17c3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xdpjb\" (UID: \"0801f13c-d14a-4b9e-b45a-a51af48d17c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdpjb" Nov 03 01:52:04.215165 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.214917 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0801f13c-d14a-4b9e-b45a-a51af48d17c3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xdpjb\" (UID: \"0801f13c-d14a-4b9e-b45a-a51af48d17c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdpjb" Nov 03 01:52:04.215165 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.214917 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-host-kubelet\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:52:04.215165 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.214942 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-host-kubelet\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:52:04.215165 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.214979 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0801f13c-d14a-4b9e-b45a-a51af48d17c3-sys-fs\") pod \"aws-ebs-csi-driver-node-xdpjb\" (UID: \"0801f13c-d14a-4b9e-b45a-a51af48d17c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdpjb" Nov 03 01:52:04.215165 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.214941 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0801f13c-d14a-4b9e-b45a-a51af48d17c3-sys-fs\") pod \"aws-ebs-csi-driver-node-xdpjb\" (UID: \"0801f13c-d14a-4b9e-b45a-a51af48d17c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdpjb" Nov 03 01:52:04.215165 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.215004 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-host-run-multus-certs\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:52:04.215165 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.215019 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0801f13c-d14a-4b9e-b45a-a51af48d17c3-etc-selinux\") pod \"aws-ebs-csi-driver-node-xdpjb\" (UID: \"0801f13c-d14a-4b9e-b45a-a51af48d17c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdpjb" Nov 03 01:52:04.215165 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.215032 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-multus-cni-dir\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:52:04.215165 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.215048 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/85148983-aa3f-45c2-a062-da2748dd9011-host-slash\") pod \"iptables-alerter-vmjc5\" (UID: \"85148983-aa3f-45c2-a062-da2748dd9011\") " pod="openshift-network-operator/iptables-alerter-vmjc5" Nov 03 01:52:04.215165 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.215063 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/28f00eb6-78c2-4d93-bde5-c1ea12601dbd-dbus\") pod \"global-pull-secret-syncer-kkngd\" (UID: \"28f00eb6-78c2-4d93-bde5-c1ea12601dbd\") " pod="kube-system/global-pull-secret-syncer-kkngd" Nov 03 01:52:04.215165 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.215080 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/df76407a-ce41-4a4a-b51e-8420b817f97c-etc-sysconfig\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:52:04.215165 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.215097 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-host-run-multus-certs\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:52:04.215165 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.215122 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/85148983-aa3f-45c2-a062-da2748dd9011-host-slash\") pod \"iptables-alerter-vmjc5\" (UID: \"85148983-aa3f-45c2-a062-da2748dd9011\") " pod="openshift-network-operator/iptables-alerter-vmjc5" Nov 03 01:52:04.215165 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.215135 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/df76407a-ce41-4a4a-b51e-8420b817f97c-etc-sysconfig\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:52:04.216399 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.215093 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0801f13c-d14a-4b9e-b45a-a51af48d17c3-etc-selinux\") pod \"aws-ebs-csi-driver-node-xdpjb\" (UID: \"0801f13c-d14a-4b9e-b45a-a51af48d17c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdpjb" Nov 03 01:52:04.216399 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.215473 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/28f00eb6-78c2-4d93-bde5-c1ea12601dbd-dbus\") pod \"global-pull-secret-syncer-kkngd\" (UID: \"28f00eb6-78c2-4d93-bde5-c1ea12601dbd\") " pod="kube-system/global-pull-secret-syncer-kkngd" Nov 03 01:52:04.216399 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.215486 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9152cf36-4997-470e-8902-802a96ebc2c7-cnibin\") pod \"multus-additional-cni-plugins-gwlr2\" (UID: \"9152cf36-4997-470e-8902-802a96ebc2c7\") " pod="openshift-multus/multus-additional-cni-plugins-gwlr2" Nov 03 01:52:04.216399 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.215531 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-multus-cni-dir\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:52:04.216399 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.215536 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-run-ovn\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:52:04.216399 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.215128 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9152cf36-4997-470e-8902-802a96ebc2c7-cnibin\") pod \"multus-additional-cni-plugins-gwlr2\" (UID: \"9152cf36-4997-470e-8902-802a96ebc2c7\") " pod="openshift-multus/multus-additional-cni-plugins-gwlr2" Nov 03 01:52:04.216399 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.215565 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-run-ovn\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:52:04.216399 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.215677 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-run-systemd\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:52:04.216399 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.215727 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-run-systemd\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:52:04.216399 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.215777 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0801f13c-d14a-4b9e-b45a-a51af48d17c3-socket-dir\") pod \"aws-ebs-csi-driver-node-xdpjb\" (UID: \"0801f13c-d14a-4b9e-b45a-a51af48d17c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdpjb" Nov 03 01:52:04.216399 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.215810 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-host-run-k8s-cni-cncf-io\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:52:04.216399 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.215839 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-cnibin\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:52:04.216399 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.215929 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0801f13c-d14a-4b9e-b45a-a51af48d17c3-socket-dir\") pod \"aws-ebs-csi-driver-node-xdpjb\" (UID: \"0801f13c-d14a-4b9e-b45a-a51af48d17c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdpjb" Nov 03 01:52:04.216399 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.215982 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-host-var-lib-cni-bin\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:52:04.216399 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.216040 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-host-cni-netd\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:52:04.216399 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.216088 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/df76407a-ce41-4a4a-b51e-8420b817f97c-host\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:52:04.216399 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.216121 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-host-run-ovn-kubernetes\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:52:04.216399 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.216168 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/df76407a-ce41-4a4a-b51e-8420b817f97c-etc-sysctl-d\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:52:04.216399 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.216203 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/df76407a-ce41-4a4a-b51e-8420b817f97c-var-lib-kubelet\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:52:04.216399 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.216251 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9152cf36-4997-470e-8902-802a96ebc2c7-system-cni-dir\") pod \"multus-additional-cni-plugins-gwlr2\" (UID: \"9152cf36-4997-470e-8902-802a96ebc2c7\") " pod="openshift-multus/multus-additional-cni-plugins-gwlr2" Nov 03 01:52:04.216399 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.216285 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:52:04.216399 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.216327 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4480cca6-b2e9-4840-ac8c-a88a40ed90f2-node-exporter-wtmp\") pod \"node-exporter-2q7kh\" (UID: \"4480cca6-b2e9-4840-ac8c-a88a40ed90f2\") " pod="openshift-monitoring/node-exporter-2q7kh" Nov 03 01:52:04.216399 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.216388 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/28f00eb6-78c2-4d93-bde5-c1ea12601dbd-kubelet-config\") pod \"global-pull-secret-syncer-kkngd\" (UID: \"28f00eb6-78c2-4d93-bde5-c1ea12601dbd\") " pod="kube-system/global-pull-secret-syncer-kkngd" Nov 03 01:52:04.217435 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.216429 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-var-lib-openvswitch\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:52:04.217435 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.216475 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-run-openvswitch\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:52:04.217435 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.216512 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/df76407a-ce41-4a4a-b51e-8420b817f97c-run\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:52:04.217435 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.216554 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/df76407a-ce41-4a4a-b51e-8420b817f97c-lib-modules\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:52:04.217435 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.216587 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7c3f175d-932c-42be-9660-e4fd3dceaf49-hosts-file\") pod \"node-resolver-7zdvv\" (UID: \"7c3f175d-932c-42be-9660-e4fd3dceaf49\") " pod="openshift-dns/node-resolver-7zdvv" Nov 03 01:52:04.217435 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.216638 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-etc-kubernetes\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:52:04.217435 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.216690 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3bc52103-b80b-49a8-9bed-15b196fdef12-crio-socket\") pod \"insights-runtime-extractor-dmrb8\" (UID: \"3bc52103-b80b-49a8-9bed-15b196fdef12\") " pod="openshift-insights/insights-runtime-extractor-dmrb8" Nov 03 01:52:04.217435 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.216719 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4480cca6-b2e9-4840-ac8c-a88a40ed90f2-sys\") pod \"node-exporter-2q7kh\" (UID: \"4480cca6-b2e9-4840-ac8c-a88a40ed90f2\") " pod="openshift-monitoring/node-exporter-2q7kh" Nov 03 01:52:04.217435 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.216766 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-host-cni-bin\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:52:04.217435 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.216802 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-multus-socket-dir-parent\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:52:04.217435 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.216848 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-host-var-lib-kubelet\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:52:04.217435 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.216880 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-hostroot\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:52:04.217435 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.216961 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-host-slash\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:52:04.217435 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.217017 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/df76407a-ce41-4a4a-b51e-8420b817f97c-etc-modprobe-d\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:52:04.217435 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.217050 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/df76407a-ce41-4a4a-b51e-8420b817f97c-etc-systemd\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:52:04.217435 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.217106 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-system-cni-dir\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:52:04.217435 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.217262 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-system-cni-dir\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:52:04.217435 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.217343 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-cnibin\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:52:04.217435 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.217419 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-host-run-k8s-cni-cncf-io\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:52:04.218269 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.217487 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-host-var-lib-cni-bin\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:52:04.218269 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.217556 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-host-cni-netd\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:52:04.218269 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.217631 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/df76407a-ce41-4a4a-b51e-8420b817f97c-host\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:52:04.218269 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.217697 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-host-run-ovn-kubernetes\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:52:04.218269 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.217865 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/df76407a-ce41-4a4a-b51e-8420b817f97c-etc-sysctl-d\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:52:04.218269 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.217941 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/df76407a-ce41-4a4a-b51e-8420b817f97c-var-lib-kubelet\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:52:04.218269 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.218009 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9152cf36-4997-470e-8902-802a96ebc2c7-system-cni-dir\") pod \"multus-additional-cni-plugins-gwlr2\" (UID: \"9152cf36-4997-470e-8902-802a96ebc2c7\") " pod="openshift-multus/multus-additional-cni-plugins-gwlr2" Nov 03 01:52:04.218269 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.218076 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:52:04.218269 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.218155 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4480cca6-b2e9-4840-ac8c-a88a40ed90f2-node-exporter-wtmp\") pod \"node-exporter-2q7kh\" (UID: \"4480cca6-b2e9-4840-ac8c-a88a40ed90f2\") " pod="openshift-monitoring/node-exporter-2q7kh" Nov 03 01:52:04.218269 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.218232 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/28f00eb6-78c2-4d93-bde5-c1ea12601dbd-kubelet-config\") pod \"global-pull-secret-syncer-kkngd\" (UID: \"28f00eb6-78c2-4d93-bde5-c1ea12601dbd\") " pod="kube-system/global-pull-secret-syncer-kkngd" Nov 03 01:52:04.218771 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.218280 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-var-lib-openvswitch\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:52:04.218771 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.218341 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-run-openvswitch\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:52:04.218771 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.218412 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/df76407a-ce41-4a4a-b51e-8420b817f97c-run\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:52:04.218771 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.218567 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/df76407a-ce41-4a4a-b51e-8420b817f97c-lib-modules\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:52:04.218771 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.218641 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7c3f175d-932c-42be-9660-e4fd3dceaf49-hosts-file\") pod \"node-resolver-7zdvv\" (UID: \"7c3f175d-932c-42be-9660-e4fd3dceaf49\") " pod="openshift-dns/node-resolver-7zdvv" Nov 03 01:52:04.218771 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.218704 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-etc-kubernetes\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:52:04.219055 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.218774 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3bc52103-b80b-49a8-9bed-15b196fdef12-crio-socket\") pod \"insights-runtime-extractor-dmrb8\" (UID: \"3bc52103-b80b-49a8-9bed-15b196fdef12\") " pod="openshift-insights/insights-runtime-extractor-dmrb8" Nov 03 01:52:04.219055 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.218822 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4480cca6-b2e9-4840-ac8c-a88a40ed90f2-sys\") pod \"node-exporter-2q7kh\" (UID: \"4480cca6-b2e9-4840-ac8c-a88a40ed90f2\") " pod="openshift-monitoring/node-exporter-2q7kh" Nov 03 01:52:04.219055 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.218894 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-host-cni-bin\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:52:04.219055 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.218972 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-multus-socket-dir-parent\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:52:04.219055 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.219038 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-host-var-lib-kubelet\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:52:04.219221 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.219073 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/56117bce-d686-4c24-98c8-c0ff122778ac-host-slash\") pod \"ovnkube-node-x452r\" (UID: \"56117bce-d686-4c24-98c8-c0ff122778ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:52:04.219221 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.219119 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/987bbde2-6a6a-44fc-b5e4-b0197bcecd8b-hostroot\") pod \"multus-trr75\" (UID: \"987bbde2-6a6a-44fc-b5e4-b0197bcecd8b\") " pod="openshift-multus/multus-trr75" Nov 03 01:52:04.219221 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.219133 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/df76407a-ce41-4a4a-b51e-8420b817f97c-etc-systemd\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:52:04.219221 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.219191 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/df76407a-ce41-4a4a-b51e-8420b817f97c-etc-modprobe-d\") pod \"tuned-qm8zc\" (UID: \"df76407a-ce41-4a4a-b51e-8420b817f97c\") " pod="openshift-cluster-node-tuning-operator/tuned-qm8zc" Nov 03 01:52:04.577636 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:04.577603 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59fbbdf587-6nm8q"] Nov 03 01:52:05.046914 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:05.046767 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-t2dwr" Nov 03 01:52:05.436065 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:05.436033 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-f5qvs" Nov 03 01:52:05.685576 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:05.685550 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:52:06.121259 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:06.121229 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-f524q" Nov 03 01:52:06.546603 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:06.546511 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-b984f6b6b-g6f2h" Nov 03 01:52:07.598729 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:07.598701 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-59fbbdf587-6nm8q" Nov 03 01:52:07.616885 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:07.616858 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:52:07.617067 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:07.617056 5605 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 01:52:08.324986 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:08.324946 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-69c467f7c8-cpfwh" Nov 03 01:52:08.329373 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:08.329338 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-69c467f7c8-cpfwh" Nov 03 01:52:08.440807 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:08.440777 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-b984f6b6b-g6f2h" Nov 03 01:52:12.276100 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:12.276011 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-69c467f7c8-cpfwh" Nov 03 01:52:17.198735 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:17.198704 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7489f79677-c9g7j"] Nov 03 01:52:17.203291 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:17.203270 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7489f79677-c9g7j" Nov 03 01:52:17.297211 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:17.297180 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7489f79677-c9g7j"] Nov 03 01:52:17.307701 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:17.307666 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cb189271-e46e-4777-aa9e-be0b00bc7e10-console-config\") pod \"console-7489f79677-c9g7j\" (UID: \"cb189271-e46e-4777-aa9e-be0b00bc7e10\") " pod="openshift-console/console-7489f79677-c9g7j" Nov 03 01:52:17.307846 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:17.307708 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cb189271-e46e-4777-aa9e-be0b00bc7e10-tmp\") pod \"console-7489f79677-c9g7j\" (UID: \"cb189271-e46e-4777-aa9e-be0b00bc7e10\") " pod="openshift-console/console-7489f79677-c9g7j" Nov 03 01:52:17.307846 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:17.307833 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cb189271-e46e-4777-aa9e-be0b00bc7e10-service-ca\") pod \"console-7489f79677-c9g7j\" (UID: \"cb189271-e46e-4777-aa9e-be0b00bc7e10\") " pod="openshift-console/console-7489f79677-c9g7j" Nov 03 01:52:17.307956 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:17.307886 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cb189271-e46e-4777-aa9e-be0b00bc7e10-console-oauth-config\") pod \"console-7489f79677-c9g7j\" (UID: \"cb189271-e46e-4777-aa9e-be0b00bc7e10\") " pod="openshift-console/console-7489f79677-c9g7j" Nov 03 01:52:17.308005 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:17.307984 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cb189271-e46e-4777-aa9e-be0b00bc7e10-oauth-serving-cert\") pod \"console-7489f79677-c9g7j\" (UID: \"cb189271-e46e-4777-aa9e-be0b00bc7e10\") " pod="openshift-console/console-7489f79677-c9g7j" Nov 03 01:52:17.308041 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:17.308027 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cb189271-e46e-4777-aa9e-be0b00bc7e10-console-serving-cert\") pod \"console-7489f79677-c9g7j\" (UID: \"cb189271-e46e-4777-aa9e-be0b00bc7e10\") " pod="openshift-console/console-7489f79677-c9g7j" Nov 03 01:52:17.308075 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:17.308049 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb189271-e46e-4777-aa9e-be0b00bc7e10-trusted-ca-bundle\") pod \"console-7489f79677-c9g7j\" (UID: \"cb189271-e46e-4777-aa9e-be0b00bc7e10\") " pod="openshift-console/console-7489f79677-c9g7j" Nov 03 01:52:17.308110 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:17.308075 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5khc\" (UniqueName: \"kubernetes.io/projected/cb189271-e46e-4777-aa9e-be0b00bc7e10-kube-api-access-k5khc\") pod \"console-7489f79677-c9g7j\" (UID: \"cb189271-e46e-4777-aa9e-be0b00bc7e10\") " pod="openshift-console/console-7489f79677-c9g7j" Nov 03 01:52:17.409200 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:17.409171 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cb189271-e46e-4777-aa9e-be0b00bc7e10-service-ca\") pod \"console-7489f79677-c9g7j\" (UID: \"cb189271-e46e-4777-aa9e-be0b00bc7e10\") " pod="openshift-console/console-7489f79677-c9g7j" Nov 03 01:52:17.409402 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:17.409208 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cb189271-e46e-4777-aa9e-be0b00bc7e10-console-oauth-config\") pod \"console-7489f79677-c9g7j\" (UID: \"cb189271-e46e-4777-aa9e-be0b00bc7e10\") " pod="openshift-console/console-7489f79677-c9g7j" Nov 03 01:52:17.409402 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:17.409239 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cb189271-e46e-4777-aa9e-be0b00bc7e10-oauth-serving-cert\") pod \"console-7489f79677-c9g7j\" (UID: \"cb189271-e46e-4777-aa9e-be0b00bc7e10\") " pod="openshift-console/console-7489f79677-c9g7j" Nov 03 01:52:17.409402 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:17.409261 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cb189271-e46e-4777-aa9e-be0b00bc7e10-console-serving-cert\") pod \"console-7489f79677-c9g7j\" (UID: \"cb189271-e46e-4777-aa9e-be0b00bc7e10\") " pod="openshift-console/console-7489f79677-c9g7j" Nov 03 01:52:17.409402 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:17.409309 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb189271-e46e-4777-aa9e-be0b00bc7e10-trusted-ca-bundle\") pod \"console-7489f79677-c9g7j\" (UID: \"cb189271-e46e-4777-aa9e-be0b00bc7e10\") " pod="openshift-console/console-7489f79677-c9g7j" Nov 03 01:52:17.409402 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:17.409338 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5khc\" (UniqueName: \"kubernetes.io/projected/cb189271-e46e-4777-aa9e-be0b00bc7e10-kube-api-access-k5khc\") pod \"console-7489f79677-c9g7j\" (UID: \"cb189271-e46e-4777-aa9e-be0b00bc7e10\") " pod="openshift-console/console-7489f79677-c9g7j" Nov 03 01:52:17.409402 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:17.409395 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cb189271-e46e-4777-aa9e-be0b00bc7e10-console-config\") pod \"console-7489f79677-c9g7j\" (UID: \"cb189271-e46e-4777-aa9e-be0b00bc7e10\") " pod="openshift-console/console-7489f79677-c9g7j" Nov 03 01:52:17.409699 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:17.409419 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cb189271-e46e-4777-aa9e-be0b00bc7e10-tmp\") pod \"console-7489f79677-c9g7j\" (UID: \"cb189271-e46e-4777-aa9e-be0b00bc7e10\") " pod="openshift-console/console-7489f79677-c9g7j" Nov 03 01:52:17.409749 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:17.409689 5605 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 01:52:17.409836 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:17.409813 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cb189271-e46e-4777-aa9e-be0b00bc7e10-tmp\") pod \"console-7489f79677-c9g7j\" (UID: \"cb189271-e46e-4777-aa9e-be0b00bc7e10\") " pod="openshift-console/console-7489f79677-c9g7j" Nov 03 01:52:17.410105 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:17.410078 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cb189271-e46e-4777-aa9e-be0b00bc7e10-service-ca\") pod \"console-7489f79677-c9g7j\" (UID: \"cb189271-e46e-4777-aa9e-be0b00bc7e10\") " pod="openshift-console/console-7489f79677-c9g7j" Nov 03 01:52:17.410105 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:17.410104 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cb189271-e46e-4777-aa9e-be0b00bc7e10-oauth-serving-cert\") pod \"console-7489f79677-c9g7j\" (UID: \"cb189271-e46e-4777-aa9e-be0b00bc7e10\") " pod="openshift-console/console-7489f79677-c9g7j" Nov 03 01:52:17.410427 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:17.410398 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cb189271-e46e-4777-aa9e-be0b00bc7e10-console-config\") pod \"console-7489f79677-c9g7j\" (UID: \"cb189271-e46e-4777-aa9e-be0b00bc7e10\") " pod="openshift-console/console-7489f79677-c9g7j" Nov 03 01:52:17.410427 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:17.410416 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb189271-e46e-4777-aa9e-be0b00bc7e10-trusted-ca-bundle\") pod \"console-7489f79677-c9g7j\" (UID: \"cb189271-e46e-4777-aa9e-be0b00bc7e10\") " pod="openshift-console/console-7489f79677-c9g7j" Nov 03 01:52:17.412039 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:17.412020 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cb189271-e46e-4777-aa9e-be0b00bc7e10-console-oauth-config\") pod \"console-7489f79677-c9g7j\" (UID: \"cb189271-e46e-4777-aa9e-be0b00bc7e10\") " pod="openshift-console/console-7489f79677-c9g7j" Nov 03 01:52:17.412134 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:17.412018 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cb189271-e46e-4777-aa9e-be0b00bc7e10-console-serving-cert\") pod \"console-7489f79677-c9g7j\" (UID: \"cb189271-e46e-4777-aa9e-be0b00bc7e10\") " pod="openshift-console/console-7489f79677-c9g7j" Nov 03 01:52:17.462761 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:17.462683 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5khc\" (UniqueName: \"kubernetes.io/projected/cb189271-e46e-4777-aa9e-be0b00bc7e10-kube-api-access-k5khc\") pod \"console-7489f79677-c9g7j\" (UID: \"cb189271-e46e-4777-aa9e-be0b00bc7e10\") " pod="openshift-console/console-7489f79677-c9g7j" Nov 03 01:52:17.514339 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:17.514302 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7489f79677-c9g7j" Nov 03 01:52:17.878803 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:17.878775 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7489f79677-c9g7j"] Nov 03 01:52:17.885337 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:17.885300 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb189271_e46e_4777_aa9e_be0b00bc7e10.slice/crio-6034fdc764814f434ad85f52dd8fc3b32fec9ac7f3df78c77f651b05f086b762 WatchSource:0}: Error finding container 6034fdc764814f434ad85f52dd8fc3b32fec9ac7f3df78c77f651b05f086b762: Status 404 returned error can't find the container with id 6034fdc764814f434ad85f52dd8fc3b32fec9ac7f3df78c77f651b05f086b762 Nov 03 01:52:17.967199 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:17.967163 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7489f79677-c9g7j" event={"ID":"cb189271-e46e-4777-aa9e-be0b00bc7e10","Type":"ContainerStarted","Data":"e6ff65955297cc6f59f96c3d8b7e1ffb2ba4888034770d11a792e5a8cad8bdc9"} Nov 03 01:52:17.967372 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:17.967220 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7489f79677-c9g7j" event={"ID":"cb189271-e46e-4777-aa9e-be0b00bc7e10","Type":"ContainerStarted","Data":"6034fdc764814f434ad85f52dd8fc3b32fec9ac7f3df78c77f651b05f086b762"} Nov 03 01:52:18.074404 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:18.074330 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7489f79677-c9g7j" podStartSLOduration=1.074314014 podStartE2EDuration="1.074314014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:52:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:52:18.073531351 +0000 UTC m=+15.332729505" watchObservedRunningTime="2026-11-03 01:52:18.074314014 +0000 UTC m=+15.333512166" Nov 03 01:52:22.237496 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:22.237458 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 01:52:22.238188 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:22.238125 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3d7d07f7-27c2-489a-8e44-fe366584a267" containerName="alertmanager" containerID="cri-o://5a89dbaf332edc96b69633d170776b866432066421f0079dc19cbeb24cc16b63" gracePeriod=120 Nov 03 01:52:22.238301 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:22.238200 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3d7d07f7-27c2-489a-8e44-fe366584a267" containerName="kube-rbac-proxy-metric" containerID="cri-o://2a036232c737f1345f0379e4433a70dbbce6f1a0b055c8986f64f79b481c82a3" gracePeriod=120 Nov 03 01:52:22.238301 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:22.238225 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3d7d07f7-27c2-489a-8e44-fe366584a267" containerName="kube-rbac-proxy-web" containerID="cri-o://9ba45f94f3b5795d52d3c4aa689200dd500416f2f0ec4785d2d22b03676b8abb" gracePeriod=120 Nov 03 01:52:22.238301 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:22.238244 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3d7d07f7-27c2-489a-8e44-fe366584a267" containerName="prom-label-proxy" containerID="cri-o://45237327ba6c01b1abbd14db2279979dfafd6ffd6fe0e98eddd6065d580bd86b" gracePeriod=120 Nov 03 01:52:22.238301 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:22.238264 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3d7d07f7-27c2-489a-8e44-fe366584a267" containerName="config-reloader" containerID="cri-o://9ba1dd1268281f89787a0064d6dd633f1a0d1c321c5397430616868590b663c5" gracePeriod=120 Nov 03 01:52:22.238507 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:22.238265 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3d7d07f7-27c2-489a-8e44-fe366584a267" containerName="kube-rbac-proxy" containerID="cri-o://087add6fe689b6c7740d6995d0dcbb3dd5a93ce47965875f6a4a96e7b644cfd7" gracePeriod=120 Nov 03 01:52:22.281401 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:22.281375 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-69c467f7c8-cpfwh" Nov 03 01:52:22.988932 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:22.988903 5605 generic.go:356] "Generic (PLEG): container finished" podID="3d7d07f7-27c2-489a-8e44-fe366584a267" containerID="45237327ba6c01b1abbd14db2279979dfafd6ffd6fe0e98eddd6065d580bd86b" exitCode=0 Nov 03 01:52:22.988932 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:22.988925 5605 generic.go:356] "Generic (PLEG): container finished" podID="3d7d07f7-27c2-489a-8e44-fe366584a267" containerID="2a036232c737f1345f0379e4433a70dbbce6f1a0b055c8986f64f79b481c82a3" exitCode=0 Nov 03 01:52:22.988932 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:22.988933 5605 generic.go:356] "Generic (PLEG): container finished" podID="3d7d07f7-27c2-489a-8e44-fe366584a267" containerID="087add6fe689b6c7740d6995d0dcbb3dd5a93ce47965875f6a4a96e7b644cfd7" exitCode=0 Nov 03 01:52:22.988932 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:22.988939 5605 generic.go:356] "Generic (PLEG): container finished" podID="3d7d07f7-27c2-489a-8e44-fe366584a267" containerID="9ba1dd1268281f89787a0064d6dd633f1a0d1c321c5397430616868590b663c5" exitCode=0 Nov 03 01:52:22.988932 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:22.988945 5605 generic.go:356] "Generic (PLEG): container finished" podID="3d7d07f7-27c2-489a-8e44-fe366584a267" containerID="5a89dbaf332edc96b69633d170776b866432066421f0079dc19cbeb24cc16b63" exitCode=0 Nov 03 01:52:22.989218 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:22.988973 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3d7d07f7-27c2-489a-8e44-fe366584a267","Type":"ContainerDied","Data":"45237327ba6c01b1abbd14db2279979dfafd6ffd6fe0e98eddd6065d580bd86b"} Nov 03 01:52:22.989218 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:22.989044 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3d7d07f7-27c2-489a-8e44-fe366584a267","Type":"ContainerDied","Data":"2a036232c737f1345f0379e4433a70dbbce6f1a0b055c8986f64f79b481c82a3"} Nov 03 01:52:22.989218 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:22.989071 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3d7d07f7-27c2-489a-8e44-fe366584a267","Type":"ContainerDied","Data":"087add6fe689b6c7740d6995d0dcbb3dd5a93ce47965875f6a4a96e7b644cfd7"} Nov 03 01:52:22.989218 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:22.989094 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3d7d07f7-27c2-489a-8e44-fe366584a267","Type":"ContainerDied","Data":"9ba1dd1268281f89787a0064d6dd633f1a0d1c321c5397430616868590b663c5"} Nov 03 01:52:22.989218 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:22.989119 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3d7d07f7-27c2-489a-8e44-fe366584a267","Type":"ContainerDied","Data":"5a89dbaf332edc96b69633d170776b866432066421f0079dc19cbeb24cc16b63"} Nov 03 01:52:23.487619 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:23.487593 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:23.566473 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:23.566447 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d7d07f7-27c2-489a-8e44-fe366584a267-tls-assets\" (UniqueName: \"kubernetes.io/projected/3d7d07f7-27c2-489a-8e44-fe366584a267-tls-assets\") pod \"3d7d07f7-27c2-489a-8e44-fe366584a267\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " Nov 03 01:52:23.566616 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:23.566514 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-secret-alertmanager-kube-rbac-proxy\") pod \"3d7d07f7-27c2-489a-8e44-fe366584a267\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " Nov 03 01:52:23.566616 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:23.566537 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d7d07f7-27c2-489a-8e44-fe366584a267-kube-api-access-nl9gt\" (UniqueName: \"kubernetes.io/projected/3d7d07f7-27c2-489a-8e44-fe366584a267-kube-api-access-nl9gt\") pod \"3d7d07f7-27c2-489a-8e44-fe366584a267\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " Nov 03 01:52:23.566616 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:23.566562 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d7d07f7-27c2-489a-8e44-fe366584a267-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3d7d07f7-27c2-489a-8e44-fe366584a267-alertmanager-main-db\") pod \"3d7d07f7-27c2-489a-8e44-fe366584a267\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " Nov 03 01:52:23.566616 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:23.566584 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-secret-alertmanager-main-tls\") pod \"3d7d07f7-27c2-489a-8e44-fe366584a267\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " Nov 03 01:52:23.566616 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:23.566607 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-web-config\" (UniqueName: \"kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-web-config\") pod \"3d7d07f7-27c2-489a-8e44-fe366584a267\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " Nov 03 01:52:23.566863 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:23.566650 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-secret-alertmanager-kube-rbac-proxy-web\") pod \"3d7d07f7-27c2-489a-8e44-fe366584a267\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " Nov 03 01:52:23.566863 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:23.566679 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-secret-alertmanager-kube-rbac-proxy-metric\") pod \"3d7d07f7-27c2-489a-8e44-fe366584a267\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " Nov 03 01:52:23.566863 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:23.566732 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d7d07f7-27c2-489a-8e44-fe366584a267-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d7d07f7-27c2-489a-8e44-fe366584a267-alertmanager-trusted-ca-bundle\") pod \"3d7d07f7-27c2-489a-8e44-fe366584a267\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " Nov 03 01:52:23.566863 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:23.566775 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-config-volume\" (UniqueName: \"kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-config-volume\") pod \"3d7d07f7-27c2-489a-8e44-fe366584a267\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " Nov 03 01:52:23.566863 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:23.566808 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d7d07f7-27c2-489a-8e44-fe366584a267-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d7d07f7-27c2-489a-8e44-fe366584a267-metrics-client-ca\") pod \"3d7d07f7-27c2-489a-8e44-fe366584a267\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " Nov 03 01:52:23.566863 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:23.566848 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d7d07f7-27c2-489a-8e44-fe366584a267-config-out\" (UniqueName: \"kubernetes.io/empty-dir/3d7d07f7-27c2-489a-8e44-fe366584a267-config-out\") pod \"3d7d07f7-27c2-489a-8e44-fe366584a267\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " Nov 03 01:52:23.567140 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:23.566873 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-cluster-tls-config\") pod \"3d7d07f7-27c2-489a-8e44-fe366584a267\" (UID: \"3d7d07f7-27c2-489a-8e44-fe366584a267\") " Nov 03 01:52:23.567140 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:23.566957 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d7d07f7-27c2-489a-8e44-fe366584a267-alertmanager-main-db" pod "3d7d07f7-27c2-489a-8e44-fe366584a267" (UID: "3d7d07f7-27c2-489a-8e44-fe366584a267"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:52:23.567237 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:23.567165 5605 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3d7d07f7-27c2-489a-8e44-fe366584a267-alertmanager-main-db\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:52:23.568301 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:23.568175 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d7d07f7-27c2-489a-8e44-fe366584a267-metrics-client-ca" pod "3d7d07f7-27c2-489a-8e44-fe366584a267" (UID: "3d7d07f7-27c2-489a-8e44-fe366584a267"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:52:23.568434 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:23.568401 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d7d07f7-27c2-489a-8e44-fe366584a267-alertmanager-trusted-ca-bundle" pod "3d7d07f7-27c2-489a-8e44-fe366584a267" (UID: "3d7d07f7-27c2-489a-8e44-fe366584a267"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:52:23.570271 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:23.570150 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d7d07f7-27c2-489a-8e44-fe366584a267-config-out" pod "3d7d07f7-27c2-489a-8e44-fe366584a267" (UID: "3d7d07f7-27c2-489a-8e44-fe366584a267"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:52:23.570271 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:23.570212 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-secret-alertmanager-kube-rbac-proxy" pod "3d7d07f7-27c2-489a-8e44-fe366584a267" (UID: "3d7d07f7-27c2-489a-8e44-fe366584a267"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:52:23.570444 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:23.570310 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d7d07f7-27c2-489a-8e44-fe366584a267-kube-api-access-nl9gt" pod "3d7d07f7-27c2-489a-8e44-fe366584a267" (UID: "3d7d07f7-27c2-489a-8e44-fe366584a267"). InnerVolumeSpecName "kube-api-access-nl9gt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:52:23.570444 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:23.570379 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-secret-alertmanager-kube-rbac-proxy-web" pod "3d7d07f7-27c2-489a-8e44-fe366584a267" (UID: "3d7d07f7-27c2-489a-8e44-fe366584a267"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:52:23.570564 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:23.570477 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d7d07f7-27c2-489a-8e44-fe366584a267-tls-assets" pod "3d7d07f7-27c2-489a-8e44-fe366584a267" (UID: "3d7d07f7-27c2-489a-8e44-fe366584a267"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:52:23.570691 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:23.570658 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-secret-alertmanager-kube-rbac-proxy-metric" pod "3d7d07f7-27c2-489a-8e44-fe366584a267" (UID: "3d7d07f7-27c2-489a-8e44-fe366584a267"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:52:23.571115 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:23.571094 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-secret-alertmanager-main-tls" pod "3d7d07f7-27c2-489a-8e44-fe366584a267" (UID: "3d7d07f7-27c2-489a-8e44-fe366584a267"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:52:23.571989 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:23.571968 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-config-volume" pod "3d7d07f7-27c2-489a-8e44-fe366584a267" (UID: "3d7d07f7-27c2-489a-8e44-fe366584a267"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:52:23.574603 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:23.574577 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-cluster-tls-config" pod "3d7d07f7-27c2-489a-8e44-fe366584a267" (UID: "3d7d07f7-27c2-489a-8e44-fe366584a267"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:52:23.579848 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:23.579826 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-web-config" pod "3d7d07f7-27c2-489a-8e44-fe366584a267" (UID: "3d7d07f7-27c2-489a-8e44-fe366584a267"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:52:23.668255 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:23.668226 5605 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d7d07f7-27c2-489a-8e44-fe366584a267-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:52:23.668255 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:23.668251 5605 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-config-volume\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:52:23.668255 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:23.668261 5605 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d7d07f7-27c2-489a-8e44-fe366584a267-metrics-client-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:52:23.668497 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:23.668270 5605 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3d7d07f7-27c2-489a-8e44-fe366584a267-config-out\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:52:23.668497 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:23.668279 5605 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-cluster-tls-config\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:52:23.668497 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:23.668287 5605 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3d7d07f7-27c2-489a-8e44-fe366584a267-tls-assets\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:52:23.668497 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:23.668296 5605 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:52:23.668497 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:23.668304 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nl9gt\" (UniqueName: \"kubernetes.io/projected/3d7d07f7-27c2-489a-8e44-fe366584a267-kube-api-access-nl9gt\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:52:23.668497 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:23.668313 5605 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-secret-alertmanager-main-tls\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:52:23.668497 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:23.668322 5605 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-web-config\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:52:23.668497 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:23.668330 5605 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:52:23.668497 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:23.668339 5605 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3d7d07f7-27c2-489a-8e44-fe366584a267-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:52:23.910278 ip-10-0-140-140 kubenswrapper[5605]: E1103 01:52:23.910201 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 01:52:23.995806 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:23.995766 5605 generic.go:356] "Generic (PLEG): container finished" podID="3d7d07f7-27c2-489a-8e44-fe366584a267" containerID="9ba45f94f3b5795d52d3c4aa689200dd500416f2f0ec4785d2d22b03676b8abb" exitCode=0 Nov 03 01:52:23.995960 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:23.995827 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3d7d07f7-27c2-489a-8e44-fe366584a267","Type":"ContainerDied","Data":"9ba45f94f3b5795d52d3c4aa689200dd500416f2f0ec4785d2d22b03676b8abb"} Nov 03 01:52:23.995960 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:23.995862 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:23.995960 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:23.995897 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3d7d07f7-27c2-489a-8e44-fe366584a267","Type":"ContainerDied","Data":"e6b490f1544909557410be518cb9d934c6f2996cede5301488d9934ab887b297"} Nov 03 01:52:23.995960 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:23.995931 5605 scope.go:122] "RemoveContainer" containerID="45237327ba6c01b1abbd14db2279979dfafd6ffd6fe0e98eddd6065d580bd86b" Nov 03 01:52:24.004176 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.004155 5605 scope.go:122] "RemoveContainer" containerID="2a036232c737f1345f0379e4433a70dbbce6f1a0b055c8986f64f79b481c82a3" Nov 03 01:52:24.010978 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.010962 5605 scope.go:122] "RemoveContainer" containerID="087add6fe689b6c7740d6995d0dcbb3dd5a93ce47965875f6a4a96e7b644cfd7" Nov 03 01:52:24.017511 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.017488 5605 scope.go:122] "RemoveContainer" containerID="9ba45f94f3b5795d52d3c4aa689200dd500416f2f0ec4785d2d22b03676b8abb" Nov 03 01:52:24.023804 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.023781 5605 scope.go:122] "RemoveContainer" containerID="9ba1dd1268281f89787a0064d6dd633f1a0d1c321c5397430616868590b663c5" Nov 03 01:52:24.030465 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.030447 5605 scope.go:122] "RemoveContainer" containerID="5a89dbaf332edc96b69633d170776b866432066421f0079dc19cbeb24cc16b63" Nov 03 01:52:24.036920 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.036903 5605 scope.go:122] "RemoveContainer" containerID="0d9d5fbaa013bb1c1bc86e40f772c92f6173c111758e1557c9ffb7578af08cf3" Nov 03 01:52:24.043157 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.043140 5605 scope.go:122] "RemoveContainer" containerID="45237327ba6c01b1abbd14db2279979dfafd6ffd6fe0e98eddd6065d580bd86b" Nov 03 01:52:24.043429 ip-10-0-140-140 kubenswrapper[5605]: E1103 01:52:24.043406 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45237327ba6c01b1abbd14db2279979dfafd6ffd6fe0e98eddd6065d580bd86b\": container with ID starting with 45237327ba6c01b1abbd14db2279979dfafd6ffd6fe0e98eddd6065d580bd86b not found: ID does not exist" containerID="45237327ba6c01b1abbd14db2279979dfafd6ffd6fe0e98eddd6065d580bd86b" Nov 03 01:52:24.043475 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.043441 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45237327ba6c01b1abbd14db2279979dfafd6ffd6fe0e98eddd6065d580bd86b"} err="failed to get container status \"45237327ba6c01b1abbd14db2279979dfafd6ffd6fe0e98eddd6065d580bd86b\": rpc error: code = NotFound desc = could not find container \"45237327ba6c01b1abbd14db2279979dfafd6ffd6fe0e98eddd6065d580bd86b\": container with ID starting with 45237327ba6c01b1abbd14db2279979dfafd6ffd6fe0e98eddd6065d580bd86b not found: ID does not exist" Nov 03 01:52:24.043522 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.043477 5605 scope.go:122] "RemoveContainer" containerID="2a036232c737f1345f0379e4433a70dbbce6f1a0b055c8986f64f79b481c82a3" Nov 03 01:52:24.043718 ip-10-0-140-140 kubenswrapper[5605]: E1103 01:52:24.043700 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a036232c737f1345f0379e4433a70dbbce6f1a0b055c8986f64f79b481c82a3\": container with ID starting with 2a036232c737f1345f0379e4433a70dbbce6f1a0b055c8986f64f79b481c82a3 not found: ID does not exist" containerID="2a036232c737f1345f0379e4433a70dbbce6f1a0b055c8986f64f79b481c82a3" Nov 03 01:52:24.043765 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.043724 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a036232c737f1345f0379e4433a70dbbce6f1a0b055c8986f64f79b481c82a3"} err="failed to get container status \"2a036232c737f1345f0379e4433a70dbbce6f1a0b055c8986f64f79b481c82a3\": rpc error: code = NotFound desc = could not find container \"2a036232c737f1345f0379e4433a70dbbce6f1a0b055c8986f64f79b481c82a3\": container with ID starting with 2a036232c737f1345f0379e4433a70dbbce6f1a0b055c8986f64f79b481c82a3 not found: ID does not exist" Nov 03 01:52:24.043765 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.043740 5605 scope.go:122] "RemoveContainer" containerID="087add6fe689b6c7740d6995d0dcbb3dd5a93ce47965875f6a4a96e7b644cfd7" Nov 03 01:52:24.043933 ip-10-0-140-140 kubenswrapper[5605]: E1103 01:52:24.043918 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"087add6fe689b6c7740d6995d0dcbb3dd5a93ce47965875f6a4a96e7b644cfd7\": container with ID starting with 087add6fe689b6c7740d6995d0dcbb3dd5a93ce47965875f6a4a96e7b644cfd7 not found: ID does not exist" containerID="087add6fe689b6c7740d6995d0dcbb3dd5a93ce47965875f6a4a96e7b644cfd7" Nov 03 01:52:24.043979 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.043936 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"087add6fe689b6c7740d6995d0dcbb3dd5a93ce47965875f6a4a96e7b644cfd7"} err="failed to get container status \"087add6fe689b6c7740d6995d0dcbb3dd5a93ce47965875f6a4a96e7b644cfd7\": rpc error: code = NotFound desc = could not find container \"087add6fe689b6c7740d6995d0dcbb3dd5a93ce47965875f6a4a96e7b644cfd7\": container with ID starting with 087add6fe689b6c7740d6995d0dcbb3dd5a93ce47965875f6a4a96e7b644cfd7 not found: ID does not exist" Nov 03 01:52:24.043979 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.043949 5605 scope.go:122] "RemoveContainer" containerID="9ba45f94f3b5795d52d3c4aa689200dd500416f2f0ec4785d2d22b03676b8abb" Nov 03 01:52:24.044141 ip-10-0-140-140 kubenswrapper[5605]: E1103 01:52:24.044123 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ba45f94f3b5795d52d3c4aa689200dd500416f2f0ec4785d2d22b03676b8abb\": container with ID starting with 9ba45f94f3b5795d52d3c4aa689200dd500416f2f0ec4785d2d22b03676b8abb not found: ID does not exist" containerID="9ba45f94f3b5795d52d3c4aa689200dd500416f2f0ec4785d2d22b03676b8abb" Nov 03 01:52:24.044205 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.044149 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ba45f94f3b5795d52d3c4aa689200dd500416f2f0ec4785d2d22b03676b8abb"} err="failed to get container status \"9ba45f94f3b5795d52d3c4aa689200dd500416f2f0ec4785d2d22b03676b8abb\": rpc error: code = NotFound desc = could not find container \"9ba45f94f3b5795d52d3c4aa689200dd500416f2f0ec4785d2d22b03676b8abb\": container with ID starting with 9ba45f94f3b5795d52d3c4aa689200dd500416f2f0ec4785d2d22b03676b8abb not found: ID does not exist" Nov 03 01:52:24.044205 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.044170 5605 scope.go:122] "RemoveContainer" containerID="9ba1dd1268281f89787a0064d6dd633f1a0d1c321c5397430616868590b663c5" Nov 03 01:52:24.044688 ip-10-0-140-140 kubenswrapper[5605]: E1103 01:52:24.044672 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ba1dd1268281f89787a0064d6dd633f1a0d1c321c5397430616868590b663c5\": container with ID starting with 9ba1dd1268281f89787a0064d6dd633f1a0d1c321c5397430616868590b663c5 not found: ID does not exist" containerID="9ba1dd1268281f89787a0064d6dd633f1a0d1c321c5397430616868590b663c5" Nov 03 01:52:24.044740 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.044690 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ba1dd1268281f89787a0064d6dd633f1a0d1c321c5397430616868590b663c5"} err="failed to get container status \"9ba1dd1268281f89787a0064d6dd633f1a0d1c321c5397430616868590b663c5\": rpc error: code = NotFound desc = could not find container \"9ba1dd1268281f89787a0064d6dd633f1a0d1c321c5397430616868590b663c5\": container with ID starting with 9ba1dd1268281f89787a0064d6dd633f1a0d1c321c5397430616868590b663c5 not found: ID does not exist" Nov 03 01:52:24.044740 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.044702 5605 scope.go:122] "RemoveContainer" containerID="5a89dbaf332edc96b69633d170776b866432066421f0079dc19cbeb24cc16b63" Nov 03 01:52:24.044939 ip-10-0-140-140 kubenswrapper[5605]: E1103 01:52:24.044921 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a89dbaf332edc96b69633d170776b866432066421f0079dc19cbeb24cc16b63\": container with ID starting with 5a89dbaf332edc96b69633d170776b866432066421f0079dc19cbeb24cc16b63 not found: ID does not exist" containerID="5a89dbaf332edc96b69633d170776b866432066421f0079dc19cbeb24cc16b63" Nov 03 01:52:24.044982 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.044943 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a89dbaf332edc96b69633d170776b866432066421f0079dc19cbeb24cc16b63"} err="failed to get container status \"5a89dbaf332edc96b69633d170776b866432066421f0079dc19cbeb24cc16b63\": rpc error: code = NotFound desc = could not find container \"5a89dbaf332edc96b69633d170776b866432066421f0079dc19cbeb24cc16b63\": container with ID starting with 5a89dbaf332edc96b69633d170776b866432066421f0079dc19cbeb24cc16b63 not found: ID does not exist" Nov 03 01:52:24.044982 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.044958 5605 scope.go:122] "RemoveContainer" containerID="0d9d5fbaa013bb1c1bc86e40f772c92f6173c111758e1557c9ffb7578af08cf3" Nov 03 01:52:24.045184 ip-10-0-140-140 kubenswrapper[5605]: E1103 01:52:24.045160 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d9d5fbaa013bb1c1bc86e40f772c92f6173c111758e1557c9ffb7578af08cf3\": container with ID starting with 0d9d5fbaa013bb1c1bc86e40f772c92f6173c111758e1557c9ffb7578af08cf3 not found: ID does not exist" containerID="0d9d5fbaa013bb1c1bc86e40f772c92f6173c111758e1557c9ffb7578af08cf3" Nov 03 01:52:24.045263 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.045188 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d9d5fbaa013bb1c1bc86e40f772c92f6173c111758e1557c9ffb7578af08cf3"} err="failed to get container status \"0d9d5fbaa013bb1c1bc86e40f772c92f6173c111758e1557c9ffb7578af08cf3\": rpc error: code = NotFound desc = could not find container \"0d9d5fbaa013bb1c1bc86e40f772c92f6173c111758e1557c9ffb7578af08cf3\": container with ID starting with 0d9d5fbaa013bb1c1bc86e40f772c92f6173c111758e1557c9ffb7578af08cf3 not found: ID does not exist" Nov 03 01:52:24.151445 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.151413 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 01:52:24.197932 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.197899 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 01:52:24.198625 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.198602 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 01:52:24.198858 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.198845 5605 cpu_manager.go:414] "containerMap: removing container" podUID="3d7d07f7-27c2-489a-8e44-fe366584a267" containerName="alertmanager" Nov 03 01:52:24.198896 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.198859 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d7d07f7-27c2-489a-8e44-fe366584a267" containerName="alertmanager" Nov 03 01:52:24.198896 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.198875 5605 cpu_manager.go:414] "containerMap: removing container" podUID="3d7d07f7-27c2-489a-8e44-fe366584a267" containerName="kube-rbac-proxy-metric" Nov 03 01:52:24.198896 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.198881 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d7d07f7-27c2-489a-8e44-fe366584a267" containerName="kube-rbac-proxy-metric" Nov 03 01:52:24.198896 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.198892 5605 cpu_manager.go:414] "containerMap: removing container" podUID="3d7d07f7-27c2-489a-8e44-fe366584a267" containerName="config-reloader" Nov 03 01:52:24.199011 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.198898 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d7d07f7-27c2-489a-8e44-fe366584a267" containerName="config-reloader" Nov 03 01:52:24.199011 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.198908 5605 cpu_manager.go:414] "containerMap: removing container" podUID="3d7d07f7-27c2-489a-8e44-fe366584a267" containerName="init-config-reloader" Nov 03 01:52:24.199011 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.198913 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d7d07f7-27c2-489a-8e44-fe366584a267" containerName="init-config-reloader" Nov 03 01:52:24.199011 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.198920 5605 cpu_manager.go:414] "containerMap: removing container" podUID="3d7d07f7-27c2-489a-8e44-fe366584a267" containerName="prom-label-proxy" Nov 03 01:52:24.199011 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.198925 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d7d07f7-27c2-489a-8e44-fe366584a267" containerName="prom-label-proxy" Nov 03 01:52:24.199011 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.198934 5605 cpu_manager.go:414] "containerMap: removing container" podUID="3d7d07f7-27c2-489a-8e44-fe366584a267" containerName="kube-rbac-proxy" Nov 03 01:52:24.199011 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.198939 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d7d07f7-27c2-489a-8e44-fe366584a267" containerName="kube-rbac-proxy" Nov 03 01:52:24.199011 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.198950 5605 cpu_manager.go:414] "containerMap: removing container" podUID="3d7d07f7-27c2-489a-8e44-fe366584a267" containerName="kube-rbac-proxy-web" Nov 03 01:52:24.199011 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.198955 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d7d07f7-27c2-489a-8e44-fe366584a267" containerName="kube-rbac-proxy-web" Nov 03 01:52:24.199011 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.198993 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d7d07f7-27c2-489a-8e44-fe366584a267" containerName="prom-label-proxy" Nov 03 01:52:24.199011 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.199000 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d7d07f7-27c2-489a-8e44-fe366584a267" containerName="kube-rbac-proxy" Nov 03 01:52:24.199011 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.199008 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d7d07f7-27c2-489a-8e44-fe366584a267" containerName="kube-rbac-proxy-web" Nov 03 01:52:24.199011 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.199014 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d7d07f7-27c2-489a-8e44-fe366584a267" containerName="alertmanager" Nov 03 01:52:24.199351 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.199019 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d7d07f7-27c2-489a-8e44-fe366584a267" containerName="kube-rbac-proxy-metric" Nov 03 01:52:24.199351 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.199026 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d7d07f7-27c2-489a-8e44-fe366584a267" containerName="config-reloader" Nov 03 01:52:24.199351 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.199031 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d7d07f7-27c2-489a-8e44-fe366584a267" containerName="init-config-reloader" Nov 03 01:52:24.204051 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.204030 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:24.253688 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.253655 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 03 01:52:24.253867 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.253769 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 03 01:52:24.253867 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.253773 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 03 01:52:24.253988 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.253845 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 03 01:52:24.253988 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.253909 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 03 01:52:24.254100 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.254083 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-rvjqm\"" Nov 03 01:52:24.254152 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.254097 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 03 01:52:24.254152 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.254133 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 03 01:52:24.254273 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.254257 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 03 01:52:24.272693 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.272664 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83c2e7de-c853-4518-a01a-80e3a29107ed-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"83c2e7de-c853-4518-a01a-80e3a29107ed\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:24.272798 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.272696 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/83c2e7de-c853-4518-a01a-80e3a29107ed-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"83c2e7de-c853-4518-a01a-80e3a29107ed\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:24.272798 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.272717 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/83c2e7de-c853-4518-a01a-80e3a29107ed-config-out\") pod \"alertmanager-main-0\" (UID: \"83c2e7de-c853-4518-a01a-80e3a29107ed\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:24.272798 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.272775 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83c2e7de-c853-4518-a01a-80e3a29107ed-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"83c2e7de-c853-4518-a01a-80e3a29107ed\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:24.272882 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.272806 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8rj8\" (UniqueName: \"kubernetes.io/projected/83c2e7de-c853-4518-a01a-80e3a29107ed-kube-api-access-v8rj8\") pod \"alertmanager-main-0\" (UID: \"83c2e7de-c853-4518-a01a-80e3a29107ed\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:24.272882 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.272853 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/83c2e7de-c853-4518-a01a-80e3a29107ed-config-volume\") pod \"alertmanager-main-0\" (UID: \"83c2e7de-c853-4518-a01a-80e3a29107ed\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:24.272989 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.272930 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/83c2e7de-c853-4518-a01a-80e3a29107ed-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"83c2e7de-c853-4518-a01a-80e3a29107ed\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:24.272989 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.272964 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/83c2e7de-c853-4518-a01a-80e3a29107ed-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"83c2e7de-c853-4518-a01a-80e3a29107ed\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:24.273043 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.273010 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/83c2e7de-c853-4518-a01a-80e3a29107ed-web-config\") pod \"alertmanager-main-0\" (UID: \"83c2e7de-c853-4518-a01a-80e3a29107ed\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:24.273043 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.273035 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/83c2e7de-c853-4518-a01a-80e3a29107ed-tls-assets\") pod \"alertmanager-main-0\" (UID: \"83c2e7de-c853-4518-a01a-80e3a29107ed\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:24.273121 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.273108 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/83c2e7de-c853-4518-a01a-80e3a29107ed-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"83c2e7de-c853-4518-a01a-80e3a29107ed\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:24.273153 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.273137 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/83c2e7de-c853-4518-a01a-80e3a29107ed-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"83c2e7de-c853-4518-a01a-80e3a29107ed\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:24.273181 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.273152 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/83c2e7de-c853-4518-a01a-80e3a29107ed-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"83c2e7de-c853-4518-a01a-80e3a29107ed\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:24.300646 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.300614 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 03 01:52:24.301321 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.301301 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 01:52:24.373546 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.373512 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/83c2e7de-c853-4518-a01a-80e3a29107ed-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"83c2e7de-c853-4518-a01a-80e3a29107ed\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:24.373546 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.373550 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/83c2e7de-c853-4518-a01a-80e3a29107ed-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"83c2e7de-c853-4518-a01a-80e3a29107ed\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:24.373754 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.373571 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/83c2e7de-c853-4518-a01a-80e3a29107ed-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"83c2e7de-c853-4518-a01a-80e3a29107ed\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:24.373754 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.373595 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83c2e7de-c853-4518-a01a-80e3a29107ed-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"83c2e7de-c853-4518-a01a-80e3a29107ed\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:24.373754 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.373648 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/83c2e7de-c853-4518-a01a-80e3a29107ed-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"83c2e7de-c853-4518-a01a-80e3a29107ed\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:24.373754 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.373688 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/83c2e7de-c853-4518-a01a-80e3a29107ed-config-out\") pod \"alertmanager-main-0\" (UID: \"83c2e7de-c853-4518-a01a-80e3a29107ed\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:24.373754 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.373731 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83c2e7de-c853-4518-a01a-80e3a29107ed-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"83c2e7de-c853-4518-a01a-80e3a29107ed\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:24.373754 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.373747 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8rj8\" (UniqueName: \"kubernetes.io/projected/83c2e7de-c853-4518-a01a-80e3a29107ed-kube-api-access-v8rj8\") pod \"alertmanager-main-0\" (UID: \"83c2e7de-c853-4518-a01a-80e3a29107ed\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:24.374029 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.373762 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/83c2e7de-c853-4518-a01a-80e3a29107ed-config-volume\") pod \"alertmanager-main-0\" (UID: \"83c2e7de-c853-4518-a01a-80e3a29107ed\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:24.374029 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.373782 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/83c2e7de-c853-4518-a01a-80e3a29107ed-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"83c2e7de-c853-4518-a01a-80e3a29107ed\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:24.374029 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.373835 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/83c2e7de-c853-4518-a01a-80e3a29107ed-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"83c2e7de-c853-4518-a01a-80e3a29107ed\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:24.374029 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.373865 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/83c2e7de-c853-4518-a01a-80e3a29107ed-web-config\") pod \"alertmanager-main-0\" (UID: \"83c2e7de-c853-4518-a01a-80e3a29107ed\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:24.374029 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.373916 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/83c2e7de-c853-4518-a01a-80e3a29107ed-tls-assets\") pod \"alertmanager-main-0\" (UID: \"83c2e7de-c853-4518-a01a-80e3a29107ed\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:24.374546 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.374427 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/83c2e7de-c853-4518-a01a-80e3a29107ed-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"83c2e7de-c853-4518-a01a-80e3a29107ed\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:24.374546 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.374528 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83c2e7de-c853-4518-a01a-80e3a29107ed-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"83c2e7de-c853-4518-a01a-80e3a29107ed\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:24.375178 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.375150 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83c2e7de-c853-4518-a01a-80e3a29107ed-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"83c2e7de-c853-4518-a01a-80e3a29107ed\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:24.376803 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.376759 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/83c2e7de-c853-4518-a01a-80e3a29107ed-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"83c2e7de-c853-4518-a01a-80e3a29107ed\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:24.376915 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.376839 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/83c2e7de-c853-4518-a01a-80e3a29107ed-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"83c2e7de-c853-4518-a01a-80e3a29107ed\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:24.377155 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.377129 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/83c2e7de-c853-4518-a01a-80e3a29107ed-config-out\") pod \"alertmanager-main-0\" (UID: \"83c2e7de-c853-4518-a01a-80e3a29107ed\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:24.377266 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.377187 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/83c2e7de-c853-4518-a01a-80e3a29107ed-tls-assets\") pod \"alertmanager-main-0\" (UID: \"83c2e7de-c853-4518-a01a-80e3a29107ed\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:24.377327 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.377283 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/83c2e7de-c853-4518-a01a-80e3a29107ed-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"83c2e7de-c853-4518-a01a-80e3a29107ed\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:24.377487 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.377465 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/83c2e7de-c853-4518-a01a-80e3a29107ed-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"83c2e7de-c853-4518-a01a-80e3a29107ed\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:24.377487 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.377481 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/83c2e7de-c853-4518-a01a-80e3a29107ed-config-volume\") pod \"alertmanager-main-0\" (UID: \"83c2e7de-c853-4518-a01a-80e3a29107ed\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:24.377605 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.377560 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/83c2e7de-c853-4518-a01a-80e3a29107ed-web-config\") pod \"alertmanager-main-0\" (UID: \"83c2e7de-c853-4518-a01a-80e3a29107ed\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:24.378638 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.378619 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/83c2e7de-c853-4518-a01a-80e3a29107ed-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"83c2e7de-c853-4518-a01a-80e3a29107ed\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:24.426414 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.426333 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8rj8\" (UniqueName: \"kubernetes.io/projected/83c2e7de-c853-4518-a01a-80e3a29107ed-kube-api-access-v8rj8\") pod \"alertmanager-main-0\" (UID: \"83c2e7de-c853-4518-a01a-80e3a29107ed\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:24.528064 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.528028 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 01:52:24.848108 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.848071 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d7d07f7-27c2-489a-8e44-fe366584a267" path="/var/lib/kubelet/pods/3d7d07f7-27c2-489a-8e44-fe366584a267/volumes" Nov 03 01:52:24.998158 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:24.998123 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 01:52:25.004948 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:52:25.004921 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83c2e7de_c853_4518_a01a_80e3a29107ed.slice/crio-2ae375e69be05af715817ea6e39ed47ae36473e76804497fd793c37e78734a2f WatchSource:0}: Error finding container 2ae375e69be05af715817ea6e39ed47ae36473e76804497fd793c37e78734a2f: Status 404 returned error can't find the container with id 2ae375e69be05af715817ea6e39ed47ae36473e76804497fd793c37e78734a2f Nov 03 01:52:26.006862 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:26.006821 5605 generic.go:356] "Generic (PLEG): container finished" podID="83c2e7de-c853-4518-a01a-80e3a29107ed" containerID="eb0ce77702c55d4b2a9001bde036016c99df69fec4f8f84aab5f8744275fe5a9" exitCode=0 Nov 03 01:52:26.007237 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:26.006884 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"83c2e7de-c853-4518-a01a-80e3a29107ed","Type":"ContainerDied","Data":"eb0ce77702c55d4b2a9001bde036016c99df69fec4f8f84aab5f8744275fe5a9"} Nov 03 01:52:26.007237 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:26.006951 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"83c2e7de-c853-4518-a01a-80e3a29107ed","Type":"ContainerStarted","Data":"2ae375e69be05af715817ea6e39ed47ae36473e76804497fd793c37e78734a2f"} Nov 03 01:52:27.014573 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:27.014538 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"83c2e7de-c853-4518-a01a-80e3a29107ed","Type":"ContainerStarted","Data":"4664ac7636bc45e7fd26bc89d2c0270e94571e6f04cddddedbd13d218f8d4608"} Nov 03 01:52:27.015057 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:27.014615 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"83c2e7de-c853-4518-a01a-80e3a29107ed","Type":"ContainerStarted","Data":"7bd313041ce30f23c4e3de1cf21b3c6b02925867b4b990533c4e3c3f15e63eb9"} Nov 03 01:52:27.015057 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:27.014653 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"83c2e7de-c853-4518-a01a-80e3a29107ed","Type":"ContainerStarted","Data":"6c3962d15b3f4189a5e38a61d231311d84deda930b74a8f54a7df5f8c3e50ad7"} Nov 03 01:52:27.015057 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:27.014690 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"83c2e7de-c853-4518-a01a-80e3a29107ed","Type":"ContainerStarted","Data":"9271f0bb466155fa16f16db82cf02223a3dbd9e02ba3023d6641e7138a311367"} Nov 03 01:52:27.015057 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:27.014729 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"83c2e7de-c853-4518-a01a-80e3a29107ed","Type":"ContainerStarted","Data":"d8f1877eeda361df8d62dc9328e043a7722d71f2b059f249b026fe24e72f9b7e"} Nov 03 01:52:27.015057 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:27.014783 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"83c2e7de-c853-4518-a01a-80e3a29107ed","Type":"ContainerStarted","Data":"76f87eeeef8cc4e5db14ece7d94abb75d13928d949d3b0136b45b029ad417f4c"} Nov 03 01:52:27.128988 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:27.128543 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.113944903 podStartE2EDuration="3.128528446s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.583543ms" isStatefulPod=false podCreationTimestamp="2026-11-03 01:52:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:52:27.127583375 +0000 UTC m=+24.386781528" watchObservedRunningTime="2026-11-03 01:52:27.128528446 +0000 UTC m=+24.387726599" Nov 03 01:52:27.514541 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:27.514497 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7489f79677-c9g7j" Nov 03 01:52:27.514736 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:27.514566 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7489f79677-c9g7j" Nov 03 01:52:27.519423 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:27.519401 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7489f79677-c9g7j" Nov 03 01:52:28.022154 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:28.022125 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7489f79677-c9g7j" Nov 03 01:52:28.161494 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:28.161462 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-69c467f7c8-cpfwh"] Nov 03 01:52:28.445543 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:28.445511 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-b984f6b6b-g6f2h" Nov 03 01:52:28.449494 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:28.449470 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-b984f6b6b-g6f2h" Nov 03 01:52:28.966677 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:28.966637 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-676fb595d5-lmt4w" podUID="13fa5796-b3e6-4d1c-9759-88b7094d4b37" containerName="console" containerID="cri-o://d499decdd20c159be35ca7f639b97bf63f9631407a3f6dde6ed02d2b27e21805" gracePeriod=15 Nov 03 01:52:29.224083 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:29.224016 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-676fb595d5-lmt4w" Nov 03 01:52:29.319945 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:29.319912 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13fa5796-b3e6-4d1c-9759-88b7094d4b37-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/13fa5796-b3e6-4d1c-9759-88b7094d4b37-console-oauth-config\") pod \"13fa5796-b3e6-4d1c-9759-88b7094d4b37\" (UID: \"13fa5796-b3e6-4d1c-9759-88b7094d4b37\") " Nov 03 01:52:29.319945 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:29.319950 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13fa5796-b3e6-4d1c-9759-88b7094d4b37-console-config\" (UniqueName: \"kubernetes.io/configmap/13fa5796-b3e6-4d1c-9759-88b7094d4b37-console-config\") pod \"13fa5796-b3e6-4d1c-9759-88b7094d4b37\" (UID: \"13fa5796-b3e6-4d1c-9759-88b7094d4b37\") " Nov 03 01:52:29.320114 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:29.319993 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13fa5796-b3e6-4d1c-9759-88b7094d4b37-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/13fa5796-b3e6-4d1c-9759-88b7094d4b37-console-serving-cert\") pod \"13fa5796-b3e6-4d1c-9759-88b7094d4b37\" (UID: \"13fa5796-b3e6-4d1c-9759-88b7094d4b37\") " Nov 03 01:52:29.320114 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:29.320037 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13fa5796-b3e6-4d1c-9759-88b7094d4b37-service-ca\" (UniqueName: \"kubernetes.io/configmap/13fa5796-b3e6-4d1c-9759-88b7094d4b37-service-ca\") pod \"13fa5796-b3e6-4d1c-9759-88b7094d4b37\" (UID: \"13fa5796-b3e6-4d1c-9759-88b7094d4b37\") " Nov 03 01:52:29.320114 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:29.320079 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13fa5796-b3e6-4d1c-9759-88b7094d4b37-kube-api-access-bb4tp\" (UniqueName: \"kubernetes.io/projected/13fa5796-b3e6-4d1c-9759-88b7094d4b37-kube-api-access-bb4tp\") pod \"13fa5796-b3e6-4d1c-9759-88b7094d4b37\" (UID: \"13fa5796-b3e6-4d1c-9759-88b7094d4b37\") " Nov 03 01:52:29.320257 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:29.320119 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13fa5796-b3e6-4d1c-9759-88b7094d4b37-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13fa5796-b3e6-4d1c-9759-88b7094d4b37-trusted-ca-bundle\") pod \"13fa5796-b3e6-4d1c-9759-88b7094d4b37\" (UID: \"13fa5796-b3e6-4d1c-9759-88b7094d4b37\") " Nov 03 01:52:29.320257 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:29.320159 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13fa5796-b3e6-4d1c-9759-88b7094d4b37-tmp\" (UniqueName: \"kubernetes.io/empty-dir/13fa5796-b3e6-4d1c-9759-88b7094d4b37-tmp\") pod \"13fa5796-b3e6-4d1c-9759-88b7094d4b37\" (UID: \"13fa5796-b3e6-4d1c-9759-88b7094d4b37\") " Nov 03 01:52:29.320257 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:29.320188 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13fa5796-b3e6-4d1c-9759-88b7094d4b37-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/13fa5796-b3e6-4d1c-9759-88b7094d4b37-oauth-serving-cert\") pod \"13fa5796-b3e6-4d1c-9759-88b7094d4b37\" (UID: \"13fa5796-b3e6-4d1c-9759-88b7094d4b37\") " Nov 03 01:52:29.320567 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:29.320535 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13fa5796-b3e6-4d1c-9759-88b7094d4b37-console-config" pod "13fa5796-b3e6-4d1c-9759-88b7094d4b37" (UID: "13fa5796-b3e6-4d1c-9759-88b7094d4b37"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:52:29.320567 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:29.320535 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13fa5796-b3e6-4d1c-9759-88b7094d4b37-tmp" pod "13fa5796-b3e6-4d1c-9759-88b7094d4b37" (UID: "13fa5796-b3e6-4d1c-9759-88b7094d4b37"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:52:29.320748 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:29.320688 5605 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/13fa5796-b3e6-4d1c-9759-88b7094d4b37-console-config\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:52:29.320748 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:29.320704 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13fa5796-b3e6-4d1c-9759-88b7094d4b37-trusted-ca-bundle" pod "13fa5796-b3e6-4d1c-9759-88b7094d4b37" (UID: "13fa5796-b3e6-4d1c-9759-88b7094d4b37"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:52:29.320748 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:29.320717 5605 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/13fa5796-b3e6-4d1c-9759-88b7094d4b37-tmp\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:52:29.320870 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:29.320752 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13fa5796-b3e6-4d1c-9759-88b7094d4b37-service-ca" pod "13fa5796-b3e6-4d1c-9759-88b7094d4b37" (UID: "13fa5796-b3e6-4d1c-9759-88b7094d4b37"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:52:29.320985 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:29.320965 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13fa5796-b3e6-4d1c-9759-88b7094d4b37-oauth-serving-cert" pod "13fa5796-b3e6-4d1c-9759-88b7094d4b37" (UID: "13fa5796-b3e6-4d1c-9759-88b7094d4b37"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:52:29.322544 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:29.322522 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13fa5796-b3e6-4d1c-9759-88b7094d4b37-kube-api-access-bb4tp" pod "13fa5796-b3e6-4d1c-9759-88b7094d4b37" (UID: "13fa5796-b3e6-4d1c-9759-88b7094d4b37"). InnerVolumeSpecName "kube-api-access-bb4tp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:52:29.322614 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:29.322560 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13fa5796-b3e6-4d1c-9759-88b7094d4b37-console-serving-cert" pod "13fa5796-b3e6-4d1c-9759-88b7094d4b37" (UID: "13fa5796-b3e6-4d1c-9759-88b7094d4b37"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:52:29.322658 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:29.322620 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13fa5796-b3e6-4d1c-9759-88b7094d4b37-console-oauth-config" pod "13fa5796-b3e6-4d1c-9759-88b7094d4b37" (UID: "13fa5796-b3e6-4d1c-9759-88b7094d4b37"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:52:29.421732 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:29.421683 5605 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13fa5796-b3e6-4d1c-9759-88b7094d4b37-trusted-ca-bundle\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:52:29.421732 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:29.421725 5605 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/13fa5796-b3e6-4d1c-9759-88b7094d4b37-oauth-serving-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:52:29.421732 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:29.421734 5605 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/13fa5796-b3e6-4d1c-9759-88b7094d4b37-console-oauth-config\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:52:29.421732 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:29.421743 5605 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/13fa5796-b3e6-4d1c-9759-88b7094d4b37-console-serving-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:52:29.421993 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:29.421753 5605 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/13fa5796-b3e6-4d1c-9759-88b7094d4b37-service-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:52:29.421993 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:29.421762 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bb4tp\" (UniqueName: \"kubernetes.io/projected/13fa5796-b3e6-4d1c-9759-88b7094d4b37-kube-api-access-bb4tp\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:52:29.936370 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:29.936320 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-59fbbdf587-6nm8q" podUID="3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6" containerName="console" containerID="cri-o://e7e44cb46e26b5bbb607e7a166a8f0c8378543626fe3447c00a07938b800f84a" gracePeriod=15 Nov 03 01:52:30.027011 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:30.026977 5605 generic.go:356] "Generic (PLEG): container finished" podID="13fa5796-b3e6-4d1c-9759-88b7094d4b37" containerID="d499decdd20c159be35ca7f639b97bf63f9631407a3f6dde6ed02d2b27e21805" exitCode=0 Nov 03 01:52:30.027163 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:30.027015 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-676fb595d5-lmt4w" event={"ID":"13fa5796-b3e6-4d1c-9759-88b7094d4b37","Type":"ContainerDied","Data":"d499decdd20c159be35ca7f639b97bf63f9631407a3f6dde6ed02d2b27e21805"} Nov 03 01:52:30.027163 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:30.027031 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-676fb595d5-lmt4w" Nov 03 01:52:30.027163 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:30.027069 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-676fb595d5-lmt4w" event={"ID":"13fa5796-b3e6-4d1c-9759-88b7094d4b37","Type":"ContainerDied","Data":"04fb7c6e6f597b086dc818c0595aee3f95569ab857b1f8ef044b5d76100b4942"} Nov 03 01:52:30.027163 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:30.027090 5605 scope.go:122] "RemoveContainer" containerID="d499decdd20c159be35ca7f639b97bf63f9631407a3f6dde6ed02d2b27e21805" Nov 03 01:52:30.046498 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:30.046474 5605 scope.go:122] "RemoveContainer" containerID="d499decdd20c159be35ca7f639b97bf63f9631407a3f6dde6ed02d2b27e21805" Nov 03 01:52:30.046775 ip-10-0-140-140 kubenswrapper[5605]: E1103 01:52:30.046756 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d499decdd20c159be35ca7f639b97bf63f9631407a3f6dde6ed02d2b27e21805\": container with ID starting with d499decdd20c159be35ca7f639b97bf63f9631407a3f6dde6ed02d2b27e21805 not found: ID does not exist" containerID="d499decdd20c159be35ca7f639b97bf63f9631407a3f6dde6ed02d2b27e21805" Nov 03 01:52:30.046857 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:30.046784 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d499decdd20c159be35ca7f639b97bf63f9631407a3f6dde6ed02d2b27e21805"} err="failed to get container status \"d499decdd20c159be35ca7f639b97bf63f9631407a3f6dde6ed02d2b27e21805\": rpc error: code = NotFound desc = could not find container \"d499decdd20c159be35ca7f639b97bf63f9631407a3f6dde6ed02d2b27e21805\": container with ID starting with d499decdd20c159be35ca7f639b97bf63f9631407a3f6dde6ed02d2b27e21805 not found: ID does not exist" Nov 03 01:52:30.179137 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:30.179111 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-676fb595d5-lmt4w"] Nov 03 01:52:30.180170 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:30.180155 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59fbbdf587-6nm8q" Nov 03 01:52:30.225739 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:30.225660 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-676fb595d5-lmt4w"] Nov 03 01:52:30.329702 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:30.329655 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-oauth-serving-cert\") pod \"3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6\" (UID: \"3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6\") " Nov 03 01:52:30.329863 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:30.329723 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-console-oauth-config\") pod \"3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6\" (UID: \"3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6\") " Nov 03 01:52:30.329863 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:30.329771 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-service-ca\") pod \"3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6\" (UID: \"3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6\") " Nov 03 01:52:30.329863 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:30.329791 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-kube-api-access-bmhdf\" (UniqueName: \"kubernetes.io/projected/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-kube-api-access-bmhdf\") pod \"3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6\" (UID: \"3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6\") " Nov 03 01:52:30.329863 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:30.329827 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-console-serving-cert\") pod \"3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6\" (UID: \"3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6\") " Nov 03 01:52:30.329863 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:30.329857 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-tmp\" (UniqueName: \"kubernetes.io/empty-dir/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-tmp\") pod \"3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6\" (UID: \"3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6\") " Nov 03 01:52:30.330089 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:30.329903 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-console-config\" (UniqueName: \"kubernetes.io/configmap/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-console-config\") pod \"3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6\" (UID: \"3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6\") " Nov 03 01:52:30.330242 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:30.330201 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-tmp" pod "3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6" (UID: "3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:52:30.330242 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:30.330222 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-service-ca" pod "3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6" (UID: "3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:52:30.330497 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:30.330214 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-oauth-serving-cert" pod "3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6" (UID: "3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:52:30.330497 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:30.330410 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-console-config" pod "3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6" (UID: "3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:52:30.332616 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:30.332589 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-console-serving-cert" pod "3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6" (UID: "3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:52:30.332750 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:30.332624 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-kube-api-access-bmhdf" pod "3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6" (UID: "3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6"). InnerVolumeSpecName "kube-api-access-bmhdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:52:30.332750 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:30.332688 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-console-oauth-config" pod "3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6" (UID: "3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:52:30.430651 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:30.430612 5605 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-service-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:52:30.430651 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:30.430645 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmhdf\" (UniqueName: \"kubernetes.io/projected/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-kube-api-access-bmhdf\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:52:30.430651 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:30.430656 5605 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-console-serving-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:52:30.430871 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:30.430665 5605 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-tmp\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:52:30.430871 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:30.430674 5605 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-console-config\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:52:30.430871 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:30.430682 5605 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-oauth-serving-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:52:30.430871 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:30.430693 5605 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6-console-oauth-config\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:52:30.850554 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:30.848749 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13fa5796-b3e6-4d1c-9759-88b7094d4b37" path="/var/lib/kubelet/pods/13fa5796-b3e6-4d1c-9759-88b7094d4b37/volumes" Nov 03 01:52:31.033118 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:31.033084 5605 generic.go:356] "Generic (PLEG): container finished" podID="3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6" containerID="e7e44cb46e26b5bbb607e7a166a8f0c8378543626fe3447c00a07938b800f84a" exitCode=0 Nov 03 01:52:31.033289 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:31.033124 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59fbbdf587-6nm8q" event={"ID":"3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6","Type":"ContainerDied","Data":"e7e44cb46e26b5bbb607e7a166a8f0c8378543626fe3447c00a07938b800f84a"} Nov 03 01:52:31.033289 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:31.033152 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59fbbdf587-6nm8q" Nov 03 01:52:31.033289 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:31.033171 5605 scope.go:122] "RemoveContainer" containerID="e7e44cb46e26b5bbb607e7a166a8f0c8378543626fe3447c00a07938b800f84a" Nov 03 01:52:31.033289 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:31.033160 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59fbbdf587-6nm8q" event={"ID":"3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6","Type":"ContainerDied","Data":"6c90f04210adfeedaf7fc82f96407539f7eef4c1ced7c1f75c523b11d27cc513"} Nov 03 01:52:31.043345 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:31.043332 5605 scope.go:122] "RemoveContainer" containerID="e7e44cb46e26b5bbb607e7a166a8f0c8378543626fe3447c00a07938b800f84a" Nov 03 01:52:31.043631 ip-10-0-140-140 kubenswrapper[5605]: E1103 01:52:31.043608 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7e44cb46e26b5bbb607e7a166a8f0c8378543626fe3447c00a07938b800f84a\": container with ID starting with e7e44cb46e26b5bbb607e7a166a8f0c8378543626fe3447c00a07938b800f84a not found: ID does not exist" containerID="e7e44cb46e26b5bbb607e7a166a8f0c8378543626fe3447c00a07938b800f84a" Nov 03 01:52:31.043697 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:31.043639 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7e44cb46e26b5bbb607e7a166a8f0c8378543626fe3447c00a07938b800f84a"} err="failed to get container status \"e7e44cb46e26b5bbb607e7a166a8f0c8378543626fe3447c00a07938b800f84a\": rpc error: code = NotFound desc = could not find container \"e7e44cb46e26b5bbb607e7a166a8f0c8378543626fe3447c00a07938b800f84a\": container with ID starting with e7e44cb46e26b5bbb607e7a166a8f0c8378543626fe3447c00a07938b800f84a not found: ID does not exist" Nov 03 01:52:31.184397 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:31.184307 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59fbbdf587-6nm8q"] Nov 03 01:52:31.231054 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:31.231023 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-59fbbdf587-6nm8q"] Nov 03 01:52:31.799633 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:31.799604 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x452r" Nov 03 01:52:32.848324 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:32.848288 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6" path="/var/lib/kubelet/pods/3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6/volumes" Nov 03 01:52:35.438808 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:35.438771 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-f5qvs" Nov 03 01:52:53.185398 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:53.185317 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-69c467f7c8-cpfwh" podUID="a4727262-eb15-4453-adf2-0fc6c2358508" containerName="console" containerID="cri-o://6ed06dd8080d8cc71cbc17d832250c0d5dc93fa2cd71732a1350fc35b6a29d49" gracePeriod=15 Nov 03 01:52:53.445999 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:53.445940 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69c467f7c8-cpfwh" Nov 03 01:52:53.524057 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:53.524024 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4727262-eb15-4453-adf2-0fc6c2358508-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4727262-eb15-4453-adf2-0fc6c2358508-trusted-ca-bundle\") pod \"a4727262-eb15-4453-adf2-0fc6c2358508\" (UID: \"a4727262-eb15-4453-adf2-0fc6c2358508\") " Nov 03 01:52:53.524179 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:53.524065 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4727262-eb15-4453-adf2-0fc6c2358508-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a4727262-eb15-4453-adf2-0fc6c2358508-oauth-serving-cert\") pod \"a4727262-eb15-4453-adf2-0fc6c2358508\" (UID: \"a4727262-eb15-4453-adf2-0fc6c2358508\") " Nov 03 01:52:53.524179 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:53.524096 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4727262-eb15-4453-adf2-0fc6c2358508-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4727262-eb15-4453-adf2-0fc6c2358508-service-ca\") pod \"a4727262-eb15-4453-adf2-0fc6c2358508\" (UID: \"a4727262-eb15-4453-adf2-0fc6c2358508\") " Nov 03 01:52:53.524179 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:53.524119 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4727262-eb15-4453-adf2-0fc6c2358508-console-config\" (UniqueName: \"kubernetes.io/configmap/a4727262-eb15-4453-adf2-0fc6c2358508-console-config\") pod \"a4727262-eb15-4453-adf2-0fc6c2358508\" (UID: \"a4727262-eb15-4453-adf2-0fc6c2358508\") " Nov 03 01:52:53.524179 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:53.524143 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4727262-eb15-4453-adf2-0fc6c2358508-kube-api-access-d9kwt\" (UniqueName: \"kubernetes.io/projected/a4727262-eb15-4453-adf2-0fc6c2358508-kube-api-access-d9kwt\") pod \"a4727262-eb15-4453-adf2-0fc6c2358508\" (UID: \"a4727262-eb15-4453-adf2-0fc6c2358508\") " Nov 03 01:52:53.524392 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:53.524181 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4727262-eb15-4453-adf2-0fc6c2358508-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a4727262-eb15-4453-adf2-0fc6c2358508-console-serving-cert\") pod \"a4727262-eb15-4453-adf2-0fc6c2358508\" (UID: \"a4727262-eb15-4453-adf2-0fc6c2358508\") " Nov 03 01:52:53.524392 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:53.524225 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4727262-eb15-4453-adf2-0fc6c2358508-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a4727262-eb15-4453-adf2-0fc6c2358508-console-oauth-config\") pod \"a4727262-eb15-4453-adf2-0fc6c2358508\" (UID: \"a4727262-eb15-4453-adf2-0fc6c2358508\") " Nov 03 01:52:53.524392 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:53.524271 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4727262-eb15-4453-adf2-0fc6c2358508-tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4727262-eb15-4453-adf2-0fc6c2358508-tmp\") pod \"a4727262-eb15-4453-adf2-0fc6c2358508\" (UID: \"a4727262-eb15-4453-adf2-0fc6c2358508\") " Nov 03 01:52:53.524594 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:53.524561 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4727262-eb15-4453-adf2-0fc6c2358508-service-ca" pod "a4727262-eb15-4453-adf2-0fc6c2358508" (UID: "a4727262-eb15-4453-adf2-0fc6c2358508"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:52:53.524594 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:53.524579 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4727262-eb15-4453-adf2-0fc6c2358508-trusted-ca-bundle" pod "a4727262-eb15-4453-adf2-0fc6c2358508" (UID: "a4727262-eb15-4453-adf2-0fc6c2358508"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:52:53.524776 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:53.524614 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4727262-eb15-4453-adf2-0fc6c2358508-oauth-serving-cert" pod "a4727262-eb15-4453-adf2-0fc6c2358508" (UID: "a4727262-eb15-4453-adf2-0fc6c2358508"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:52:53.524776 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:53.524623 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4727262-eb15-4453-adf2-0fc6c2358508-console-config" pod "a4727262-eb15-4453-adf2-0fc6c2358508" (UID: "a4727262-eb15-4453-adf2-0fc6c2358508"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:52:53.524776 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:53.524749 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4727262-eb15-4453-adf2-0fc6c2358508-tmp" pod "a4727262-eb15-4453-adf2-0fc6c2358508" (UID: "a4727262-eb15-4453-adf2-0fc6c2358508"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:52:53.526640 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:53.526612 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4727262-eb15-4453-adf2-0fc6c2358508-kube-api-access-d9kwt" pod "a4727262-eb15-4453-adf2-0fc6c2358508" (UID: "a4727262-eb15-4453-adf2-0fc6c2358508"). InnerVolumeSpecName "kube-api-access-d9kwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:52:53.526724 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:53.526621 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4727262-eb15-4453-adf2-0fc6c2358508-console-serving-cert" pod "a4727262-eb15-4453-adf2-0fc6c2358508" (UID: "a4727262-eb15-4453-adf2-0fc6c2358508"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:52:53.526724 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:53.526671 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4727262-eb15-4453-adf2-0fc6c2358508-console-oauth-config" pod "a4727262-eb15-4453-adf2-0fc6c2358508" (UID: "a4727262-eb15-4453-adf2-0fc6c2358508"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:52:53.625592 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:53.625556 5605 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4727262-eb15-4453-adf2-0fc6c2358508-tmp\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:52:53.625592 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:53.625586 5605 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4727262-eb15-4453-adf2-0fc6c2358508-trusted-ca-bundle\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:52:53.625592 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:53.625597 5605 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a4727262-eb15-4453-adf2-0fc6c2358508-oauth-serving-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:52:53.625847 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:53.625607 5605 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a4727262-eb15-4453-adf2-0fc6c2358508-service-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:52:53.625847 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:53.625617 5605 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a4727262-eb15-4453-adf2-0fc6c2358508-console-config\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:52:53.625847 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:53.625624 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9kwt\" (UniqueName: \"kubernetes.io/projected/a4727262-eb15-4453-adf2-0fc6c2358508-kube-api-access-d9kwt\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:52:53.625847 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:53.625633 5605 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a4727262-eb15-4453-adf2-0fc6c2358508-console-serving-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:52:53.625847 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:53.625642 5605 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a4727262-eb15-4453-adf2-0fc6c2358508-console-oauth-config\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:52:53.910924 ip-10-0-140-140 kubenswrapper[5605]: E1103 01:52:53.910883 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 01:52:54.116997 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:54.116961 5605 generic.go:356] "Generic (PLEG): container finished" podID="a4727262-eb15-4453-adf2-0fc6c2358508" containerID="6ed06dd8080d8cc71cbc17d832250c0d5dc93fa2cd71732a1350fc35b6a29d49" exitCode=0 Nov 03 01:52:54.117166 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:54.117008 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69c467f7c8-cpfwh" Nov 03 01:52:54.117166 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:54.117010 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69c467f7c8-cpfwh" event={"ID":"a4727262-eb15-4453-adf2-0fc6c2358508","Type":"ContainerDied","Data":"6ed06dd8080d8cc71cbc17d832250c0d5dc93fa2cd71732a1350fc35b6a29d49"} Nov 03 01:52:54.117166 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:54.117137 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69c467f7c8-cpfwh" event={"ID":"a4727262-eb15-4453-adf2-0fc6c2358508","Type":"ContainerDied","Data":"59950987f839d6591cdddd6cab36a4cf1540b371be239279dc6c981d90a53e6a"} Nov 03 01:52:54.117266 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:54.117170 5605 scope.go:122] "RemoveContainer" containerID="6ed06dd8080d8cc71cbc17d832250c0d5dc93fa2cd71732a1350fc35b6a29d49" Nov 03 01:52:54.125642 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:54.125623 5605 scope.go:122] "RemoveContainer" containerID="6ed06dd8080d8cc71cbc17d832250c0d5dc93fa2cd71732a1350fc35b6a29d49" Nov 03 01:52:54.125930 ip-10-0-140-140 kubenswrapper[5605]: E1103 01:52:54.125914 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ed06dd8080d8cc71cbc17d832250c0d5dc93fa2cd71732a1350fc35b6a29d49\": container with ID starting with 6ed06dd8080d8cc71cbc17d832250c0d5dc93fa2cd71732a1350fc35b6a29d49 not found: ID does not exist" containerID="6ed06dd8080d8cc71cbc17d832250c0d5dc93fa2cd71732a1350fc35b6a29d49" Nov 03 01:52:54.125976 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:54.125940 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ed06dd8080d8cc71cbc17d832250c0d5dc93fa2cd71732a1350fc35b6a29d49"} err="failed to get container status \"6ed06dd8080d8cc71cbc17d832250c0d5dc93fa2cd71732a1350fc35b6a29d49\": rpc error: code = NotFound desc = could not find container \"6ed06dd8080d8cc71cbc17d832250c0d5dc93fa2cd71732a1350fc35b6a29d49\": container with ID starting with 6ed06dd8080d8cc71cbc17d832250c0d5dc93fa2cd71732a1350fc35b6a29d49 not found: ID does not exist" Nov 03 01:52:54.268250 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:54.268176 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-69c467f7c8-cpfwh"] Nov 03 01:52:54.310623 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:54.310589 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-69c467f7c8-cpfwh"] Nov 03 01:52:54.847092 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:52:54.847057 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4727262-eb15-4453-adf2-0fc6c2358508" path="/var/lib/kubelet/pods/a4727262-eb15-4453-adf2-0fc6c2358508/volumes" Nov 03 01:53:23.910809 ip-10-0-140-140 kubenswrapper[5605]: E1103 01:53:23.910768 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 01:53:53.911761 ip-10-0-140-140 kubenswrapper[5605]: E1103 01:53:53.911684 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 01:54:20.608582 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:20.608537 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxmtxt"] Nov 03 01:54:20.609218 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:20.608949 5605 cpu_manager.go:414] "containerMap: removing container" podUID="13fa5796-b3e6-4d1c-9759-88b7094d4b37" containerName="console" Nov 03 01:54:20.609218 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:20.608968 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13fa5796-b3e6-4d1c-9759-88b7094d4b37" containerName="console" Nov 03 01:54:20.609218 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:20.608983 5605 cpu_manager.go:414] "containerMap: removing container" podUID="a4727262-eb15-4453-adf2-0fc6c2358508" containerName="console" Nov 03 01:54:20.609218 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:20.608991 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4727262-eb15-4453-adf2-0fc6c2358508" containerName="console" Nov 03 01:54:20.609218 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:20.609018 5605 cpu_manager.go:414] "containerMap: removing container" podUID="3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6" containerName="console" Nov 03 01:54:20.609218 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:20.609025 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6" containerName="console" Nov 03 01:54:20.609218 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:20.609103 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="13fa5796-b3e6-4d1c-9759-88b7094d4b37" containerName="console" Nov 03 01:54:20.609218 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:20.609114 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4727262-eb15-4453-adf2-0fc6c2358508" containerName="console" Nov 03 01:54:20.609218 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:20.609126 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fde4b63-e3dc-4b4e-97a5-ad6dc8fe48f6" containerName="console" Nov 03 01:54:20.612257 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:20.612237 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxmtxt" Nov 03 01:54:20.676196 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:20.676164 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a938e094-9132-4ac7-856f-a7df518006b3-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxmtxt\" (UID: \"a938e094-9132-4ac7-856f-a7df518006b3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxmtxt" Nov 03 01:54:20.676332 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:20.676215 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a938e094-9132-4ac7-856f-a7df518006b3-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxmtxt\" (UID: \"a938e094-9132-4ac7-856f-a7df518006b3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxmtxt" Nov 03 01:54:20.676332 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:20.676306 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqzgn\" (UniqueName: \"kubernetes.io/projected/a938e094-9132-4ac7-856f-a7df518006b3-kube-api-access-qqzgn\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxmtxt\" (UID: \"a938e094-9132-4ac7-856f-a7df518006b3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxmtxt" Nov 03 01:54:20.710531 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:20.710505 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 03 01:54:20.710622 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:20.710527 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-qfgzh\"" Nov 03 01:54:20.710622 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:20.710537 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 03 01:54:20.711198 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:20.711183 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxmtxt"] Nov 03 01:54:20.777478 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:20.777443 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a938e094-9132-4ac7-856f-a7df518006b3-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxmtxt\" (UID: \"a938e094-9132-4ac7-856f-a7df518006b3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxmtxt" Nov 03 01:54:20.777478 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:20.777482 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a938e094-9132-4ac7-856f-a7df518006b3-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxmtxt\" (UID: \"a938e094-9132-4ac7-856f-a7df518006b3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxmtxt" Nov 03 01:54:20.777697 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:20.777516 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqzgn\" (UniqueName: \"kubernetes.io/projected/a938e094-9132-4ac7-856f-a7df518006b3-kube-api-access-qqzgn\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxmtxt\" (UID: \"a938e094-9132-4ac7-856f-a7df518006b3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxmtxt" Nov 03 01:54:20.777862 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:20.777842 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a938e094-9132-4ac7-856f-a7df518006b3-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxmtxt\" (UID: \"a938e094-9132-4ac7-856f-a7df518006b3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxmtxt" Nov 03 01:54:20.777912 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:20.777896 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a938e094-9132-4ac7-856f-a7df518006b3-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxmtxt\" (UID: \"a938e094-9132-4ac7-856f-a7df518006b3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxmtxt" Nov 03 01:54:20.830923 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:20.830887 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqzgn\" (UniqueName: \"kubernetes.io/projected/a938e094-9132-4ac7-856f-a7df518006b3-kube-api-access-qqzgn\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxmtxt\" (UID: \"a938e094-9132-4ac7-856f-a7df518006b3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxmtxt" Nov 03 01:54:20.926782 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:20.926706 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxmtxt" Nov 03 01:54:21.077917 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:21.077890 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-4c766"] Nov 03 01:54:21.081126 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:21.081105 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4c766" Nov 03 01:54:21.130176 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:21.130148 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 01:54:21.155994 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:21.155963 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-4c766"] Nov 03 01:54:21.167942 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:21.167918 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 01:54:21.168072 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:21.167976 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 01:54:21.168072 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:21.167991 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 01:54:21.168072 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:21.168006 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-2k54n\"" Nov 03 01:54:21.181375 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:21.181308 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0c4868b2-22cd-41c5-8e7d-d7dffacf1973-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-4c766\" (UID: \"0c4868b2-22cd-41c5-8e7d-d7dffacf1973\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4c766" Nov 03 01:54:21.181375 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:21.181351 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0c4868b2-22cd-41c5-8e7d-d7dffacf1973-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-4c766\" (UID: \"0c4868b2-22cd-41c5-8e7d-d7dffacf1973\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4c766" Nov 03 01:54:21.181511 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:21.181450 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0c4868b2-22cd-41c5-8e7d-d7dffacf1973-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-4c766\" (UID: \"0c4868b2-22cd-41c5-8e7d-d7dffacf1973\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4c766" Nov 03 01:54:21.181511 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:21.181476 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brb8q\" (UniqueName: \"kubernetes.io/projected/0c4868b2-22cd-41c5-8e7d-d7dffacf1973-kube-api-access-brb8q\") pod \"istiod-openshift-gateway-7876bfcb74-4c766\" (UID: \"0c4868b2-22cd-41c5-8e7d-d7dffacf1973\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4c766" Nov 03 01:54:21.181575 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:21.181559 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c4868b2-22cd-41c5-8e7d-d7dffacf1973-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-4c766\" (UID: \"0c4868b2-22cd-41c5-8e7d-d7dffacf1973\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4c766" Nov 03 01:54:21.181608 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:21.181584 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c4868b2-22cd-41c5-8e7d-d7dffacf1973-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-4c766\" (UID: \"0c4868b2-22cd-41c5-8e7d-d7dffacf1973\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4c766" Nov 03 01:54:21.181608 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:21.181601 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0c4868b2-22cd-41c5-8e7d-d7dffacf1973-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-4c766\" (UID: \"0c4868b2-22cd-41c5-8e7d-d7dffacf1973\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4c766" Nov 03 01:54:21.230586 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:21.230551 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 01:54:21.230731 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:21.230591 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 01:54:21.282991 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:21.282958 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c4868b2-22cd-41c5-8e7d-d7dffacf1973-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-4c766\" (UID: \"0c4868b2-22cd-41c5-8e7d-d7dffacf1973\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4c766" Nov 03 01:54:21.283084 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:21.282996 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c4868b2-22cd-41c5-8e7d-d7dffacf1973-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-4c766\" (UID: \"0c4868b2-22cd-41c5-8e7d-d7dffacf1973\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4c766" Nov 03 01:54:21.283084 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:21.283016 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0c4868b2-22cd-41c5-8e7d-d7dffacf1973-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-4c766\" (UID: \"0c4868b2-22cd-41c5-8e7d-d7dffacf1973\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4c766" Nov 03 01:54:21.283170 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:21.283081 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0c4868b2-22cd-41c5-8e7d-d7dffacf1973-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-4c766\" (UID: \"0c4868b2-22cd-41c5-8e7d-d7dffacf1973\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4c766" Nov 03 01:54:21.283170 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:21.283127 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0c4868b2-22cd-41c5-8e7d-d7dffacf1973-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-4c766\" (UID: \"0c4868b2-22cd-41c5-8e7d-d7dffacf1973\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4c766" Nov 03 01:54:21.283170 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:21.283154 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0c4868b2-22cd-41c5-8e7d-d7dffacf1973-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-4c766\" (UID: \"0c4868b2-22cd-41c5-8e7d-d7dffacf1973\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4c766" Nov 03 01:54:21.283309 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:21.283180 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brb8q\" (UniqueName: \"kubernetes.io/projected/0c4868b2-22cd-41c5-8e7d-d7dffacf1973-kube-api-access-brb8q\") pod \"istiod-openshift-gateway-7876bfcb74-4c766\" (UID: \"0c4868b2-22cd-41c5-8e7d-d7dffacf1973\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4c766" Nov 03 01:54:21.283855 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:21.283828 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0c4868b2-22cd-41c5-8e7d-d7dffacf1973-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-4c766\" (UID: \"0c4868b2-22cd-41c5-8e7d-d7dffacf1973\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4c766" Nov 03 01:54:21.285529 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:21.285501 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c4868b2-22cd-41c5-8e7d-d7dffacf1973-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-4c766\" (UID: \"0c4868b2-22cd-41c5-8e7d-d7dffacf1973\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4c766" Nov 03 01:54:21.285529 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:21.285520 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0c4868b2-22cd-41c5-8e7d-d7dffacf1973-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-4c766\" (UID: \"0c4868b2-22cd-41c5-8e7d-d7dffacf1973\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4c766" Nov 03 01:54:21.285717 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:21.285632 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0c4868b2-22cd-41c5-8e7d-d7dffacf1973-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-4c766\" (UID: \"0c4868b2-22cd-41c5-8e7d-d7dffacf1973\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4c766" Nov 03 01:54:21.285767 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:21.285711 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0c4868b2-22cd-41c5-8e7d-d7dffacf1973-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-4c766\" (UID: \"0c4868b2-22cd-41c5-8e7d-d7dffacf1973\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4c766" Nov 03 01:54:21.288886 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:21.288863 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxmtxt"] Nov 03 01:54:21.295855 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:54:21.295829 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda938e094_9132_4ac7_856f_a7df518006b3.slice/crio-8285300ef0d5bff23067eadcda017ba4933a69bdcfaa5eeea5bdff1b1b6108b1 WatchSource:0}: Error finding container 8285300ef0d5bff23067eadcda017ba4933a69bdcfaa5eeea5bdff1b1b6108b1: Status 404 returned error can't find the container with id 8285300ef0d5bff23067eadcda017ba4933a69bdcfaa5eeea5bdff1b1b6108b1 Nov 03 01:54:21.297879 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:21.297858 5605 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 01:54:21.336525 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:21.336495 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brb8q\" (UniqueName: \"kubernetes.io/projected/0c4868b2-22cd-41c5-8e7d-d7dffacf1973-kube-api-access-brb8q\") pod \"istiod-openshift-gateway-7876bfcb74-4c766\" (UID: \"0c4868b2-22cd-41c5-8e7d-d7dffacf1973\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4c766" Nov 03 01:54:21.336869 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:21.336848 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c4868b2-22cd-41c5-8e7d-d7dffacf1973-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-4c766\" (UID: \"0c4868b2-22cd-41c5-8e7d-d7dffacf1973\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4c766" Nov 03 01:54:21.392829 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:21.392792 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4c766" Nov 03 01:54:21.401917 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:21.401889 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxmtxt" event={"ID":"a938e094-9132-4ac7-856f-a7df518006b3","Type":"ContainerStarted","Data":"8285300ef0d5bff23067eadcda017ba4933a69bdcfaa5eeea5bdff1b1b6108b1"} Nov 03 01:54:21.763764 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:21.763692 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-4c766"] Nov 03 01:54:21.770983 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:54:21.770944 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c4868b2_22cd_41c5_8e7d_d7dffacf1973.slice/crio-3d5ddf46404020c7e6e82f0d6cd51f03b8bb339b57e9223289c188c3a2a70db2 WatchSource:0}: Error finding container 3d5ddf46404020c7e6e82f0d6cd51f03b8bb339b57e9223289c188c3a2a70db2: Status 404 returned error can't find the container with id 3d5ddf46404020c7e6e82f0d6cd51f03b8bb339b57e9223289c188c3a2a70db2 Nov 03 01:54:22.406352 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:22.406313 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4c766" event={"ID":"0c4868b2-22cd-41c5-8e7d-d7dffacf1973","Type":"ContainerStarted","Data":"3d5ddf46404020c7e6e82f0d6cd51f03b8bb339b57e9223289c188c3a2a70db2"} Nov 03 01:54:23.917004 ip-10-0-140-140 kubenswrapper[5605]: E1103 01:54:23.916962 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 01:54:27.427977 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:27.427876 5605 generic.go:356] "Generic (PLEG): container finished" podID="a938e094-9132-4ac7-856f-a7df518006b3" containerID="107102b7f1848e8cd6285c5ad73e0f7145c39f02cea795baad4be5f8773bdc0e" exitCode=0 Nov 03 01:54:27.427977 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:27.427953 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxmtxt" event={"ID":"a938e094-9132-4ac7-856f-a7df518006b3","Type":"ContainerDied","Data":"107102b7f1848e8cd6285c5ad73e0f7145c39f02cea795baad4be5f8773bdc0e"} Nov 03 01:54:28.630977 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:28.630946 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:54:28.631253 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:28.631016 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:54:29.437080 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:29.437044 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4c766" event={"ID":"0c4868b2-22cd-41c5-8e7d-d7dffacf1973","Type":"ContainerStarted","Data":"50e10272ec6a0643166e5bf551cc84efc9e56a276f53a18e12acc6c3f53a8d5a"} Nov 03 01:54:29.437293 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:29.437257 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4c766" Nov 03 01:54:29.438845 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:29.438820 5605 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-4c766 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:54:29.438928 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:29.438876 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4c766" podUID="0c4868b2-22cd-41c5-8e7d-d7dffacf1973" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:54:29.629548 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:29.629484 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4c766" podStartSLOduration=1.77225743 podStartE2EDuration="8.629470917s" totalImagesPullingTime="6.857213487s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:54:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:54:29.591396166 +0000 UTC m=+146.850594319" watchObservedRunningTime="2026-11-03 01:54:29.629470917 +0000 UTC m=+146.888669064" Nov 03 01:54:30.441298 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:30.441267 5605 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-4c766 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:54:30.441695 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:30.441317 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4c766" podUID="0c4868b2-22cd-41c5-8e7d-d7dffacf1973" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:54:32.450633 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:32.450599 5605 generic.go:356] "Generic (PLEG): container finished" podID="a938e094-9132-4ac7-856f-a7df518006b3" containerID="f65e86a84158a9b4afdc6feada84c4cfa78792589eae7b7afdb70068eaa32c6f" exitCode=0 Nov 03 01:54:32.451026 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:32.450678 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxmtxt" event={"ID":"a938e094-9132-4ac7-856f-a7df518006b3","Type":"ContainerDied","Data":"f65e86a84158a9b4afdc6feada84c4cfa78792589eae7b7afdb70068eaa32c6f"} Nov 03 01:54:33.441779 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:33.441747 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4c766" Nov 03 01:54:39.477500 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:39.477409 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxmtxt" event={"ID":"a938e094-9132-4ac7-856f-a7df518006b3","Type":"ContainerStarted","Data":"2549b597b5da2e9b82a8a7687c82cb13ac628d354417365d02134cc27304ce05"} Nov 03 01:54:39.584604 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:39.584552 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxmtxt" podStartSLOduration=2.690395971 podStartE2EDuration="19.584535593s" totalImagesPullingTime="16.894139622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:54:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:54:39.583625173 +0000 UTC m=+156.842823328" watchObservedRunningTime="2026-11-03 01:54:39.584535593 +0000 UTC m=+156.843733788" Nov 03 01:54:40.483443 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:40.483408 5605 generic.go:356] "Generic (PLEG): container finished" podID="a938e094-9132-4ac7-856f-a7df518006b3" containerID="2549b597b5da2e9b82a8a7687c82cb13ac628d354417365d02134cc27304ce05" exitCode=0 Nov 03 01:54:40.483868 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:40.483496 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxmtxt" event={"ID":"a938e094-9132-4ac7-856f-a7df518006b3","Type":"ContainerDied","Data":"2549b597b5da2e9b82a8a7687c82cb13ac628d354417365d02134cc27304ce05"} Nov 03 01:54:41.607263 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:41.607234 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxmtxt" Nov 03 01:54:41.756695 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:41.756613 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a938e094-9132-4ac7-856f-a7df518006b3-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a938e094-9132-4ac7-856f-a7df518006b3-bundle\") pod \"a938e094-9132-4ac7-856f-a7df518006b3\" (UID: \"a938e094-9132-4ac7-856f-a7df518006b3\") " Nov 03 01:54:41.756695 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:41.756655 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a938e094-9132-4ac7-856f-a7df518006b3-util\" (UniqueName: \"kubernetes.io/empty-dir/a938e094-9132-4ac7-856f-a7df518006b3-util\") pod \"a938e094-9132-4ac7-856f-a7df518006b3\" (UID: \"a938e094-9132-4ac7-856f-a7df518006b3\") " Nov 03 01:54:41.756891 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:41.756749 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a938e094-9132-4ac7-856f-a7df518006b3-kube-api-access-qqzgn\" (UniqueName: \"kubernetes.io/projected/a938e094-9132-4ac7-856f-a7df518006b3-kube-api-access-qqzgn\") pod \"a938e094-9132-4ac7-856f-a7df518006b3\" (UID: \"a938e094-9132-4ac7-856f-a7df518006b3\") " Nov 03 01:54:41.757253 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:41.757219 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a938e094-9132-4ac7-856f-a7df518006b3-bundle" pod "a938e094-9132-4ac7-856f-a7df518006b3" (UID: "a938e094-9132-4ac7-856f-a7df518006b3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:54:41.759121 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:41.759098 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a938e094-9132-4ac7-856f-a7df518006b3-kube-api-access-qqzgn" pod "a938e094-9132-4ac7-856f-a7df518006b3" (UID: "a938e094-9132-4ac7-856f-a7df518006b3"). InnerVolumeSpecName "kube-api-access-qqzgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:54:41.761338 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:41.761317 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a938e094-9132-4ac7-856f-a7df518006b3-util" pod "a938e094-9132-4ac7-856f-a7df518006b3" (UID: "a938e094-9132-4ac7-856f-a7df518006b3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:54:41.857543 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:41.857509 5605 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a938e094-9132-4ac7-856f-a7df518006b3-bundle\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:54:41.857543 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:41.857539 5605 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a938e094-9132-4ac7-856f-a7df518006b3-util\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:54:41.857543 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:41.857551 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqzgn\" (UniqueName: \"kubernetes.io/projected/a938e094-9132-4ac7-856f-a7df518006b3-kube-api-access-qqzgn\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:54:42.492190 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:42.492157 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxmtxt" event={"ID":"a938e094-9132-4ac7-856f-a7df518006b3","Type":"ContainerDied","Data":"8285300ef0d5bff23067eadcda017ba4933a69bdcfaa5eeea5bdff1b1b6108b1"} Nov 03 01:54:42.492346 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:42.492205 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8285300ef0d5bff23067eadcda017ba4933a69bdcfaa5eeea5bdff1b1b6108b1" Nov 03 01:54:42.492346 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:54:42.492168 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxmtxt" Nov 03 01:54:53.911386 ip-10-0-140-140 kubenswrapper[5605]: E1103 01:54:53.911333 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 01:55:11.784853 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:11.784772 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-qzhd5"] Nov 03 01:55:11.785223 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:11.785096 5605 cpu_manager.go:414] "containerMap: removing container" podUID="a938e094-9132-4ac7-856f-a7df518006b3" containerName="util" Nov 03 01:55:11.785223 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:11.785109 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a938e094-9132-4ac7-856f-a7df518006b3" containerName="util" Nov 03 01:55:11.785223 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:11.785126 5605 cpu_manager.go:414] "containerMap: removing container" podUID="a938e094-9132-4ac7-856f-a7df518006b3" containerName="extract" Nov 03 01:55:11.785223 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:11.785130 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a938e094-9132-4ac7-856f-a7df518006b3" containerName="extract" Nov 03 01:55:11.785223 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:11.785142 5605 cpu_manager.go:414] "containerMap: removing container" podUID="a938e094-9132-4ac7-856f-a7df518006b3" containerName="pull" Nov 03 01:55:11.785223 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:11.785146 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a938e094-9132-4ac7-856f-a7df518006b3" containerName="pull" Nov 03 01:55:11.785223 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:11.785201 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="a938e094-9132-4ac7-856f-a7df518006b3" containerName="extract" Nov 03 01:55:11.787504 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:11.787486 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-qzhd5" Nov 03 01:55:11.801942 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:11.801917 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/acaa1c0b-e83d-43b9-9939-57a43a070719-webhook-cert\") pod \"opendatahub-operator-controller-manager-864bf9f5c8-qzhd5\" (UID: \"acaa1c0b-e83d-43b9-9939-57a43a070719\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-qzhd5" Nov 03 01:55:11.802071 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:11.801951 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c9xm\" (UniqueName: \"kubernetes.io/projected/acaa1c0b-e83d-43b9-9939-57a43a070719-kube-api-access-7c9xm\") pod \"opendatahub-operator-controller-manager-864bf9f5c8-qzhd5\" (UID: \"acaa1c0b-e83d-43b9-9939-57a43a070719\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-qzhd5" Nov 03 01:55:11.802071 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:11.801976 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/acaa1c0b-e83d-43b9-9939-57a43a070719-manifests\") pod \"opendatahub-operator-controller-manager-864bf9f5c8-qzhd5\" (UID: \"acaa1c0b-e83d-43b9-9939-57a43a070719\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-qzhd5" Nov 03 01:55:11.802187 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:11.802148 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/acaa1c0b-e83d-43b9-9939-57a43a070719-metrics-cert\") pod \"opendatahub-operator-controller-manager-864bf9f5c8-qzhd5\" (UID: \"acaa1c0b-e83d-43b9-9939-57a43a070719\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-qzhd5" Nov 03 01:55:11.802230 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:11.802187 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/acaa1c0b-e83d-43b9-9939-57a43a070719-tmp\") pod \"opendatahub-operator-controller-manager-864bf9f5c8-qzhd5\" (UID: \"acaa1c0b-e83d-43b9-9939-57a43a070719\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-qzhd5" Nov 03 01:55:11.802264 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:11.802248 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/acaa1c0b-e83d-43b9-9939-57a43a070719-apiservice-cert\") pod \"opendatahub-operator-controller-manager-864bf9f5c8-qzhd5\" (UID: \"acaa1c0b-e83d-43b9-9939-57a43a070719\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-qzhd5" Nov 03 01:55:11.831507 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:11.831482 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-qzhd5"] Nov 03 01:55:11.869510 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:11.869477 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 01:55:11.869677 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:11.869508 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 01:55:11.869751 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:11.869733 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 01:55:11.869789 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:11.869742 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-hwpp2\"" Nov 03 01:55:11.869829 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:11.869786 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 01:55:11.869872 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:11.869861 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 01:55:11.902652 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:11.902615 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/acaa1c0b-e83d-43b9-9939-57a43a070719-metrics-cert\") pod \"opendatahub-operator-controller-manager-864bf9f5c8-qzhd5\" (UID: \"acaa1c0b-e83d-43b9-9939-57a43a070719\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-qzhd5" Nov 03 01:55:11.902815 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:11.902656 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/acaa1c0b-e83d-43b9-9939-57a43a070719-tmp\") pod \"opendatahub-operator-controller-manager-864bf9f5c8-qzhd5\" (UID: \"acaa1c0b-e83d-43b9-9939-57a43a070719\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-qzhd5" Nov 03 01:55:11.902815 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:11.902692 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/acaa1c0b-e83d-43b9-9939-57a43a070719-apiservice-cert\") pod \"opendatahub-operator-controller-manager-864bf9f5c8-qzhd5\" (UID: \"acaa1c0b-e83d-43b9-9939-57a43a070719\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-qzhd5" Nov 03 01:55:11.902815 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:11.902752 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/acaa1c0b-e83d-43b9-9939-57a43a070719-webhook-cert\") pod \"opendatahub-operator-controller-manager-864bf9f5c8-qzhd5\" (UID: \"acaa1c0b-e83d-43b9-9939-57a43a070719\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-qzhd5" Nov 03 01:55:11.902815 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:11.902784 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c9xm\" (UniqueName: \"kubernetes.io/projected/acaa1c0b-e83d-43b9-9939-57a43a070719-kube-api-access-7c9xm\") pod \"opendatahub-operator-controller-manager-864bf9f5c8-qzhd5\" (UID: \"acaa1c0b-e83d-43b9-9939-57a43a070719\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-qzhd5" Nov 03 01:55:11.903010 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:11.902818 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/acaa1c0b-e83d-43b9-9939-57a43a070719-manifests\") pod \"opendatahub-operator-controller-manager-864bf9f5c8-qzhd5\" (UID: \"acaa1c0b-e83d-43b9-9939-57a43a070719\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-qzhd5" Nov 03 01:55:11.903165 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:11.903141 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/acaa1c0b-e83d-43b9-9939-57a43a070719-manifests\") pod \"opendatahub-operator-controller-manager-864bf9f5c8-qzhd5\" (UID: \"acaa1c0b-e83d-43b9-9939-57a43a070719\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-qzhd5" Nov 03 01:55:11.905191 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:11.905139 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/acaa1c0b-e83d-43b9-9939-57a43a070719-tmp\") pod \"opendatahub-operator-controller-manager-864bf9f5c8-qzhd5\" (UID: \"acaa1c0b-e83d-43b9-9939-57a43a070719\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-qzhd5" Nov 03 01:55:11.905335 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:11.905319 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/acaa1c0b-e83d-43b9-9939-57a43a070719-webhook-cert\") pod \"opendatahub-operator-controller-manager-864bf9f5c8-qzhd5\" (UID: \"acaa1c0b-e83d-43b9-9939-57a43a070719\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-qzhd5" Nov 03 01:55:11.905653 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:11.905632 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/acaa1c0b-e83d-43b9-9939-57a43a070719-metrics-cert\") pod \"opendatahub-operator-controller-manager-864bf9f5c8-qzhd5\" (UID: \"acaa1c0b-e83d-43b9-9939-57a43a070719\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-qzhd5" Nov 03 01:55:11.908369 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:11.908337 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/acaa1c0b-e83d-43b9-9939-57a43a070719-apiservice-cert\") pod \"opendatahub-operator-controller-manager-864bf9f5c8-qzhd5\" (UID: \"acaa1c0b-e83d-43b9-9939-57a43a070719\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-qzhd5" Nov 03 01:55:11.962095 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:11.962060 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c9xm\" (UniqueName: \"kubernetes.io/projected/acaa1c0b-e83d-43b9-9939-57a43a070719-kube-api-access-7c9xm\") pod \"opendatahub-operator-controller-manager-864bf9f5c8-qzhd5\" (UID: \"acaa1c0b-e83d-43b9-9939-57a43a070719\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-qzhd5" Nov 03 01:55:12.099955 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:12.099921 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-qzhd5" Nov 03 01:55:12.581192 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:12.581157 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-qzhd5"] Nov 03 01:55:12.588457 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:55:12.588421 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacaa1c0b_e83d_43b9_9939_57a43a070719.slice/crio-64b748fc90d14c24d6abccf35bee71b2de666f2886b3e4436fdb273ef150a9af WatchSource:0}: Error finding container 64b748fc90d14c24d6abccf35bee71b2de666f2886b3e4436fdb273ef150a9af: Status 404 returned error can't find the container with id 64b748fc90d14c24d6abccf35bee71b2de666f2886b3e4436fdb273ef150a9af Nov 03 01:55:12.594533 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:12.594506 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-qzhd5" event={"ID":"acaa1c0b-e83d-43b9-9939-57a43a070719","Type":"ContainerStarted","Data":"64b748fc90d14c24d6abccf35bee71b2de666f2886b3e4436fdb273ef150a9af"} Nov 03 01:55:17.617153 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:17.617120 5605 generic.go:356] "Generic (PLEG): container finished" podID="acaa1c0b-e83d-43b9-9939-57a43a070719" containerID="b6fdc9dd7c0ec57a4255c1ac65a9438130df6a3ceb25e0b4707f4203c5a2cddc" exitCode=0 Nov 03 01:55:17.617574 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:17.617194 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-qzhd5" event={"ID":"acaa1c0b-e83d-43b9-9939-57a43a070719","Type":"ContainerDied","Data":"b6fdc9dd7c0ec57a4255c1ac65a9438130df6a3ceb25e0b4707f4203c5a2cddc"} Nov 03 01:55:18.623778 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:18.623689 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-qzhd5" event={"ID":"acaa1c0b-e83d-43b9-9939-57a43a070719","Type":"ContainerStarted","Data":"d1ff4f755d578969a6105a6a49317f9afe64487e11a627ae7d39c054c46b766e"} Nov 03 01:55:18.624260 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:18.623852 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-qzhd5" Nov 03 01:55:18.837826 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:18.837766 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-qzhd5" podStartSLOduration=2.412443503 podStartE2EDuration="7.837746298s" totalImagesPullingTime="4.980390239s" totalInitContainerRuntime="444.912556ms" isStatefulPod=false podCreationTimestamp="2026-11-03 01:55:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:55:18.790898599 +0000 UTC m=+196.050096754" watchObservedRunningTime="2026-11-03 01:55:18.837746298 +0000 UTC m=+196.096944453" Nov 03 01:55:23.910701 ip-10-0-140-140 kubenswrapper[5605]: E1103 01:55:23.910660 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 01:55:29.629518 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:29.629485 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-qzhd5" Nov 03 01:55:37.721573 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:37.721544 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-dj2vp"] Nov 03 01:55:37.727682 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:37.727657 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-dj2vp" Nov 03 01:55:37.781200 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:37.781168 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 03 01:55:37.818908 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:37.818869 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfwd6\" (UniqueName: \"kubernetes.io/projected/551a02c9-0404-4555-b57d-21804cb85404-kube-api-access-rfwd6\") pod \"cert-manager-cainjector-5c57bbd4fc-dj2vp\" (UID: \"551a02c9-0404-4555-b57d-21804cb85404\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-dj2vp" Nov 03 01:55:37.819078 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:37.818933 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/551a02c9-0404-4555-b57d-21804cb85404-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-dj2vp\" (UID: \"551a02c9-0404-4555-b57d-21804cb85404\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-dj2vp" Nov 03 01:55:37.825752 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:37.825724 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 03 01:55:37.825979 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:37.825962 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-rpvck\"" Nov 03 01:55:37.826719 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:37.826699 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-dj2vp"] Nov 03 01:55:37.920307 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:37.920273 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfwd6\" (UniqueName: \"kubernetes.io/projected/551a02c9-0404-4555-b57d-21804cb85404-kube-api-access-rfwd6\") pod \"cert-manager-cainjector-5c57bbd4fc-dj2vp\" (UID: \"551a02c9-0404-4555-b57d-21804cb85404\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-dj2vp" Nov 03 01:55:37.920500 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:37.920397 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/551a02c9-0404-4555-b57d-21804cb85404-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-dj2vp\" (UID: \"551a02c9-0404-4555-b57d-21804cb85404\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-dj2vp" Nov 03 01:55:37.984477 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:37.984399 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfwd6\" (UniqueName: \"kubernetes.io/projected/551a02c9-0404-4555-b57d-21804cb85404-kube-api-access-rfwd6\") pod \"cert-manager-cainjector-5c57bbd4fc-dj2vp\" (UID: \"551a02c9-0404-4555-b57d-21804cb85404\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-dj2vp" Nov 03 01:55:37.985285 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:37.985260 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/551a02c9-0404-4555-b57d-21804cb85404-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-dj2vp\" (UID: \"551a02c9-0404-4555-b57d-21804cb85404\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-dj2vp" Nov 03 01:55:38.052214 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:38.052183 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-dj2vp" Nov 03 01:55:38.417649 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:38.417614 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-dj2vp"] Nov 03 01:55:38.425647 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:55:38.425621 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod551a02c9_0404_4555_b57d_21804cb85404.slice/crio-6c46bf4a80a339d406b1591f77bb86af332b8c7e7b93d0a9dfb72c3837053515 WatchSource:0}: Error finding container 6c46bf4a80a339d406b1591f77bb86af332b8c7e7b93d0a9dfb72c3837053515: Status 404 returned error can't find the container with id 6c46bf4a80a339d406b1591f77bb86af332b8c7e7b93d0a9dfb72c3837053515 Nov 03 01:55:38.695248 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:38.695168 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-dj2vp" event={"ID":"551a02c9-0404-4555-b57d-21804cb85404","Type":"ContainerStarted","Data":"6c46bf4a80a339d406b1591f77bb86af332b8c7e7b93d0a9dfb72c3837053515"} Nov 03 01:55:41.791866 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:41.791826 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-tfr75"] Nov 03 01:55:41.795392 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:41.795351 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-tfr75" Nov 03 01:55:41.890957 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:41.890926 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-bbnqz\"" Nov 03 01:55:41.891823 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:41.891789 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-tfr75"] Nov 03 01:55:41.959141 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:41.959102 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/df415c67-faa0-4116-a313-c0b0aed62e65-bound-sa-token\") pod \"cert-manager-597b95cb84-tfr75\" (UID: \"df415c67-faa0-4116-a313-c0b0aed62e65\") " pod="cert-manager/cert-manager-597b95cb84-tfr75" Nov 03 01:55:41.959326 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:41.959157 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg7cr\" (UniqueName: \"kubernetes.io/projected/df415c67-faa0-4116-a313-c0b0aed62e65-kube-api-access-sg7cr\") pod \"cert-manager-597b95cb84-tfr75\" (UID: \"df415c67-faa0-4116-a313-c0b0aed62e65\") " pod="cert-manager/cert-manager-597b95cb84-tfr75" Nov 03 01:55:42.060384 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:42.060329 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/df415c67-faa0-4116-a313-c0b0aed62e65-bound-sa-token\") pod \"cert-manager-597b95cb84-tfr75\" (UID: \"df415c67-faa0-4116-a313-c0b0aed62e65\") " pod="cert-manager/cert-manager-597b95cb84-tfr75" Nov 03 01:55:42.060605 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:42.060404 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg7cr\" (UniqueName: \"kubernetes.io/projected/df415c67-faa0-4116-a313-c0b0aed62e65-kube-api-access-sg7cr\") pod \"cert-manager-597b95cb84-tfr75\" (UID: \"df415c67-faa0-4116-a313-c0b0aed62e65\") " pod="cert-manager/cert-manager-597b95cb84-tfr75" Nov 03 01:55:42.114039 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:42.114003 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/df415c67-faa0-4116-a313-c0b0aed62e65-bound-sa-token\") pod \"cert-manager-597b95cb84-tfr75\" (UID: \"df415c67-faa0-4116-a313-c0b0aed62e65\") " pod="cert-manager/cert-manager-597b95cb84-tfr75" Nov 03 01:55:42.114344 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:42.114322 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg7cr\" (UniqueName: \"kubernetes.io/projected/df415c67-faa0-4116-a313-c0b0aed62e65-kube-api-access-sg7cr\") pod \"cert-manager-597b95cb84-tfr75\" (UID: \"df415c67-faa0-4116-a313-c0b0aed62e65\") " pod="cert-manager/cert-manager-597b95cb84-tfr75" Nov 03 01:55:42.409012 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:42.408926 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-tfr75" Nov 03 01:55:42.713077 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:42.712980 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-dj2vp" event={"ID":"551a02c9-0404-4555-b57d-21804cb85404","Type":"ContainerStarted","Data":"e3ceb61b96e6e6198573341db57560e5f9a118b994ba06dbdab219309116201d"} Nov 03 01:55:42.785313 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:42.785280 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-tfr75"] Nov 03 01:55:42.792281 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:55:42.792248 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddf415c67_faa0_4116_a313_c0b0aed62e65.slice/crio-30a20bdf10345f020336eb6ce04b42872d90e306ad4e4650ec287d15bba84351 WatchSource:0}: Error finding container 30a20bdf10345f020336eb6ce04b42872d90e306ad4e4650ec287d15bba84351: Status 404 returned error can't find the container with id 30a20bdf10345f020336eb6ce04b42872d90e306ad4e4650ec287d15bba84351 Nov 03 01:55:42.819395 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:42.819316 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-dj2vp" podStartSLOduration=1.816377389 podStartE2EDuration="5.819301143s" totalImagesPullingTime="4.002923754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:55:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:55:42.818527107 +0000 UTC m=+220.077725259" watchObservedRunningTime="2026-11-03 01:55:42.819301143 +0000 UTC m=+220.078499296" Nov 03 01:55:43.719240 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:43.719187 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-tfr75" event={"ID":"df415c67-faa0-4116-a313-c0b0aed62e65","Type":"ContainerStarted","Data":"eb7d87436cf62b0f3af870db9db4ca02abcd710bd416003a0d211a23d44918c3"} Nov 03 01:55:43.719508 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:43.719255 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-tfr75" event={"ID":"df415c67-faa0-4116-a313-c0b0aed62e65","Type":"ContainerStarted","Data":"30a20bdf10345f020336eb6ce04b42872d90e306ad4e4650ec287d15bba84351"} Nov 03 01:55:43.827374 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:43.826862 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-tfr75" podStartSLOduration=2.8268439709999997 podStartE2EDuration="2.826843971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:55:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:55:43.825025578 +0000 UTC m=+221.084223732" watchObservedRunningTime="2026-11-03 01:55:43.826843971 +0000 UTC m=+221.086042125" Nov 03 01:55:44.926925 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:44.926890 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6dc8f585b4-sb2dx"] Nov 03 01:55:44.930148 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:44.930127 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6dc8f585b4-sb2dx" Nov 03 01:55:44.979216 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:44.979181 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 03 01:55:44.979216 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:44.979213 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 03 01:55:45.025941 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:45.025910 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-l7n64\"" Nov 03 01:55:45.026774 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:45.026755 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6dc8f585b4-sb2dx"] Nov 03 01:55:45.090414 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:45.090378 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e0e5200-c6f3-4e09-aa80-dfd996cf38d1-tls-certs\") pod \"kube-auth-proxy-6dc8f585b4-sb2dx\" (UID: \"8e0e5200-c6f3-4e09-aa80-dfd996cf38d1\") " pod="openshift-ingress/kube-auth-proxy-6dc8f585b4-sb2dx" Nov 03 01:55:45.090414 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:45.090415 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e0e5200-c6f3-4e09-aa80-dfd996cf38d1-tmp\") pod \"kube-auth-proxy-6dc8f585b4-sb2dx\" (UID: \"8e0e5200-c6f3-4e09-aa80-dfd996cf38d1\") " pod="openshift-ingress/kube-auth-proxy-6dc8f585b4-sb2dx" Nov 03 01:55:45.090663 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:45.090542 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fhmx\" (UniqueName: \"kubernetes.io/projected/8e0e5200-c6f3-4e09-aa80-dfd996cf38d1-kube-api-access-7fhmx\") pod \"kube-auth-proxy-6dc8f585b4-sb2dx\" (UID: \"8e0e5200-c6f3-4e09-aa80-dfd996cf38d1\") " pod="openshift-ingress/kube-auth-proxy-6dc8f585b4-sb2dx" Nov 03 01:55:45.191445 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:45.191347 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e0e5200-c6f3-4e09-aa80-dfd996cf38d1-tls-certs\") pod \"kube-auth-proxy-6dc8f585b4-sb2dx\" (UID: \"8e0e5200-c6f3-4e09-aa80-dfd996cf38d1\") " pod="openshift-ingress/kube-auth-proxy-6dc8f585b4-sb2dx" Nov 03 01:55:45.191445 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:45.191407 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e0e5200-c6f3-4e09-aa80-dfd996cf38d1-tmp\") pod \"kube-auth-proxy-6dc8f585b4-sb2dx\" (UID: \"8e0e5200-c6f3-4e09-aa80-dfd996cf38d1\") " pod="openshift-ingress/kube-auth-proxy-6dc8f585b4-sb2dx" Nov 03 01:55:45.191674 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:45.191456 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fhmx\" (UniqueName: \"kubernetes.io/projected/8e0e5200-c6f3-4e09-aa80-dfd996cf38d1-kube-api-access-7fhmx\") pod \"kube-auth-proxy-6dc8f585b4-sb2dx\" (UID: \"8e0e5200-c6f3-4e09-aa80-dfd996cf38d1\") " pod="openshift-ingress/kube-auth-proxy-6dc8f585b4-sb2dx" Nov 03 01:55:45.193673 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:45.193635 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e0e5200-c6f3-4e09-aa80-dfd996cf38d1-tmp\") pod \"kube-auth-proxy-6dc8f585b4-sb2dx\" (UID: \"8e0e5200-c6f3-4e09-aa80-dfd996cf38d1\") " pod="openshift-ingress/kube-auth-proxy-6dc8f585b4-sb2dx" Nov 03 01:55:45.194013 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:45.193992 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e0e5200-c6f3-4e09-aa80-dfd996cf38d1-tls-certs\") pod \"kube-auth-proxy-6dc8f585b4-sb2dx\" (UID: \"8e0e5200-c6f3-4e09-aa80-dfd996cf38d1\") " pod="openshift-ingress/kube-auth-proxy-6dc8f585b4-sb2dx" Nov 03 01:55:45.248111 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:45.248072 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fhmx\" (UniqueName: \"kubernetes.io/projected/8e0e5200-c6f3-4e09-aa80-dfd996cf38d1-kube-api-access-7fhmx\") pod \"kube-auth-proxy-6dc8f585b4-sb2dx\" (UID: \"8e0e5200-c6f3-4e09-aa80-dfd996cf38d1\") " pod="openshift-ingress/kube-auth-proxy-6dc8f585b4-sb2dx" Nov 03 01:55:45.541941 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:45.541854 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6dc8f585b4-sb2dx" Nov 03 01:55:45.908595 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:45.908560 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6dc8f585b4-sb2dx"] Nov 03 01:55:45.915271 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:55:45.915243 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e0e5200_c6f3_4e09_aa80_dfd996cf38d1.slice/crio-23bcfdf79af1679f06cdff1dd13803223d26f7606a603c3fc9580df0d4e1c3d3 WatchSource:0}: Error finding container 23bcfdf79af1679f06cdff1dd13803223d26f7606a603c3fc9580df0d4e1c3d3: Status 404 returned error can't find the container with id 23bcfdf79af1679f06cdff1dd13803223d26f7606a603c3fc9580df0d4e1c3d3 Nov 03 01:55:46.733485 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:46.733451 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6dc8f585b4-sb2dx" event={"ID":"8e0e5200-c6f3-4e09-aa80-dfd996cf38d1","Type":"ContainerStarted","Data":"23bcfdf79af1679f06cdff1dd13803223d26f7606a603c3fc9580df0d4e1c3d3"} Nov 03 01:55:48.295495 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:48.295455 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 01:55:48.306747 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:48.306715 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-skgg9" Nov 03 01:55:48.357592 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:48.357558 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 03 01:55:48.377942 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:48.377913 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 01:55:48.377942 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:48.377924 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 01:55:48.378166 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:48.377940 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-bck82\"" Nov 03 01:55:48.404667 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:48.404634 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 01:55:48.421135 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:48.421095 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ce015707-8aae-4215-bb4b-6ebb7bef21c5-config-file\") pod \"limitador-limitador-7c9b87d574-skgg9\" (UID: \"ce015707-8aae-4215-bb4b-6ebb7bef21c5\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-skgg9" Nov 03 01:55:48.421316 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:48.421155 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slvwg\" (UniqueName: \"kubernetes.io/projected/ce015707-8aae-4215-bb4b-6ebb7bef21c5-kube-api-access-slvwg\") pod \"limitador-limitador-7c9b87d574-skgg9\" (UID: \"ce015707-8aae-4215-bb4b-6ebb7bef21c5\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-skgg9" Nov 03 01:55:48.522729 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:48.522570 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ce015707-8aae-4215-bb4b-6ebb7bef21c5-config-file\") pod \"limitador-limitador-7c9b87d574-skgg9\" (UID: \"ce015707-8aae-4215-bb4b-6ebb7bef21c5\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-skgg9" Nov 03 01:55:48.522729 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:48.522632 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slvwg\" (UniqueName: \"kubernetes.io/projected/ce015707-8aae-4215-bb4b-6ebb7bef21c5-kube-api-access-slvwg\") pod \"limitador-limitador-7c9b87d574-skgg9\" (UID: \"ce015707-8aae-4215-bb4b-6ebb7bef21c5\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-skgg9" Nov 03 01:55:48.523224 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:48.523200 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ce015707-8aae-4215-bb4b-6ebb7bef21c5-config-file\") pod \"limitador-limitador-7c9b87d574-skgg9\" (UID: \"ce015707-8aae-4215-bb4b-6ebb7bef21c5\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-skgg9" Nov 03 01:55:48.575860 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:48.575831 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slvwg\" (UniqueName: \"kubernetes.io/projected/ce015707-8aae-4215-bb4b-6ebb7bef21c5-kube-api-access-slvwg\") pod \"limitador-limitador-7c9b87d574-skgg9\" (UID: \"ce015707-8aae-4215-bb4b-6ebb7bef21c5\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-skgg9" Nov 03 01:55:48.597734 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:48.597699 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 01:55:48.621847 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:48.621807 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-skgg9" Nov 03 01:55:49.057551 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:49.056906 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 01:55:49.754183 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:49.754142 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-skgg9" event={"ID":"ce015707-8aae-4215-bb4b-6ebb7bef21c5","Type":"ContainerStarted","Data":"cb3a125a0dfc450081ed057690f7a8e6bc89c4dfb05115ed84c3e63d5a152eb8"} Nov 03 01:55:49.755628 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:49.755604 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6dc8f585b4-sb2dx" event={"ID":"8e0e5200-c6f3-4e09-aa80-dfd996cf38d1","Type":"ContainerStarted","Data":"e3e34f6f32a5d820be75215f803de3fa13b47c4d2068396b6905dd7b4a7d8654"} Nov 03 01:55:49.863100 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:49.862933 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6dc8f585b4-sb2dx" podStartSLOduration=2.184560635 podStartE2EDuration="5.862915325s" totalImagesPullingTime="3.67835469s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:55:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:55:49.862338877 +0000 UTC m=+227.121537030" watchObservedRunningTime="2026-11-03 01:55:49.862915325 +0000 UTC m=+227.122113479" Nov 03 01:55:53.778092 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:53.778050 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-skgg9" event={"ID":"ce015707-8aae-4215-bb4b-6ebb7bef21c5","Type":"ContainerStarted","Data":"eaf0ae27488b43c6597c5b488cc63691f9e55fb3157bdba806d1b18714796b83"} Nov 03 01:55:53.778813 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:53.778184 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-skgg9" Nov 03 01:55:53.896891 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:55:53.896820 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-skgg9" podStartSLOduration=2.052236424 podStartE2EDuration="5.896803539s" totalImagesPullingTime="3.844567115s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:55:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:55:53.896506857 +0000 UTC m=+231.155705010" watchObservedRunningTime="2026-11-03 01:55:53.896803539 +0000 UTC m=+231.156001692" Nov 03 01:55:53.912839 ip-10-0-140-140 kubenswrapper[5605]: E1103 01:55:53.912799 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 01:56:04.782940 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:56:04.782907 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-skgg9" Nov 03 01:56:23.912016 ip-10-0-140-140 kubenswrapper[5605]: E1103 01:56:23.911974 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 01:56:53.910894 ip-10-0-140-140 kubenswrapper[5605]: E1103 01:56:53.910816 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 01:56:57.030604 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:56:57.030566 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-plhfr"] Nov 03 01:56:57.033850 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:56:57.033832 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-plhfr" Nov 03 01:56:57.104997 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:56:57.104958 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/462c5bfd-f256-4620-87b3-20ed3c1fd9e4-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-plhfr\" (UID: \"462c5bfd-f256-4620-87b3-20ed3c1fd9e4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-plhfr" Nov 03 01:56:57.104997 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:56:57.104999 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb8tt\" (UniqueName: \"kubernetes.io/projected/462c5bfd-f256-4620-87b3-20ed3c1fd9e4-kube-api-access-zb8tt\") pod \"istiod-openshift-gateway-7ddd9cfdc8-plhfr\" (UID: \"462c5bfd-f256-4620-87b3-20ed3c1fd9e4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-plhfr" Nov 03 01:56:57.105218 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:56:57.105099 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/462c5bfd-f256-4620-87b3-20ed3c1fd9e4-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-plhfr\" (UID: \"462c5bfd-f256-4620-87b3-20ed3c1fd9e4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-plhfr" Nov 03 01:56:57.105218 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:56:57.105122 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/462c5bfd-f256-4620-87b3-20ed3c1fd9e4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-plhfr\" (UID: \"462c5bfd-f256-4620-87b3-20ed3c1fd9e4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-plhfr" Nov 03 01:56:57.105218 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:56:57.105195 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/462c5bfd-f256-4620-87b3-20ed3c1fd9e4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-plhfr\" (UID: \"462c5bfd-f256-4620-87b3-20ed3c1fd9e4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-plhfr" Nov 03 01:56:57.105343 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:56:57.105241 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/462c5bfd-f256-4620-87b3-20ed3c1fd9e4-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-plhfr\" (UID: \"462c5bfd-f256-4620-87b3-20ed3c1fd9e4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-plhfr" Nov 03 01:56:57.105343 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:56:57.105256 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/462c5bfd-f256-4620-87b3-20ed3c1fd9e4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-plhfr\" (UID: \"462c5bfd-f256-4620-87b3-20ed3c1fd9e4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-plhfr" Nov 03 01:56:57.130108 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:56:57.130070 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-plhfr"] Nov 03 01:56:57.206554 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:56:57.206519 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/462c5bfd-f256-4620-87b3-20ed3c1fd9e4-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-plhfr\" (UID: \"462c5bfd-f256-4620-87b3-20ed3c1fd9e4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-plhfr" Nov 03 01:56:57.206554 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:56:57.206553 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb8tt\" (UniqueName: \"kubernetes.io/projected/462c5bfd-f256-4620-87b3-20ed3c1fd9e4-kube-api-access-zb8tt\") pod \"istiod-openshift-gateway-7ddd9cfdc8-plhfr\" (UID: \"462c5bfd-f256-4620-87b3-20ed3c1fd9e4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-plhfr" Nov 03 01:56:57.206816 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:56:57.206688 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/462c5bfd-f256-4620-87b3-20ed3c1fd9e4-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-plhfr\" (UID: \"462c5bfd-f256-4620-87b3-20ed3c1fd9e4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-plhfr" Nov 03 01:56:57.206816 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:56:57.206727 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/462c5bfd-f256-4620-87b3-20ed3c1fd9e4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-plhfr\" (UID: \"462c5bfd-f256-4620-87b3-20ed3c1fd9e4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-plhfr" Nov 03 01:56:57.206816 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:56:57.206763 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/462c5bfd-f256-4620-87b3-20ed3c1fd9e4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-plhfr\" (UID: \"462c5bfd-f256-4620-87b3-20ed3c1fd9e4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-plhfr" Nov 03 01:56:57.206963 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:56:57.206875 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/462c5bfd-f256-4620-87b3-20ed3c1fd9e4-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-plhfr\" (UID: \"462c5bfd-f256-4620-87b3-20ed3c1fd9e4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-plhfr" Nov 03 01:56:57.206963 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:56:57.206910 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/462c5bfd-f256-4620-87b3-20ed3c1fd9e4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-plhfr\" (UID: \"462c5bfd-f256-4620-87b3-20ed3c1fd9e4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-plhfr" Nov 03 01:56:57.207561 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:56:57.207506 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/462c5bfd-f256-4620-87b3-20ed3c1fd9e4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-plhfr\" (UID: \"462c5bfd-f256-4620-87b3-20ed3c1fd9e4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-plhfr" Nov 03 01:56:57.209059 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:56:57.209029 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/462c5bfd-f256-4620-87b3-20ed3c1fd9e4-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-plhfr\" (UID: \"462c5bfd-f256-4620-87b3-20ed3c1fd9e4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-plhfr" Nov 03 01:56:57.209163 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:56:57.209132 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/462c5bfd-f256-4620-87b3-20ed3c1fd9e4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-plhfr\" (UID: \"462c5bfd-f256-4620-87b3-20ed3c1fd9e4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-plhfr" Nov 03 01:56:57.209236 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:56:57.209220 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/462c5bfd-f256-4620-87b3-20ed3c1fd9e4-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-plhfr\" (UID: \"462c5bfd-f256-4620-87b3-20ed3c1fd9e4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-plhfr" Nov 03 01:56:57.209371 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:56:57.209335 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/462c5bfd-f256-4620-87b3-20ed3c1fd9e4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-plhfr\" (UID: \"462c5bfd-f256-4620-87b3-20ed3c1fd9e4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-plhfr" Nov 03 01:56:57.260785 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:56:57.260747 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/462c5bfd-f256-4620-87b3-20ed3c1fd9e4-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-plhfr\" (UID: \"462c5bfd-f256-4620-87b3-20ed3c1fd9e4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-plhfr" Nov 03 01:56:57.260927 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:56:57.260804 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb8tt\" (UniqueName: \"kubernetes.io/projected/462c5bfd-f256-4620-87b3-20ed3c1fd9e4-kube-api-access-zb8tt\") pod \"istiod-openshift-gateway-7ddd9cfdc8-plhfr\" (UID: \"462c5bfd-f256-4620-87b3-20ed3c1fd9e4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-plhfr" Nov 03 01:56:57.345715 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:56:57.345677 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-plhfr" Nov 03 01:56:57.762711 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:56:57.762627 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-plhfr"] Nov 03 01:56:57.768878 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:56:57.768849 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod462c5bfd_f256_4620_87b3_20ed3c1fd9e4.slice/crio-fe90232c414aeac8da3b46fbdeadde9397629b3d8a7d79ab8c3793cecdf92cc7 WatchSource:0}: Error finding container fe90232c414aeac8da3b46fbdeadde9397629b3d8a7d79ab8c3793cecdf92cc7: Status 404 returned error can't find the container with id fe90232c414aeac8da3b46fbdeadde9397629b3d8a7d79ab8c3793cecdf92cc7 Nov 03 01:56:57.770926 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:56:57.770892 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:56:57.771020 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:56:57.770968 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:56:58.030981 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:56:58.030881 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-plhfr" event={"ID":"462c5bfd-f256-4620-87b3-20ed3c1fd9e4","Type":"ContainerStarted","Data":"12831d5535f775f63c8058daa29e761266df8b3de540ebc9ae6805d8c76d855f"} Nov 03 01:56:58.030981 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:56:58.030935 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-plhfr" event={"ID":"462c5bfd-f256-4620-87b3-20ed3c1fd9e4","Type":"ContainerStarted","Data":"fe90232c414aeac8da3b46fbdeadde9397629b3d8a7d79ab8c3793cecdf92cc7"} Nov 03 01:56:58.031422 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:56:58.031152 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-plhfr" Nov 03 01:56:58.032695 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:56:58.032670 5605 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-plhfr container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:56:58.032809 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:56:58.032722 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-plhfr" podUID="462c5bfd-f256-4620-87b3-20ed3c1fd9e4" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:56:58.140733 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:56:58.140676 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-plhfr" podStartSLOduration=2.140658388 podStartE2EDuration="2.140658388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:56:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:56:58.140451696 +0000 UTC m=+295.399649852" watchObservedRunningTime="2026-11-03 01:56:58.140658388 +0000 UTC m=+295.399856542" Nov 03 01:56:59.037890 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:56:59.037860 5605 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-plhfr container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:56:59.038272 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:56:59.037917 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-plhfr" podUID="462c5bfd-f256-4620-87b3-20ed3c1fd9e4" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:57:02.038247 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:57:02.038213 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-plhfr" Nov 03 01:57:02.244336 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:57:02.244300 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-4c766"] Nov 03 01:57:02.244605 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:57:02.244581 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4c766" podUID="0c4868b2-22cd-41c5-8e7d-d7dffacf1973" containerName="discovery" containerID="cri-o://50e10272ec6a0643166e5bf551cc84efc9e56a276f53a18e12acc6c3f53a8d5a" gracePeriod=30 Nov 03 01:57:02.500855 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:57:02.500828 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4c766" Nov 03 01:57:02.558812 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:57:02.558727 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c4868b2-22cd-41c5-8e7d-d7dffacf1973-cacerts\" (UniqueName: \"kubernetes.io/secret/0c4868b2-22cd-41c5-8e7d-d7dffacf1973-cacerts\") pod \"0c4868b2-22cd-41c5-8e7d-d7dffacf1973\" (UID: \"0c4868b2-22cd-41c5-8e7d-d7dffacf1973\") " Nov 03 01:57:02.558812 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:57:02.558769 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c4868b2-22cd-41c5-8e7d-d7dffacf1973-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0c4868b2-22cd-41c5-8e7d-d7dffacf1973-istio-csr-ca-configmap\") pod \"0c4868b2-22cd-41c5-8e7d-d7dffacf1973\" (UID: \"0c4868b2-22cd-41c5-8e7d-d7dffacf1973\") " Nov 03 01:57:02.559023 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:57:02.558846 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c4868b2-22cd-41c5-8e7d-d7dffacf1973-kube-api-access-brb8q\" (UniqueName: \"kubernetes.io/projected/0c4868b2-22cd-41c5-8e7d-d7dffacf1973-kube-api-access-brb8q\") pod \"0c4868b2-22cd-41c5-8e7d-d7dffacf1973\" (UID: \"0c4868b2-22cd-41c5-8e7d-d7dffacf1973\") " Nov 03 01:57:02.559023 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:57:02.558873 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c4868b2-22cd-41c5-8e7d-d7dffacf1973-istio-token\" (UniqueName: \"kubernetes.io/projected/0c4868b2-22cd-41c5-8e7d-d7dffacf1973-istio-token\") pod \"0c4868b2-22cd-41c5-8e7d-d7dffacf1973\" (UID: \"0c4868b2-22cd-41c5-8e7d-d7dffacf1973\") " Nov 03 01:57:02.559023 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:57:02.558927 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c4868b2-22cd-41c5-8e7d-d7dffacf1973-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c4868b2-22cd-41c5-8e7d-d7dffacf1973-local-certs\") pod \"0c4868b2-22cd-41c5-8e7d-d7dffacf1973\" (UID: \"0c4868b2-22cd-41c5-8e7d-d7dffacf1973\") " Nov 03 01:57:02.559023 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:57:02.558963 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c4868b2-22cd-41c5-8e7d-d7dffacf1973-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0c4868b2-22cd-41c5-8e7d-d7dffacf1973-istio-csr-dns-cert\") pod \"0c4868b2-22cd-41c5-8e7d-d7dffacf1973\" (UID: \"0c4868b2-22cd-41c5-8e7d-d7dffacf1973\") " Nov 03 01:57:02.559023 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:57:02.559000 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c4868b2-22cd-41c5-8e7d-d7dffacf1973-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0c4868b2-22cd-41c5-8e7d-d7dffacf1973-istio-kubeconfig\") pod \"0c4868b2-22cd-41c5-8e7d-d7dffacf1973\" (UID: \"0c4868b2-22cd-41c5-8e7d-d7dffacf1973\") " Nov 03 01:57:02.559315 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:57:02.559285 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c4868b2-22cd-41c5-8e7d-d7dffacf1973-istio-csr-ca-configmap" pod "0c4868b2-22cd-41c5-8e7d-d7dffacf1973" (UID: "0c4868b2-22cd-41c5-8e7d-d7dffacf1973"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:57:02.561783 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:57:02.561746 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c4868b2-22cd-41c5-8e7d-d7dffacf1973-kube-api-access-brb8q" pod "0c4868b2-22cd-41c5-8e7d-d7dffacf1973" (UID: "0c4868b2-22cd-41c5-8e7d-d7dffacf1973"). InnerVolumeSpecName "kube-api-access-brb8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:57:02.561783 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:57:02.561763 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c4868b2-22cd-41c5-8e7d-d7dffacf1973-istio-csr-dns-cert" pod "0c4868b2-22cd-41c5-8e7d-d7dffacf1973" (UID: "0c4868b2-22cd-41c5-8e7d-d7dffacf1973"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:57:02.561932 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:57:02.561785 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c4868b2-22cd-41c5-8e7d-d7dffacf1973-istio-kubeconfig" pod "0c4868b2-22cd-41c5-8e7d-d7dffacf1973" (UID: "0c4868b2-22cd-41c5-8e7d-d7dffacf1973"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:57:02.561932 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:57:02.561814 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c4868b2-22cd-41c5-8e7d-d7dffacf1973-istio-token" pod "0c4868b2-22cd-41c5-8e7d-d7dffacf1973" (UID: "0c4868b2-22cd-41c5-8e7d-d7dffacf1973"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:57:02.561932 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:57:02.561876 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c4868b2-22cd-41c5-8e7d-d7dffacf1973-cacerts" pod "0c4868b2-22cd-41c5-8e7d-d7dffacf1973" (UID: "0c4868b2-22cd-41c5-8e7d-d7dffacf1973"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:57:02.561932 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:57:02.561890 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c4868b2-22cd-41c5-8e7d-d7dffacf1973-local-certs" pod "0c4868b2-22cd-41c5-8e7d-d7dffacf1973" (UID: "0c4868b2-22cd-41c5-8e7d-d7dffacf1973"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:57:02.659732 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:57:02.659701 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brb8q\" (UniqueName: \"kubernetes.io/projected/0c4868b2-22cd-41c5-8e7d-d7dffacf1973-kube-api-access-brb8q\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:57:02.659732 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:57:02.659732 5605 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c4868b2-22cd-41c5-8e7d-d7dffacf1973-istio-token\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:57:02.659924 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:57:02.659744 5605 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c4868b2-22cd-41c5-8e7d-d7dffacf1973-local-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:57:02.659924 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:57:02.659752 5605 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0c4868b2-22cd-41c5-8e7d-d7dffacf1973-istio-csr-dns-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:57:02.659924 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:57:02.659761 5605 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0c4868b2-22cd-41c5-8e7d-d7dffacf1973-istio-kubeconfig\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:57:02.659924 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:57:02.659771 5605 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0c4868b2-22cd-41c5-8e7d-d7dffacf1973-cacerts\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:57:02.659924 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:57:02.659779 5605 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0c4868b2-22cd-41c5-8e7d-d7dffacf1973-istio-csr-ca-configmap\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:57:03.053876 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:57:03.053842 5605 generic.go:356] "Generic (PLEG): container finished" podID="0c4868b2-22cd-41c5-8e7d-d7dffacf1973" containerID="50e10272ec6a0643166e5bf551cc84efc9e56a276f53a18e12acc6c3f53a8d5a" exitCode=0 Nov 03 01:57:03.054351 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:57:03.053894 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4c766" Nov 03 01:57:03.054351 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:57:03.053901 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4c766" event={"ID":"0c4868b2-22cd-41c5-8e7d-d7dffacf1973","Type":"ContainerDied","Data":"50e10272ec6a0643166e5bf551cc84efc9e56a276f53a18e12acc6c3f53a8d5a"} Nov 03 01:57:03.054351 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:57:03.053942 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4c766" event={"ID":"0c4868b2-22cd-41c5-8e7d-d7dffacf1973","Type":"ContainerDied","Data":"3d5ddf46404020c7e6e82f0d6cd51f03b8bb339b57e9223289c188c3a2a70db2"} Nov 03 01:57:03.054351 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:57:03.053960 5605 scope.go:122] "RemoveContainer" containerID="50e10272ec6a0643166e5bf551cc84efc9e56a276f53a18e12acc6c3f53a8d5a" Nov 03 01:57:03.062165 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:57:03.062149 5605 scope.go:122] "RemoveContainer" containerID="50e10272ec6a0643166e5bf551cc84efc9e56a276f53a18e12acc6c3f53a8d5a" Nov 03 01:57:03.062453 ip-10-0-140-140 kubenswrapper[5605]: E1103 01:57:03.062433 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50e10272ec6a0643166e5bf551cc84efc9e56a276f53a18e12acc6c3f53a8d5a\": container with ID starting with 50e10272ec6a0643166e5bf551cc84efc9e56a276f53a18e12acc6c3f53a8d5a not found: ID does not exist" containerID="50e10272ec6a0643166e5bf551cc84efc9e56a276f53a18e12acc6c3f53a8d5a" Nov 03 01:57:03.062514 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:57:03.062462 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50e10272ec6a0643166e5bf551cc84efc9e56a276f53a18e12acc6c3f53a8d5a"} err="failed to get container status \"50e10272ec6a0643166e5bf551cc84efc9e56a276f53a18e12acc6c3f53a8d5a\": rpc error: code = NotFound desc = could not find container \"50e10272ec6a0643166e5bf551cc84efc9e56a276f53a18e12acc6c3f53a8d5a\": container with ID starting with 50e10272ec6a0643166e5bf551cc84efc9e56a276f53a18e12acc6c3f53a8d5a not found: ID does not exist" Nov 03 01:57:03.212288 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:57:03.212209 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-4c766"] Nov 03 01:57:03.220112 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:57:03.220079 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-4c766"] Nov 03 01:57:03.323207 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:57:03.323177 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-ccnd9"] Nov 03 01:57:03.323488 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:57:03.323476 5605 cpu_manager.go:414] "containerMap: removing container" podUID="0c4868b2-22cd-41c5-8e7d-d7dffacf1973" containerName="discovery" Nov 03 01:57:03.323547 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:57:03.323489 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c4868b2-22cd-41c5-8e7d-d7dffacf1973" containerName="discovery" Nov 03 01:57:03.323586 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:57:03.323558 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c4868b2-22cd-41c5-8e7d-d7dffacf1973" containerName="discovery" Nov 03 01:57:03.327495 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:57:03.327477 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ccnd9" Nov 03 01:57:03.365139 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:57:03.365104 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/653d89f2-5cbd-41ba-a091-a984621c8852-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-ccnd9\" (UID: \"653d89f2-5cbd-41ba-a091-a984621c8852\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ccnd9" Nov 03 01:57:03.365328 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:57:03.365152 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrqz4\" (UniqueName: \"kubernetes.io/projected/653d89f2-5cbd-41ba-a091-a984621c8852-kube-api-access-rrqz4\") pod \"llmisvc-controller-manager-869d5dcc5d-ccnd9\" (UID: \"653d89f2-5cbd-41ba-a091-a984621c8852\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ccnd9" Nov 03 01:57:03.379871 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:57:03.379844 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-ccnd9"] Nov 03 01:57:03.381623 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:57:03.381578 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 01:57:03.381844 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:57:03.381816 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 01:57:03.382383 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:57:03.382310 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-9t8nn\"" Nov 03 01:57:03.382491 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:57:03.382394 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Nov 03 01:57:03.465944 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:57:03.465859 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/653d89f2-5cbd-41ba-a091-a984621c8852-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-ccnd9\" (UID: \"653d89f2-5cbd-41ba-a091-a984621c8852\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ccnd9" Nov 03 01:57:03.465944 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:57:03.465905 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrqz4\" (UniqueName: \"kubernetes.io/projected/653d89f2-5cbd-41ba-a091-a984621c8852-kube-api-access-rrqz4\") pod \"llmisvc-controller-manager-869d5dcc5d-ccnd9\" (UID: \"653d89f2-5cbd-41ba-a091-a984621c8852\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ccnd9" Nov 03 01:57:03.468278 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:57:03.468248 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/653d89f2-5cbd-41ba-a091-a984621c8852-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-ccnd9\" (UID: \"653d89f2-5cbd-41ba-a091-a984621c8852\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ccnd9" Nov 03 01:57:03.522248 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:57:03.522222 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrqz4\" (UniqueName: \"kubernetes.io/projected/653d89f2-5cbd-41ba-a091-a984621c8852-kube-api-access-rrqz4\") pod \"llmisvc-controller-manager-869d5dcc5d-ccnd9\" (UID: \"653d89f2-5cbd-41ba-a091-a984621c8852\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ccnd9" Nov 03 01:57:03.639027 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:57:03.638995 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ccnd9" Nov 03 01:57:04.002636 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:57:04.002598 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-ccnd9"] Nov 03 01:57:04.009599 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:57:04.009566 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod653d89f2_5cbd_41ba_a091_a984621c8852.slice/crio-1488d74a5e7218b3f0aa27480de389fd5f37f701a4b1984aa236e0836c1c9cca WatchSource:0}: Error finding container 1488d74a5e7218b3f0aa27480de389fd5f37f701a4b1984aa236e0836c1c9cca: Status 404 returned error can't find the container with id 1488d74a5e7218b3f0aa27480de389fd5f37f701a4b1984aa236e0836c1c9cca Nov 03 01:57:04.059578 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:57:04.059540 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ccnd9" event={"ID":"653d89f2-5cbd-41ba-a091-a984621c8852","Type":"ContainerStarted","Data":"1488d74a5e7218b3f0aa27480de389fd5f37f701a4b1984aa236e0836c1c9cca"} Nov 03 01:57:04.849123 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:57:04.849090 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c4868b2-22cd-41c5-8e7d-d7dffacf1973" path="/var/lib/kubelet/pods/0c4868b2-22cd-41c5-8e7d-d7dffacf1973/volumes" Nov 03 01:57:09.082214 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:57:09.082179 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ccnd9" event={"ID":"653d89f2-5cbd-41ba-a091-a984621c8852","Type":"ContainerStarted","Data":"01d64934982bbc24937cd1430167c2ca57c3b1ac0470bd2f6ca2013710f6e3f4"} Nov 03 01:57:09.082664 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:57:09.082307 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ccnd9" Nov 03 01:57:09.189460 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:57:09.189399 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ccnd9" podStartSLOduration=1.621385165 podStartE2EDuration="6.18938312s" totalImagesPullingTime="4.567997955s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:57:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:57:09.188133663 +0000 UTC m=+306.447331816" watchObservedRunningTime="2026-11-03 01:57:09.18938312 +0000 UTC m=+306.448581273" Nov 03 01:57:23.910567 ip-10-0-140-140 kubenswrapper[5605]: E1103 01:57:23.910527 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 01:57:35.172574 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:57:35.172534 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 01:57:40.088630 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:57:40.088597 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ccnd9" Nov 03 01:57:53.912905 ip-10-0-140-140 kubenswrapper[5605]: E1103 01:57:53.912866 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 01:58:09.481294 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:58:09.481254 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-rbsfv"] Nov 03 01:58:09.485936 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:58:09.485913 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-rbsfv" Nov 03 01:58:09.521458 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:58:09.521425 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq5hk\" (UniqueName: \"kubernetes.io/projected/fc905c8d-4bfa-49ee-b0b7-8c434727e284-kube-api-access-zq5hk\") pod \"llmisvc-controller-manager-6466c554d8-rbsfv\" (UID: \"fc905c8d-4bfa-49ee-b0b7-8c434727e284\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-rbsfv" Nov 03 01:58:09.521580 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:58:09.521490 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fc905c8d-4bfa-49ee-b0b7-8c434727e284-cert\") pod \"llmisvc-controller-manager-6466c554d8-rbsfv\" (UID: \"fc905c8d-4bfa-49ee-b0b7-8c434727e284\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-rbsfv" Nov 03 01:58:09.537279 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:58:09.537257 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-rbsfv"] Nov 03 01:58:09.622440 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:58:09.622408 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fc905c8d-4bfa-49ee-b0b7-8c434727e284-cert\") pod \"llmisvc-controller-manager-6466c554d8-rbsfv\" (UID: \"fc905c8d-4bfa-49ee-b0b7-8c434727e284\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-rbsfv" Nov 03 01:58:09.622620 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:58:09.622500 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq5hk\" (UniqueName: \"kubernetes.io/projected/fc905c8d-4bfa-49ee-b0b7-8c434727e284-kube-api-access-zq5hk\") pod \"llmisvc-controller-manager-6466c554d8-rbsfv\" (UID: \"fc905c8d-4bfa-49ee-b0b7-8c434727e284\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-rbsfv" Nov 03 01:58:09.624863 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:58:09.624830 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fc905c8d-4bfa-49ee-b0b7-8c434727e284-cert\") pod \"llmisvc-controller-manager-6466c554d8-rbsfv\" (UID: \"fc905c8d-4bfa-49ee-b0b7-8c434727e284\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-rbsfv" Nov 03 01:58:09.675391 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:58:09.675345 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq5hk\" (UniqueName: \"kubernetes.io/projected/fc905c8d-4bfa-49ee-b0b7-8c434727e284-kube-api-access-zq5hk\") pod \"llmisvc-controller-manager-6466c554d8-rbsfv\" (UID: \"fc905c8d-4bfa-49ee-b0b7-8c434727e284\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-rbsfv" Nov 03 01:58:09.797997 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:58:09.797906 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-rbsfv" Nov 03 01:58:10.160374 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:58:10.160330 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-rbsfv"] Nov 03 01:58:10.166670 ip-10-0-140-140 kubenswrapper[5605]: W1103 01:58:10.166635 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc905c8d_4bfa_49ee_b0b7_8c434727e284.slice/crio-6b0eebc2c3333d9891b9726f515ab1bcf17d771027cc5322dcd1c825e163a859 WatchSource:0}: Error finding container 6b0eebc2c3333d9891b9726f515ab1bcf17d771027cc5322dcd1c825e163a859: Status 404 returned error can't find the container with id 6b0eebc2c3333d9891b9726f515ab1bcf17d771027cc5322dcd1c825e163a859 Nov 03 01:58:10.329127 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:58:10.329090 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-rbsfv" event={"ID":"fc905c8d-4bfa-49ee-b0b7-8c434727e284","Type":"ContainerStarted","Data":"6b0eebc2c3333d9891b9726f515ab1bcf17d771027cc5322dcd1c825e163a859"} Nov 03 01:58:11.334462 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:58:11.334425 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-rbsfv" event={"ID":"fc905c8d-4bfa-49ee-b0b7-8c434727e284","Type":"ContainerStarted","Data":"258fc453df00b9434d6034e51cd6f89915a6b4e8b208b57f5556fbbd92a5d11b"} Nov 03 01:58:11.334848 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:58:11.334493 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-rbsfv" Nov 03 01:58:11.440709 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:58:11.440657 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-rbsfv" podStartSLOduration=1.627223952 podStartE2EDuration="2.440643099s" totalImagesPullingTime="813.419147ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:58:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:58:11.439903433 +0000 UTC m=+368.699101588" watchObservedRunningTime="2026-11-03 01:58:11.440643099 +0000 UTC m=+368.699841253" Nov 03 01:58:23.912288 ip-10-0-140-140 kubenswrapper[5605]: E1103 01:58:23.912252 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 01:58:42.341296 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:58:42.341257 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-rbsfv" Nov 03 01:58:42.494155 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:58:42.494121 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-ccnd9"] Nov 03 01:58:42.494424 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:58:42.494381 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ccnd9" podUID="653d89f2-5cbd-41ba-a091-a984621c8852" containerName="manager" containerID="cri-o://01d64934982bbc24937cd1430167c2ca57c3b1ac0470bd2f6ca2013710f6e3f4" gracePeriod=30 Nov 03 01:58:42.730407 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:58:42.730373 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ccnd9" Nov 03 01:58:42.911889 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:58:42.911806 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/653d89f2-5cbd-41ba-a091-a984621c8852-kube-api-access-rrqz4\" (UniqueName: \"kubernetes.io/projected/653d89f2-5cbd-41ba-a091-a984621c8852-kube-api-access-rrqz4\") pod \"653d89f2-5cbd-41ba-a091-a984621c8852\" (UID: \"653d89f2-5cbd-41ba-a091-a984621c8852\") " Nov 03 01:58:42.912030 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:58:42.911897 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/653d89f2-5cbd-41ba-a091-a984621c8852-cert\" (UniqueName: \"kubernetes.io/secret/653d89f2-5cbd-41ba-a091-a984621c8852-cert\") pod \"653d89f2-5cbd-41ba-a091-a984621c8852\" (UID: \"653d89f2-5cbd-41ba-a091-a984621c8852\") " Nov 03 01:58:42.914098 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:58:42.914067 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/653d89f2-5cbd-41ba-a091-a984621c8852-cert" pod "653d89f2-5cbd-41ba-a091-a984621c8852" (UID: "653d89f2-5cbd-41ba-a091-a984621c8852"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:58:42.914173 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:58:42.914122 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/653d89f2-5cbd-41ba-a091-a984621c8852-kube-api-access-rrqz4" pod "653d89f2-5cbd-41ba-a091-a984621c8852" (UID: "653d89f2-5cbd-41ba-a091-a984621c8852"). InnerVolumeSpecName "kube-api-access-rrqz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:58:43.012629 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:58:43.012592 5605 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/653d89f2-5cbd-41ba-a091-a984621c8852-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:58:43.012629 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:58:43.012622 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrqz4\" (UniqueName: \"kubernetes.io/projected/653d89f2-5cbd-41ba-a091-a984621c8852-kube-api-access-rrqz4\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:58:43.460351 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:58:43.460316 5605 generic.go:356] "Generic (PLEG): container finished" podID="653d89f2-5cbd-41ba-a091-a984621c8852" containerID="01d64934982bbc24937cd1430167c2ca57c3b1ac0470bd2f6ca2013710f6e3f4" exitCode=0 Nov 03 01:58:43.460881 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:58:43.460384 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ccnd9" Nov 03 01:58:43.460881 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:58:43.460418 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ccnd9" event={"ID":"653d89f2-5cbd-41ba-a091-a984621c8852","Type":"ContainerDied","Data":"01d64934982bbc24937cd1430167c2ca57c3b1ac0470bd2f6ca2013710f6e3f4"} Nov 03 01:58:43.460881 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:58:43.460487 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ccnd9" event={"ID":"653d89f2-5cbd-41ba-a091-a984621c8852","Type":"ContainerDied","Data":"1488d74a5e7218b3f0aa27480de389fd5f37f701a4b1984aa236e0836c1c9cca"} Nov 03 01:58:43.460881 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:58:43.460520 5605 scope.go:122] "RemoveContainer" containerID="01d64934982bbc24937cd1430167c2ca57c3b1ac0470bd2f6ca2013710f6e3f4" Nov 03 01:58:43.468801 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:58:43.468781 5605 scope.go:122] "RemoveContainer" containerID="01d64934982bbc24937cd1430167c2ca57c3b1ac0470bd2f6ca2013710f6e3f4" Nov 03 01:58:43.469064 ip-10-0-140-140 kubenswrapper[5605]: E1103 01:58:43.469040 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01d64934982bbc24937cd1430167c2ca57c3b1ac0470bd2f6ca2013710f6e3f4\": container with ID starting with 01d64934982bbc24937cd1430167c2ca57c3b1ac0470bd2f6ca2013710f6e3f4 not found: ID does not exist" containerID="01d64934982bbc24937cd1430167c2ca57c3b1ac0470bd2f6ca2013710f6e3f4" Nov 03 01:58:43.469106 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:58:43.469067 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01d64934982bbc24937cd1430167c2ca57c3b1ac0470bd2f6ca2013710f6e3f4"} err="failed to get container status \"01d64934982bbc24937cd1430167c2ca57c3b1ac0470bd2f6ca2013710f6e3f4\": rpc error: code = NotFound desc = could not find container \"01d64934982bbc24937cd1430167c2ca57c3b1ac0470bd2f6ca2013710f6e3f4\": container with ID starting with 01d64934982bbc24937cd1430167c2ca57c3b1ac0470bd2f6ca2013710f6e3f4 not found: ID does not exist" Nov 03 01:58:43.612607 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:58:43.612576 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-ccnd9"] Nov 03 01:58:43.614373 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:58:43.614339 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-ccnd9"] Nov 03 01:58:44.848335 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:58:44.848304 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="653d89f2-5cbd-41ba-a091-a984621c8852" path="/var/lib/kubelet/pods/653d89f2-5cbd-41ba-a091-a984621c8852/volumes" Nov 03 01:58:53.912069 ip-10-0-140-140 kubenswrapper[5605]: E1103 01:58:53.912025 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 01:59:23.912023 ip-10-0-140-140 kubenswrapper[5605]: E1103 01:59:23.911973 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 01:59:33.227892 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:59:33.227858 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 01:59:43.377713 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:59:43.377634 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 01:59:51.534025 ip-10-0-140-140 kubenswrapper[5605]: I1103 01:59:51.533988 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 01:59:53.911341 ip-10-0-140-140 kubenswrapper[5605]: E1103 01:59:53.911305 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:00:03.158594 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:03.158557 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:00:03.498057 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:03.497977 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-l6wzt"] Nov 03 02:00:03.498301 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:03.498289 5605 cpu_manager.go:414] "containerMap: removing container" podUID="653d89f2-5cbd-41ba-a091-a984621c8852" containerName="manager" Nov 03 02:00:03.498343 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:03.498302 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="653d89f2-5cbd-41ba-a091-a984621c8852" containerName="manager" Nov 03 02:00:03.498396 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:03.498369 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="653d89f2-5cbd-41ba-a091-a984621c8852" containerName="manager" Nov 03 02:00:03.501432 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:03.501405 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-l6wzt" Nov 03 02:00:03.596790 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:03.596755 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1ba0fd6-b847-4fda-92f9-a9b29eb4e881-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-l6wzt\" (UID: \"c1ba0fd6-b847-4fda-92f9-a9b29eb4e881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-l6wzt" Nov 03 02:00:03.596968 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:03.596800 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1ba0fd6-b847-4fda-92f9-a9b29eb4e881-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-l6wzt\" (UID: \"c1ba0fd6-b847-4fda-92f9-a9b29eb4e881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-l6wzt" Nov 03 02:00:03.596968 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:03.596824 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1ba0fd6-b847-4fda-92f9-a9b29eb4e881-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-l6wzt\" (UID: \"c1ba0fd6-b847-4fda-92f9-a9b29eb4e881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-l6wzt" Nov 03 02:00:03.596968 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:03.596856 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1ba0fd6-b847-4fda-92f9-a9b29eb4e881-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-l6wzt\" (UID: \"c1ba0fd6-b847-4fda-92f9-a9b29eb4e881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-l6wzt" Nov 03 02:00:03.596968 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:03.596945 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1ba0fd6-b847-4fda-92f9-a9b29eb4e881-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-l6wzt\" (UID: \"c1ba0fd6-b847-4fda-92f9-a9b29eb4e881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-l6wzt" Nov 03 02:00:03.597205 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:03.597093 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flc98\" (UniqueName: \"kubernetes.io/projected/c1ba0fd6-b847-4fda-92f9-a9b29eb4e881-kube-api-access-flc98\") pod \"maas-default-gateway-openshift-default-5bc884dd87-l6wzt\" (UID: \"c1ba0fd6-b847-4fda-92f9-a9b29eb4e881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-l6wzt" Nov 03 02:00:03.597205 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:03.597129 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1ba0fd6-b847-4fda-92f9-a9b29eb4e881-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-l6wzt\" (UID: \"c1ba0fd6-b847-4fda-92f9-a9b29eb4e881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-l6wzt" Nov 03 02:00:03.597205 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:03.597159 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1ba0fd6-b847-4fda-92f9-a9b29eb4e881-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-l6wzt\" (UID: \"c1ba0fd6-b847-4fda-92f9-a9b29eb4e881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-l6wzt" Nov 03 02:00:03.597370 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:03.597264 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1ba0fd6-b847-4fda-92f9-a9b29eb4e881-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-l6wzt\" (UID: \"c1ba0fd6-b847-4fda-92f9-a9b29eb4e881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-l6wzt" Nov 03 02:00:03.597996 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:03.597978 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-l6wzt"] Nov 03 02:00:03.637525 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:03.637495 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-fwpk8\"" Nov 03 02:00:03.698701 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:03.698668 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1ba0fd6-b847-4fda-92f9-a9b29eb4e881-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-l6wzt\" (UID: \"c1ba0fd6-b847-4fda-92f9-a9b29eb4e881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-l6wzt" Nov 03 02:00:03.698701 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:03.698705 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1ba0fd6-b847-4fda-92f9-a9b29eb4e881-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-l6wzt\" (UID: \"c1ba0fd6-b847-4fda-92f9-a9b29eb4e881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-l6wzt" Nov 03 02:00:03.698895 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:03.698733 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1ba0fd6-b847-4fda-92f9-a9b29eb4e881-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-l6wzt\" (UID: \"c1ba0fd6-b847-4fda-92f9-a9b29eb4e881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-l6wzt" Nov 03 02:00:03.698939 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:03.698895 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1ba0fd6-b847-4fda-92f9-a9b29eb4e881-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-l6wzt\" (UID: \"c1ba0fd6-b847-4fda-92f9-a9b29eb4e881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-l6wzt" Nov 03 02:00:03.699008 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:03.698990 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flc98\" (UniqueName: \"kubernetes.io/projected/c1ba0fd6-b847-4fda-92f9-a9b29eb4e881-kube-api-access-flc98\") pod \"maas-default-gateway-openshift-default-5bc884dd87-l6wzt\" (UID: \"c1ba0fd6-b847-4fda-92f9-a9b29eb4e881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-l6wzt" Nov 03 02:00:03.699064 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:03.699027 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1ba0fd6-b847-4fda-92f9-a9b29eb4e881-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-l6wzt\" (UID: \"c1ba0fd6-b847-4fda-92f9-a9b29eb4e881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-l6wzt" Nov 03 02:00:03.699064 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:03.699059 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1ba0fd6-b847-4fda-92f9-a9b29eb4e881-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-l6wzt\" (UID: \"c1ba0fd6-b847-4fda-92f9-a9b29eb4e881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-l6wzt" Nov 03 02:00:03.699172 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:03.699111 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1ba0fd6-b847-4fda-92f9-a9b29eb4e881-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-l6wzt\" (UID: \"c1ba0fd6-b847-4fda-92f9-a9b29eb4e881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-l6wzt" Nov 03 02:00:03.699231 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:03.699171 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1ba0fd6-b847-4fda-92f9-a9b29eb4e881-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-l6wzt\" (UID: \"c1ba0fd6-b847-4fda-92f9-a9b29eb4e881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-l6wzt" Nov 03 02:00:03.699231 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:03.699190 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1ba0fd6-b847-4fda-92f9-a9b29eb4e881-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-l6wzt\" (UID: \"c1ba0fd6-b847-4fda-92f9-a9b29eb4e881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-l6wzt" Nov 03 02:00:03.699510 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:03.699486 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1ba0fd6-b847-4fda-92f9-a9b29eb4e881-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-l6wzt\" (UID: \"c1ba0fd6-b847-4fda-92f9-a9b29eb4e881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-l6wzt" Nov 03 02:00:03.699659 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:03.699633 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1ba0fd6-b847-4fda-92f9-a9b29eb4e881-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-l6wzt\" (UID: \"c1ba0fd6-b847-4fda-92f9-a9b29eb4e881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-l6wzt" Nov 03 02:00:03.699715 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:03.699690 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1ba0fd6-b847-4fda-92f9-a9b29eb4e881-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-l6wzt\" (UID: \"c1ba0fd6-b847-4fda-92f9-a9b29eb4e881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-l6wzt" Nov 03 02:00:03.699754 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:03.699706 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1ba0fd6-b847-4fda-92f9-a9b29eb4e881-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-l6wzt\" (UID: \"c1ba0fd6-b847-4fda-92f9-a9b29eb4e881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-l6wzt" Nov 03 02:00:03.701105 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:03.701086 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1ba0fd6-b847-4fda-92f9-a9b29eb4e881-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-l6wzt\" (UID: \"c1ba0fd6-b847-4fda-92f9-a9b29eb4e881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-l6wzt" Nov 03 02:00:03.701226 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:03.701210 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1ba0fd6-b847-4fda-92f9-a9b29eb4e881-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-l6wzt\" (UID: \"c1ba0fd6-b847-4fda-92f9-a9b29eb4e881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-l6wzt" Nov 03 02:00:03.755578 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:03.755511 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1ba0fd6-b847-4fda-92f9-a9b29eb4e881-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-l6wzt\" (UID: \"c1ba0fd6-b847-4fda-92f9-a9b29eb4e881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-l6wzt" Nov 03 02:00:03.756184 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:03.756167 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flc98\" (UniqueName: \"kubernetes.io/projected/c1ba0fd6-b847-4fda-92f9-a9b29eb4e881-kube-api-access-flc98\") pod \"maas-default-gateway-openshift-default-5bc884dd87-l6wzt\" (UID: \"c1ba0fd6-b847-4fda-92f9-a9b29eb4e881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-l6wzt" Nov 03 02:00:03.814163 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:03.814132 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-l6wzt" Nov 03 02:00:04.229231 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:04.229197 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-l6wzt"] Nov 03 02:00:04.235669 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:00:04.235640 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1ba0fd6_b847_4fda_92f9_a9b29eb4e881.slice/crio-1eec86d28eae5ab243e996560ae0d2bb5dac79215ef755c5360fcd6bf2251c21 WatchSource:0}: Error finding container 1eec86d28eae5ab243e996560ae0d2bb5dac79215ef755c5360fcd6bf2251c21: Status 404 returned error can't find the container with id 1eec86d28eae5ab243e996560ae0d2bb5dac79215ef755c5360fcd6bf2251c21 Nov 03 02:00:04.237740 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:04.237722 5605 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 02:00:04.786295 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:04.786256 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-l6wzt" event={"ID":"c1ba0fd6-b847-4fda-92f9-a9b29eb4e881","Type":"ContainerStarted","Data":"1eec86d28eae5ab243e996560ae0d2bb5dac79215ef755c5360fcd6bf2251c21"} Nov 03 02:00:16.282457 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:16.282423 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:00:16.282797 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:16.282493 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:00:16.282797 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:16.282523 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:00:16.838078 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:16.838044 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-l6wzt" event={"ID":"c1ba0fd6-b847-4fda-92f9-a9b29eb4e881","Type":"ContainerStarted","Data":"e7f4bf9f1864e7ce89492bb637110b3b21ad958ec41a4ef5555f0ff882078fa9"} Nov 03 02:00:17.041242 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:17.041186 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-l6wzt" podStartSLOduration=1.997498533 podStartE2EDuration="14.041173128s" totalImagesPullingTime="12.043674595s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:00:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:00:16.994652565 +0000 UTC m=+494.253850719" watchObservedRunningTime="2026-11-03 02:00:17.041173128 +0000 UTC m=+494.300371352" Nov 03 02:00:17.814435 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:17.814398 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-l6wzt" Nov 03 02:00:17.815895 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:17.815868 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-l6wzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body= Nov 03 02:00:17.816017 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:17.815925 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-l6wzt" podUID="c1ba0fd6-b847-4fda-92f9-a9b29eb4e881" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" Nov 03 02:00:18.627320 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:18.627286 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s9dp7"] Nov 03 02:00:18.630665 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:18.630644 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s9dp7" Nov 03 02:00:18.738944 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:18.738907 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ef2131-720c-4db4-8c4f-e618edf4b5e9-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s9dp7\" (UID: \"c3ef2131-720c-4db4-8c4f-e618edf4b5e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s9dp7" Nov 03 02:00:18.739128 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:18.738956 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3ef2131-720c-4db4-8c4f-e618edf4b5e9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s9dp7\" (UID: \"c3ef2131-720c-4db4-8c4f-e618edf4b5e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s9dp7" Nov 03 02:00:18.739128 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:18.739024 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3ef2131-720c-4db4-8c4f-e618edf4b5e9-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s9dp7\" (UID: \"c3ef2131-720c-4db4-8c4f-e618edf4b5e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s9dp7" Nov 03 02:00:18.739128 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:18.739049 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79pd6\" (UniqueName: \"kubernetes.io/projected/c3ef2131-720c-4db4-8c4f-e618edf4b5e9-kube-api-access-79pd6\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s9dp7\" (UID: \"c3ef2131-720c-4db4-8c4f-e618edf4b5e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s9dp7" Nov 03 02:00:18.739128 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:18.739074 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3ef2131-720c-4db4-8c4f-e618edf4b5e9-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s9dp7\" (UID: \"c3ef2131-720c-4db4-8c4f-e618edf4b5e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s9dp7" Nov 03 02:00:18.739274 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:18.739194 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ef2131-720c-4db4-8c4f-e618edf4b5e9-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s9dp7\" (UID: \"c3ef2131-720c-4db4-8c4f-e618edf4b5e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s9dp7" Nov 03 02:00:18.739274 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:18.739245 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3ef2131-720c-4db4-8c4f-e618edf4b5e9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s9dp7\" (UID: \"c3ef2131-720c-4db4-8c4f-e618edf4b5e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s9dp7" Nov 03 02:00:18.739338 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:18.739317 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3ef2131-720c-4db4-8c4f-e618edf4b5e9-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s9dp7\" (UID: \"c3ef2131-720c-4db4-8c4f-e618edf4b5e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s9dp7" Nov 03 02:00:18.739406 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:18.739346 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3ef2131-720c-4db4-8c4f-e618edf4b5e9-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s9dp7\" (UID: \"c3ef2131-720c-4db4-8c4f-e618edf4b5e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s9dp7" Nov 03 02:00:18.815003 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:18.814943 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-l6wzt" Nov 03 02:00:18.815429 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:18.815202 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-l6wzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body= Nov 03 02:00:18.815429 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:18.815247 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-l6wzt" podUID="c1ba0fd6-b847-4fda-92f9-a9b29eb4e881" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" Nov 03 02:00:18.820408 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:18.820351 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s9dp7"] Nov 03 02:00:18.840250 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:18.840221 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ef2131-720c-4db4-8c4f-e618edf4b5e9-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s9dp7\" (UID: \"c3ef2131-720c-4db4-8c4f-e618edf4b5e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s9dp7" Nov 03 02:00:18.840429 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:18.840265 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3ef2131-720c-4db4-8c4f-e618edf4b5e9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s9dp7\" (UID: \"c3ef2131-720c-4db4-8c4f-e618edf4b5e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s9dp7" Nov 03 02:00:18.840429 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:18.840306 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3ef2131-720c-4db4-8c4f-e618edf4b5e9-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s9dp7\" (UID: \"c3ef2131-720c-4db4-8c4f-e618edf4b5e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s9dp7" Nov 03 02:00:18.840429 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:18.840331 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3ef2131-720c-4db4-8c4f-e618edf4b5e9-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s9dp7\" (UID: \"c3ef2131-720c-4db4-8c4f-e618edf4b5e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s9dp7" Nov 03 02:00:18.840429 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:18.840351 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ef2131-720c-4db4-8c4f-e618edf4b5e9-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s9dp7\" (UID: \"c3ef2131-720c-4db4-8c4f-e618edf4b5e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s9dp7" Nov 03 02:00:18.840429 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:18.840398 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3ef2131-720c-4db4-8c4f-e618edf4b5e9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s9dp7\" (UID: \"c3ef2131-720c-4db4-8c4f-e618edf4b5e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s9dp7" Nov 03 02:00:18.840739 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:18.840711 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3ef2131-720c-4db4-8c4f-e618edf4b5e9-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s9dp7\" (UID: \"c3ef2131-720c-4db4-8c4f-e618edf4b5e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s9dp7" Nov 03 02:00:18.840800 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:18.840735 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ef2131-720c-4db4-8c4f-e618edf4b5e9-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s9dp7\" (UID: \"c3ef2131-720c-4db4-8c4f-e618edf4b5e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s9dp7" Nov 03 02:00:18.840800 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:18.840765 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79pd6\" (UniqueName: \"kubernetes.io/projected/c3ef2131-720c-4db4-8c4f-e618edf4b5e9-kube-api-access-79pd6\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s9dp7\" (UID: \"c3ef2131-720c-4db4-8c4f-e618edf4b5e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s9dp7" Nov 03 02:00:18.840898 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:18.840800 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3ef2131-720c-4db4-8c4f-e618edf4b5e9-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s9dp7\" (UID: \"c3ef2131-720c-4db4-8c4f-e618edf4b5e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s9dp7" Nov 03 02:00:18.841303 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:18.841277 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3ef2131-720c-4db4-8c4f-e618edf4b5e9-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s9dp7\" (UID: \"c3ef2131-720c-4db4-8c4f-e618edf4b5e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s9dp7" Nov 03 02:00:18.841507 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:18.841379 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ef2131-720c-4db4-8c4f-e618edf4b5e9-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s9dp7\" (UID: \"c3ef2131-720c-4db4-8c4f-e618edf4b5e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s9dp7" Nov 03 02:00:18.841644 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:18.841494 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3ef2131-720c-4db4-8c4f-e618edf4b5e9-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s9dp7\" (UID: \"c3ef2131-720c-4db4-8c4f-e618edf4b5e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s9dp7" Nov 03 02:00:18.841644 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:18.841585 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3ef2131-720c-4db4-8c4f-e618edf4b5e9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s9dp7\" (UID: \"c3ef2131-720c-4db4-8c4f-e618edf4b5e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s9dp7" Nov 03 02:00:18.843029 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:18.843013 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3ef2131-720c-4db4-8c4f-e618edf4b5e9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s9dp7\" (UID: \"c3ef2131-720c-4db4-8c4f-e618edf4b5e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s9dp7" Nov 03 02:00:18.843468 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:18.843439 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3ef2131-720c-4db4-8c4f-e618edf4b5e9-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s9dp7\" (UID: \"c3ef2131-720c-4db4-8c4f-e618edf4b5e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s9dp7" Nov 03 02:00:18.894038 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:18.893963 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3ef2131-720c-4db4-8c4f-e618edf4b5e9-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s9dp7\" (UID: \"c3ef2131-720c-4db4-8c4f-e618edf4b5e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s9dp7" Nov 03 02:00:18.894177 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:18.894074 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79pd6\" (UniqueName: \"kubernetes.io/projected/c3ef2131-720c-4db4-8c4f-e618edf4b5e9-kube-api-access-79pd6\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s9dp7\" (UID: \"c3ef2131-720c-4db4-8c4f-e618edf4b5e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s9dp7" Nov 03 02:00:18.944558 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:18.944528 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s9dp7" Nov 03 02:00:19.389646 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:19.389613 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s9dp7"] Nov 03 02:00:19.398008 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:00:19.397974 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3ef2131_720c_4db4_8c4f_e618edf4b5e9.slice/crio-6227d73ecd88a25dd8ecc6867d767b3923ef0c304922f111c280b3c3057bd0de WatchSource:0}: Error finding container 6227d73ecd88a25dd8ecc6867d767b3923ef0c304922f111c280b3c3057bd0de: Status 404 returned error can't find the container with id 6227d73ecd88a25dd8ecc6867d767b3923ef0c304922f111c280b3c3057bd0de Nov 03 02:00:19.401238 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:19.401202 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:00:19.401337 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:19.401276 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:00:19.401337 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:19.401306 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:00:19.814827 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:19.814794 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-l6wzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body= Nov 03 02:00:19.815020 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:19.814850 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-l6wzt" podUID="c1ba0fd6-b847-4fda-92f9-a9b29eb4e881" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" Nov 03 02:00:19.853601 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:19.853557 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s9dp7" event={"ID":"c3ef2131-720c-4db4-8c4f-e618edf4b5e9","Type":"ContainerStarted","Data":"3fdf31d4ddd19f1cda9ab61a21b75e60d82c89698c3f569c1ba5ccc47daacc3c"} Nov 03 02:00:19.853778 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:19.853624 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s9dp7" event={"ID":"c3ef2131-720c-4db4-8c4f-e618edf4b5e9","Type":"ContainerStarted","Data":"6227d73ecd88a25dd8ecc6867d767b3923ef0c304922f111c280b3c3057bd0de"} Nov 03 02:00:19.945181 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:19.945139 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s9dp7" Nov 03 02:00:20.815193 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:20.815154 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-l6wzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body= Nov 03 02:00:20.815595 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:20.815219 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-l6wzt" podUID="c1ba0fd6-b847-4fda-92f9-a9b29eb4e881" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" Nov 03 02:00:20.946859 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:20.946823 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-s9dp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Nov 03 02:00:20.947019 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:20.946882 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s9dp7" podUID="c3ef2131-720c-4db4-8c4f-e618edf4b5e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Nov 03 02:00:21.814881 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:21.814842 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-l6wzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body= Nov 03 02:00:21.815190 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:21.814904 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-l6wzt" podUID="c1ba0fd6-b847-4fda-92f9-a9b29eb4e881" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" Nov 03 02:00:21.944880 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:21.944847 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-s9dp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Nov 03 02:00:21.945329 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:21.945302 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s9dp7" podUID="c3ef2131-720c-4db4-8c4f-e618edf4b5e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Nov 03 02:00:22.814958 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:22.814925 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-l6wzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body= Nov 03 02:00:22.815120 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:22.814975 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-l6wzt" podUID="c1ba0fd6-b847-4fda-92f9-a9b29eb4e881" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" Nov 03 02:00:22.945568 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:22.945531 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-s9dp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Nov 03 02:00:22.945938 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:22.945583 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s9dp7" podUID="c3ef2131-720c-4db4-8c4f-e618edf4b5e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Nov 03 02:00:23.814987 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:23.814953 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-l6wzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body= Nov 03 02:00:23.815164 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:23.815003 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-l6wzt" podUID="c1ba0fd6-b847-4fda-92f9-a9b29eb4e881" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" Nov 03 02:00:23.912895 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:00:23.912858 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:00:23.945693 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:23.945652 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-s9dp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Nov 03 02:00:23.946046 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:23.945720 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s9dp7" podUID="c3ef2131-720c-4db4-8c4f-e618edf4b5e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Nov 03 02:00:24.944891 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:24.944843 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-s9dp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Nov 03 02:00:24.945086 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:24.944907 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s9dp7" podUID="c3ef2131-720c-4db4-8c4f-e618edf4b5e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Nov 03 02:00:25.495887 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:25.495851 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-l6wzt" Nov 03 02:00:25.497508 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:25.497485 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-l6wzt" Nov 03 02:00:25.606140 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:25.606089 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s9dp7" podStartSLOduration=7.606074518 podStartE2EDuration="7.606074518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:00:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:00:19.977765907 +0000 UTC m=+497.236964074" watchObservedRunningTime="2026-11-03 02:00:25.606074518 +0000 UTC m=+502.865272731" Nov 03 02:00:25.945201 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:25.945161 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-s9dp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Nov 03 02:00:25.945414 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:25.945221 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s9dp7" podUID="c3ef2131-720c-4db4-8c4f-e618edf4b5e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Nov 03 02:00:26.945865 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:26.945825 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-s9dp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Nov 03 02:00:26.946243 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:26.945891 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s9dp7" podUID="c3ef2131-720c-4db4-8c4f-e618edf4b5e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Nov 03 02:00:27.949143 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:27.949112 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s9dp7" Nov 03 02:00:27.949544 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:27.949412 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s9dp7" Nov 03 02:00:27.950079 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:00:27.950059 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s9dp7" Nov 03 02:00:53.912161 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:00:53.912123 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:01:20.933012 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:01:20.932933 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-7683f606-69d94fc9c9-9npx5"] Nov 03 02:01:20.936346 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:01:20.936323 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7683f606-69d94fc9c9-9npx5" Nov 03 02:01:20.989010 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:01:20.988978 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-7683f606-69d94fc9c9-9npx5"] Nov 03 02:01:21.034466 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:01:21.034428 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-7683f606\"" Nov 03 02:01:21.034466 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:01:21.034435 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-7683f606-dockercfg-vxjhq\"" Nov 03 02:01:21.051425 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:01:21.051386 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49fe9781-4bf8-4cde-9461-6e993dcc0725-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-7683f606-69d94fc9c9-9npx5\" (UID: \"49fe9781-4bf8-4cde-9461-6e993dcc0725\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7683f606-69d94fc9c9-9npx5" Nov 03 02:01:21.051600 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:01:21.051439 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5sqg\" (UniqueName: \"kubernetes.io/projected/49fe9781-4bf8-4cde-9461-6e993dcc0725-kube-api-access-z5sqg\") pod \"payload-pre-processing-e2e-migrate-7683f606-69d94fc9c9-9npx5\" (UID: \"49fe9781-4bf8-4cde-9461-6e993dcc0725\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7683f606-69d94fc9c9-9npx5" Nov 03 02:01:21.152082 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:01:21.152046 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49fe9781-4bf8-4cde-9461-6e993dcc0725-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-7683f606-69d94fc9c9-9npx5\" (UID: \"49fe9781-4bf8-4cde-9461-6e993dcc0725\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7683f606-69d94fc9c9-9npx5" Nov 03 02:01:21.152256 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:01:21.152092 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5sqg\" (UniqueName: \"kubernetes.io/projected/49fe9781-4bf8-4cde-9461-6e993dcc0725-kube-api-access-z5sqg\") pod \"payload-pre-processing-e2e-migrate-7683f606-69d94fc9c9-9npx5\" (UID: \"49fe9781-4bf8-4cde-9461-6e993dcc0725\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7683f606-69d94fc9c9-9npx5" Nov 03 02:01:21.152847 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:01:21.152825 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49fe9781-4bf8-4cde-9461-6e993dcc0725-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-7683f606-69d94fc9c9-9npx5\" (UID: \"49fe9781-4bf8-4cde-9461-6e993dcc0725\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7683f606-69d94fc9c9-9npx5" Nov 03 02:01:21.206405 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:01:21.206300 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5sqg\" (UniqueName: \"kubernetes.io/projected/49fe9781-4bf8-4cde-9461-6e993dcc0725-kube-api-access-z5sqg\") pod \"payload-pre-processing-e2e-migrate-7683f606-69d94fc9c9-9npx5\" (UID: \"49fe9781-4bf8-4cde-9461-6e993dcc0725\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7683f606-69d94fc9c9-9npx5" Nov 03 02:01:21.248180 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:01:21.248128 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7683f606-69d94fc9c9-9npx5" Nov 03 02:01:21.612755 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:01:21.612715 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-7683f606-69d94fc9c9-9npx5"] Nov 03 02:01:21.620249 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:01:21.620216 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49fe9781_4bf8_4cde_9461_6e993dcc0725.slice/crio-5b0d7fd53d89ffa9afe28ac57307871c3f69843e2ff4987eb18af6c44e16255f WatchSource:0}: Error finding container 5b0d7fd53d89ffa9afe28ac57307871c3f69843e2ff4987eb18af6c44e16255f: Status 404 returned error can't find the container with id 5b0d7fd53d89ffa9afe28ac57307871c3f69843e2ff4987eb18af6c44e16255f Nov 03 02:01:22.149070 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:01:22.149035 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7683f606-69d94fc9c9-9npx5" event={"ID":"49fe9781-4bf8-4cde-9461-6e993dcc0725","Type":"ContainerStarted","Data":"5b0d7fd53d89ffa9afe28ac57307871c3f69843e2ff4987eb18af6c44e16255f"} Nov 03 02:01:23.911087 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:01:23.911046 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:01:24.533209 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:01:24.533134 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:01:25.164336 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:01:25.164300 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7683f606-69d94fc9c9-9npx5" event={"ID":"49fe9781-4bf8-4cde-9461-6e993dcc0725","Type":"ContainerStarted","Data":"4060ffa7fb169399f6029b0d3740a87a06b2548b4442f6b31a13f0b450f77892"} Nov 03 02:01:25.164723 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:01:25.164422 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7683f606-69d94fc9c9-9npx5" Nov 03 02:01:25.270538 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:01:25.270300 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7683f606-69d94fc9c9-9npx5" podStartSLOduration=1.881635247 podStartE2EDuration="5.270287518s" totalImagesPullingTime="3.388652271s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:01:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:01:25.269556106 +0000 UTC m=+562.528754259" watchObservedRunningTime="2026-11-03 02:01:25.270287518 +0000 UTC m=+562.529485650" Nov 03 02:01:32.853164 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:01:32.853133 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:01:36.171669 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:01:36.171639 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7683f606-69d94fc9c9-9npx5" Nov 03 02:01:42.361535 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:01:42.361501 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:01:49.054555 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:01:49.054514 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ac43c49-crt5l"] Nov 03 02:01:49.057710 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:01:49.057687 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ac43c49-crt5l" Nov 03 02:01:49.106675 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:01:49.106644 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:01:49.124856 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:01:49.124822 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6610fe35-60c6-4a10-ab6b-2ffc9acbd904-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-6ac43c49-crt5l\" (UID: \"6610fe35-60c6-4a10-ab6b-2ffc9acbd904\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ac43c49-crt5l" Nov 03 02:01:49.125029 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:01:49.124862 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6610fe35-60c6-4a10-ab6b-2ffc9acbd904-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-6ac43c49-crt5l\" (UID: \"6610fe35-60c6-4a10-ab6b-2ffc9acbd904\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ac43c49-crt5l" Nov 03 02:01:49.125029 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:01:49.124889 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgp5w\" (UniqueName: \"kubernetes.io/projected/6610fe35-60c6-4a10-ab6b-2ffc9acbd904-kube-api-access-lgp5w\") pod \"maas-api-revoke-keys-e2e-ait-6ac43c49-crt5l\" (UID: \"6610fe35-60c6-4a10-ab6b-2ffc9acbd904\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ac43c49-crt5l" Nov 03 02:01:49.153326 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:01:49.153281 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:01:49.153326 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:01:49.153297 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:01:49.153546 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:01:49.153315 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:01:49.154376 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:01:49.154338 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ac43c49-crt5l"] Nov 03 02:01:49.225534 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:01:49.225492 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6610fe35-60c6-4a10-ab6b-2ffc9acbd904-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-6ac43c49-crt5l\" (UID: \"6610fe35-60c6-4a10-ab6b-2ffc9acbd904\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ac43c49-crt5l" Nov 03 02:01:49.225534 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:01:49.225542 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgp5w\" (UniqueName: \"kubernetes.io/projected/6610fe35-60c6-4a10-ab6b-2ffc9acbd904-kube-api-access-lgp5w\") pod \"maas-api-revoke-keys-e2e-ait-6ac43c49-crt5l\" (UID: \"6610fe35-60c6-4a10-ab6b-2ffc9acbd904\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ac43c49-crt5l" Nov 03 02:01:49.225804 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:01:49.225631 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6610fe35-60c6-4a10-ab6b-2ffc9acbd904-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-6ac43c49-crt5l\" (UID: \"6610fe35-60c6-4a10-ab6b-2ffc9acbd904\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ac43c49-crt5l" Nov 03 02:01:49.226164 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:01:49.226139 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6610fe35-60c6-4a10-ab6b-2ffc9acbd904-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-6ac43c49-crt5l\" (UID: \"6610fe35-60c6-4a10-ab6b-2ffc9acbd904\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ac43c49-crt5l" Nov 03 02:01:49.228018 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:01:49.227991 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6610fe35-60c6-4a10-ab6b-2ffc9acbd904-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-6ac43c49-crt5l\" (UID: \"6610fe35-60c6-4a10-ab6b-2ffc9acbd904\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ac43c49-crt5l" Nov 03 02:01:49.280204 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:01:49.280174 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgp5w\" (UniqueName: \"kubernetes.io/projected/6610fe35-60c6-4a10-ab6b-2ffc9acbd904-kube-api-access-lgp5w\") pod \"maas-api-revoke-keys-e2e-ait-6ac43c49-crt5l\" (UID: \"6610fe35-60c6-4a10-ab6b-2ffc9acbd904\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ac43c49-crt5l" Nov 03 02:01:49.370674 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:01:49.370571 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ac43c49-crt5l" Nov 03 02:01:49.738056 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:01:49.737907 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ac43c49-crt5l"] Nov 03 02:01:49.745380 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:01:49.745323 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6610fe35_60c6_4a10_ab6b_2ffc9acbd904.slice/crio-f000eaa20361731bf8681309a68e726812bc5653cb3c4b9c713a75145ebad4ba WatchSource:0}: Error finding container f000eaa20361731bf8681309a68e726812bc5653cb3c4b9c713a75145ebad4ba: Status 404 returned error can't find the container with id f000eaa20361731bf8681309a68e726812bc5653cb3c4b9c713a75145ebad4ba Nov 03 02:01:50.268020 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:01:50.267984 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ac43c49-crt5l" event={"ID":"6610fe35-60c6-4a10-ab6b-2ffc9acbd904","Type":"ContainerStarted","Data":"f000eaa20361731bf8681309a68e726812bc5653cb3c4b9c713a75145ebad4ba"} Nov 03 02:01:52.344812 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:01:52.344776 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:01:53.913816 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:01:53.913773 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:01:55.290003 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:01:55.289914 5605 generic.go:356] "Generic (PLEG): container finished" podID="6610fe35-60c6-4a10-ab6b-2ffc9acbd904" containerID="8d63b207c57462451c6bbf4621064f18b1cee3c22077b418b70864741b0f236c" exitCode=0 Nov 03 02:01:55.290003 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:01:55.289957 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ac43c49-crt5l" event={"ID":"6610fe35-60c6-4a10-ab6b-2ffc9acbd904","Type":"ContainerDied","Data":"8d63b207c57462451c6bbf4621064f18b1cee3c22077b418b70864741b0f236c"} Nov 03 02:01:56.417916 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:01:56.417886 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ac43c49-crt5l" Nov 03 02:01:56.491836 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:01:56.491805 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6610fe35-60c6-4a10-ab6b-2ffc9acbd904-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6610fe35-60c6-4a10-ab6b-2ffc9acbd904-maas-api-openshift-ca\") pod \"6610fe35-60c6-4a10-ab6b-2ffc9acbd904\" (UID: \"6610fe35-60c6-4a10-ab6b-2ffc9acbd904\") " Nov 03 02:01:56.491982 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:01:56.491851 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6610fe35-60c6-4a10-ab6b-2ffc9acbd904-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6610fe35-60c6-4a10-ab6b-2ffc9acbd904-maas-api-xks-ca\") pod \"6610fe35-60c6-4a10-ab6b-2ffc9acbd904\" (UID: \"6610fe35-60c6-4a10-ab6b-2ffc9acbd904\") " Nov 03 02:01:56.491982 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:01:56.491903 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6610fe35-60c6-4a10-ab6b-2ffc9acbd904-kube-api-access-lgp5w\" (UniqueName: \"kubernetes.io/projected/6610fe35-60c6-4a10-ab6b-2ffc9acbd904-kube-api-access-lgp5w\") pod \"6610fe35-60c6-4a10-ab6b-2ffc9acbd904\" (UID: \"6610fe35-60c6-4a10-ab6b-2ffc9acbd904\") " Nov 03 02:01:56.492164 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:01:56.492138 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6610fe35-60c6-4a10-ab6b-2ffc9acbd904-maas-api-openshift-ca" pod "6610fe35-60c6-4a10-ab6b-2ffc9acbd904" (UID: "6610fe35-60c6-4a10-ab6b-2ffc9acbd904"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:01:56.494103 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:01:56.494078 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6610fe35-60c6-4a10-ab6b-2ffc9acbd904-maas-api-xks-ca" pod "6610fe35-60c6-4a10-ab6b-2ffc9acbd904" (UID: "6610fe35-60c6-4a10-ab6b-2ffc9acbd904"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:01:56.494206 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:01:56.494137 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6610fe35-60c6-4a10-ab6b-2ffc9acbd904-kube-api-access-lgp5w" pod "6610fe35-60c6-4a10-ab6b-2ffc9acbd904" (UID: "6610fe35-60c6-4a10-ab6b-2ffc9acbd904"). InnerVolumeSpecName "kube-api-access-lgp5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:01:56.592718 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:01:56.592685 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6610fe35-60c6-4a10-ab6b-2ffc9acbd904-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:01:56.592718 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:01:56.592714 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6610fe35-60c6-4a10-ab6b-2ffc9acbd904-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:01:56.592718 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:01:56.592723 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgp5w\" (UniqueName: \"kubernetes.io/projected/6610fe35-60c6-4a10-ab6b-2ffc9acbd904-kube-api-access-lgp5w\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:01:57.299132 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:01:57.299105 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ac43c49-crt5l" Nov 03 02:01:57.299288 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:01:57.299132 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ac43c49-crt5l" event={"ID":"6610fe35-60c6-4a10-ab6b-2ffc9acbd904","Type":"ContainerDied","Data":"f000eaa20361731bf8681309a68e726812bc5653cb3c4b9c713a75145ebad4ba"} Nov 03 02:01:57.299288 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:01:57.299185 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f000eaa20361731bf8681309a68e726812bc5653cb3c4b9c713a75145ebad4ba" Nov 03 02:02:04.673985 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:04.673943 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aead6b17-scjlw"] Nov 03 02:02:04.674605 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:04.674452 5605 cpu_manager.go:414] "containerMap: removing container" podUID="6610fe35-60c6-4a10-ab6b-2ffc9acbd904" containerName="revoke-keys" Nov 03 02:02:04.674605 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:04.674472 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6610fe35-60c6-4a10-ab6b-2ffc9acbd904" containerName="revoke-keys" Nov 03 02:02:04.674605 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:04.674567 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="6610fe35-60c6-4a10-ab6b-2ffc9acbd904" containerName="revoke-keys" Nov 03 02:02:04.680912 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:04.680888 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aead6b17-scjlw" Nov 03 02:02:04.730685 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:04.730598 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aead6b17-scjlw"] Nov 03 02:02:04.773323 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:04.773292 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hb6n\" (UniqueName: \"kubernetes.io/projected/1dfb5119-fdfb-4bf9-9722-fa3e5158af5f-kube-api-access-2hb6n\") pod \"maas-api-revoke-keys-e2e-ait-aead6b17-scjlw\" (UID: \"1dfb5119-fdfb-4bf9-9722-fa3e5158af5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aead6b17-scjlw" Nov 03 02:02:04.773523 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:04.773332 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dfb5119-fdfb-4bf9-9722-fa3e5158af5f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-aead6b17-scjlw\" (UID: \"1dfb5119-fdfb-4bf9-9722-fa3e5158af5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aead6b17-scjlw" Nov 03 02:02:04.773523 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:04.773383 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dfb5119-fdfb-4bf9-9722-fa3e5158af5f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-aead6b17-scjlw\" (UID: \"1dfb5119-fdfb-4bf9-9722-fa3e5158af5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aead6b17-scjlw" Nov 03 02:02:04.775919 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:04.775899 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:02:04.775919 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:04.775905 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:02:04.776330 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:04.776316 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:02:04.776442 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:04.776424 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:02:04.874165 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:04.874128 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hb6n\" (UniqueName: \"kubernetes.io/projected/1dfb5119-fdfb-4bf9-9722-fa3e5158af5f-kube-api-access-2hb6n\") pod \"maas-api-revoke-keys-e2e-ait-aead6b17-scjlw\" (UID: \"1dfb5119-fdfb-4bf9-9722-fa3e5158af5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aead6b17-scjlw" Nov 03 02:02:04.874165 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:04.874165 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dfb5119-fdfb-4bf9-9722-fa3e5158af5f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-aead6b17-scjlw\" (UID: \"1dfb5119-fdfb-4bf9-9722-fa3e5158af5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aead6b17-scjlw" Nov 03 02:02:04.874446 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:04.874218 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dfb5119-fdfb-4bf9-9722-fa3e5158af5f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-aead6b17-scjlw\" (UID: \"1dfb5119-fdfb-4bf9-9722-fa3e5158af5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aead6b17-scjlw" Nov 03 02:02:04.874869 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:04.874843 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dfb5119-fdfb-4bf9-9722-fa3e5158af5f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-aead6b17-scjlw\" (UID: \"1dfb5119-fdfb-4bf9-9722-fa3e5158af5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aead6b17-scjlw" Nov 03 02:02:04.876771 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:04.876752 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dfb5119-fdfb-4bf9-9722-fa3e5158af5f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-aead6b17-scjlw\" (UID: \"1dfb5119-fdfb-4bf9-9722-fa3e5158af5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aead6b17-scjlw" Nov 03 02:02:04.927004 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:04.926938 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hb6n\" (UniqueName: \"kubernetes.io/projected/1dfb5119-fdfb-4bf9-9722-fa3e5158af5f-kube-api-access-2hb6n\") pod \"maas-api-revoke-keys-e2e-ait-aead6b17-scjlw\" (UID: \"1dfb5119-fdfb-4bf9-9722-fa3e5158af5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aead6b17-scjlw" Nov 03 02:02:04.993184 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:04.993152 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aead6b17-scjlw" Nov 03 02:02:05.357038 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:05.357002 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aead6b17-scjlw"] Nov 03 02:02:05.363808 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:02:05.363781 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dfb5119_fdfb_4bf9_9722_fa3e5158af5f.slice/crio-511dac2e19b6f2e3e4c9964af19f2e6360b280815719f0eaae181d29c1bdd22e WatchSource:0}: Error finding container 511dac2e19b6f2e3e4c9964af19f2e6360b280815719f0eaae181d29c1bdd22e: Status 404 returned error can't find the container with id 511dac2e19b6f2e3e4c9964af19f2e6360b280815719f0eaae181d29c1bdd22e Nov 03 02:02:06.339672 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:06.339634 5605 generic.go:356] "Generic (PLEG): container finished" podID="1dfb5119-fdfb-4bf9-9722-fa3e5158af5f" containerID="f58061b775986f9ae6ff279a95a1f849fcc81e40c5682ae17d7c0059b6c3e12e" exitCode=0 Nov 03 02:02:06.340059 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:06.339716 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aead6b17-scjlw" event={"ID":"1dfb5119-fdfb-4bf9-9722-fa3e5158af5f","Type":"ContainerDied","Data":"f58061b775986f9ae6ff279a95a1f849fcc81e40c5682ae17d7c0059b6c3e12e"} Nov 03 02:02:06.340059 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:06.339769 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aead6b17-scjlw" event={"ID":"1dfb5119-fdfb-4bf9-9722-fa3e5158af5f","Type":"ContainerStarted","Data":"511dac2e19b6f2e3e4c9964af19f2e6360b280815719f0eaae181d29c1bdd22e"} Nov 03 02:02:07.468606 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:07.468582 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aead6b17-scjlw" Nov 03 02:02:07.497650 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:07.497621 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dfb5119-fdfb-4bf9-9722-fa3e5158af5f-kube-api-access-2hb6n\" (UniqueName: \"kubernetes.io/projected/1dfb5119-fdfb-4bf9-9722-fa3e5158af5f-kube-api-access-2hb6n\") pod \"1dfb5119-fdfb-4bf9-9722-fa3e5158af5f\" (UID: \"1dfb5119-fdfb-4bf9-9722-fa3e5158af5f\") " Nov 03 02:02:07.497815 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:07.497722 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dfb5119-fdfb-4bf9-9722-fa3e5158af5f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dfb5119-fdfb-4bf9-9722-fa3e5158af5f-maas-api-openshift-ca\") pod \"1dfb5119-fdfb-4bf9-9722-fa3e5158af5f\" (UID: \"1dfb5119-fdfb-4bf9-9722-fa3e5158af5f\") " Nov 03 02:02:07.497815 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:07.497749 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1dfb5119-fdfb-4bf9-9722-fa3e5158af5f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dfb5119-fdfb-4bf9-9722-fa3e5158af5f-maas-api-xks-ca\") pod \"1dfb5119-fdfb-4bf9-9722-fa3e5158af5f\" (UID: \"1dfb5119-fdfb-4bf9-9722-fa3e5158af5f\") " Nov 03 02:02:07.498066 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:07.498037 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dfb5119-fdfb-4bf9-9722-fa3e5158af5f-maas-api-openshift-ca" pod "1dfb5119-fdfb-4bf9-9722-fa3e5158af5f" (UID: "1dfb5119-fdfb-4bf9-9722-fa3e5158af5f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:02:07.499695 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:07.499668 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dfb5119-fdfb-4bf9-9722-fa3e5158af5f-maas-api-xks-ca" pod "1dfb5119-fdfb-4bf9-9722-fa3e5158af5f" (UID: "1dfb5119-fdfb-4bf9-9722-fa3e5158af5f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:02:07.499819 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:07.499802 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dfb5119-fdfb-4bf9-9722-fa3e5158af5f-kube-api-access-2hb6n" pod "1dfb5119-fdfb-4bf9-9722-fa3e5158af5f" (UID: "1dfb5119-fdfb-4bf9-9722-fa3e5158af5f"). InnerVolumeSpecName "kube-api-access-2hb6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:02:07.599002 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:07.598922 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hb6n\" (UniqueName: \"kubernetes.io/projected/1dfb5119-fdfb-4bf9-9722-fa3e5158af5f-kube-api-access-2hb6n\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:07.599002 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:07.598965 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dfb5119-fdfb-4bf9-9722-fa3e5158af5f-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:07.599002 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:07.598982 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dfb5119-fdfb-4bf9-9722-fa3e5158af5f-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:07.627073 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:07.627040 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-7683f606-69d94fc9c9-9npx5"] Nov 03 02:02:08.349220 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:08.349191 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aead6b17-scjlw" Nov 03 02:02:08.349220 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:08.349207 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aead6b17-scjlw" event={"ID":"1dfb5119-fdfb-4bf9-9722-fa3e5158af5f","Type":"ContainerDied","Data":"511dac2e19b6f2e3e4c9964af19f2e6360b280815719f0eaae181d29c1bdd22e"} Nov 03 02:02:08.349471 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:08.349248 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="511dac2e19b6f2e3e4c9964af19f2e6360b280815719f0eaae181d29c1bdd22e" Nov 03 02:02:10.199212 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:10.196611 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ac43c49-crt5l"] Nov 03 02:02:10.241984 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:10.241939 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ac43c49-crt5l"] Nov 03 02:02:10.639682 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:10.639648 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fbf24ed8-jlrxd"] Nov 03 02:02:10.639998 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:10.639984 5605 cpu_manager.go:414] "containerMap: removing container" podUID="1dfb5119-fdfb-4bf9-9722-fa3e5158af5f" containerName="revoke-keys" Nov 03 02:02:10.640051 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:10.639999 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dfb5119-fdfb-4bf9-9722-fa3e5158af5f" containerName="revoke-keys" Nov 03 02:02:10.640084 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:10.640071 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dfb5119-fdfb-4bf9-9722-fa3e5158af5f" containerName="revoke-keys" Nov 03 02:02:10.643121 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:10.643104 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fbf24ed8-jlrxd" Nov 03 02:02:10.696383 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:10.696332 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fbf24ed8-jlrxd"] Nov 03 02:02:10.729643 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:10.729611 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd9h8\" (UniqueName: \"kubernetes.io/projected/82d1df7e-29ab-422d-94ba-949b56cd4a95-kube-api-access-qd9h8\") pod \"maas-api-revoke-keys-e2e-derive-fbf24ed8-jlrxd\" (UID: \"82d1df7e-29ab-422d-94ba-949b56cd4a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fbf24ed8-jlrxd" Nov 03 02:02:10.729807 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:10.729669 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82d1df7e-29ab-422d-94ba-949b56cd4a95-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-fbf24ed8-jlrxd\" (UID: \"82d1df7e-29ab-422d-94ba-949b56cd4a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fbf24ed8-jlrxd" Nov 03 02:02:10.729807 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:10.729787 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82d1df7e-29ab-422d-94ba-949b56cd4a95-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-fbf24ed8-jlrxd\" (UID: \"82d1df7e-29ab-422d-94ba-949b56cd4a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fbf24ed8-jlrxd" Nov 03 02:02:10.741626 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:10.741607 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:02:10.741718 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:10.741643 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:02:10.741718 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:10.741643 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:02:10.741718 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:10.741659 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:02:10.831170 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:10.831135 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82d1df7e-29ab-422d-94ba-949b56cd4a95-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-fbf24ed8-jlrxd\" (UID: \"82d1df7e-29ab-422d-94ba-949b56cd4a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fbf24ed8-jlrxd" Nov 03 02:02:10.831329 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:10.831184 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd9h8\" (UniqueName: \"kubernetes.io/projected/82d1df7e-29ab-422d-94ba-949b56cd4a95-kube-api-access-qd9h8\") pod \"maas-api-revoke-keys-e2e-derive-fbf24ed8-jlrxd\" (UID: \"82d1df7e-29ab-422d-94ba-949b56cd4a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fbf24ed8-jlrxd" Nov 03 02:02:10.831329 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:10.831213 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82d1df7e-29ab-422d-94ba-949b56cd4a95-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-fbf24ed8-jlrxd\" (UID: \"82d1df7e-29ab-422d-94ba-949b56cd4a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fbf24ed8-jlrxd" Nov 03 02:02:10.831828 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:10.831805 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82d1df7e-29ab-422d-94ba-949b56cd4a95-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-fbf24ed8-jlrxd\" (UID: \"82d1df7e-29ab-422d-94ba-949b56cd4a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fbf24ed8-jlrxd" Nov 03 02:02:10.833511 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:10.833493 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82d1df7e-29ab-422d-94ba-949b56cd4a95-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-fbf24ed8-jlrxd\" (UID: \"82d1df7e-29ab-422d-94ba-949b56cd4a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fbf24ed8-jlrxd" Nov 03 02:02:10.848655 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:10.848629 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6610fe35-60c6-4a10-ab6b-2ffc9acbd904" path="/var/lib/kubelet/pods/6610fe35-60c6-4a10-ab6b-2ffc9acbd904/volumes" Nov 03 02:02:10.884477 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:10.884450 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd9h8\" (UniqueName: \"kubernetes.io/projected/82d1df7e-29ab-422d-94ba-949b56cd4a95-kube-api-access-qd9h8\") pod \"maas-api-revoke-keys-e2e-derive-fbf24ed8-jlrxd\" (UID: \"82d1df7e-29ab-422d-94ba-949b56cd4a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fbf24ed8-jlrxd" Nov 03 02:02:10.955740 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:10.955659 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fbf24ed8-jlrxd" Nov 03 02:02:11.319481 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:11.319449 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fbf24ed8-jlrxd"] Nov 03 02:02:11.326687 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:02:11.326658 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82d1df7e_29ab_422d_94ba_949b56cd4a95.slice/crio-9727ed1b51881cf52bb885163a81d896ef7c8e14edf3ef841811502f1689157a WatchSource:0}: Error finding container 9727ed1b51881cf52bb885163a81d896ef7c8e14edf3ef841811502f1689157a: Status 404 returned error can't find the container with id 9727ed1b51881cf52bb885163a81d896ef7c8e14edf3ef841811502f1689157a Nov 03 02:02:11.362397 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:11.362349 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fbf24ed8-jlrxd" event={"ID":"82d1df7e-29ab-422d-94ba-949b56cd4a95","Type":"ContainerStarted","Data":"9727ed1b51881cf52bb885163a81d896ef7c8e14edf3ef841811502f1689157a"} Nov 03 02:02:11.691779 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:11.691747 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v"] Nov 03 02:02:11.695321 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:11.695297 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v" Nov 03 02:02:11.740012 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:11.739972 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f-istio-token\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v\" (UID: \"3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v" Nov 03 02:02:11.740201 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:11.740022 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f-istio-envoy\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v\" (UID: \"3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v" Nov 03 02:02:11.740201 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:11.740113 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f-workload-socket\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v\" (UID: \"3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v" Nov 03 02:02:11.740201 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:11.740158 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f-workload-certs\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v\" (UID: \"3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v" Nov 03 02:02:11.740201 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:11.740183 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f-credential-socket\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v\" (UID: \"3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v" Nov 03 02:02:11.740201 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:11.740200 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x7qx\" (UniqueName: \"kubernetes.io/projected/3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f-kube-api-access-4x7qx\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v\" (UID: \"3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v" Nov 03 02:02:11.740521 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:11.740222 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f-istio-data\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v\" (UID: \"3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v" Nov 03 02:02:11.740521 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:11.740251 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f-istio-podinfo\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v\" (UID: \"3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v" Nov 03 02:02:11.740521 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:11.740287 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f-istiod-ca-cert\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v\" (UID: \"3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v" Nov 03 02:02:11.744143 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:11.744108 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-521f1f-openshift-default-dockercfg-c9pnp\"" Nov 03 02:02:11.768823 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:11.768783 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v"] Nov 03 02:02:11.848687 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:11.848593 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f-istio-token\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v\" (UID: \"3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v" Nov 03 02:02:11.848687 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:11.848655 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f-istio-envoy\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v\" (UID: \"3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v" Nov 03 02:02:11.848917 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:11.848737 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f-workload-socket\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v\" (UID: \"3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v" Nov 03 02:02:11.848917 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:11.848756 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f-workload-certs\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v\" (UID: \"3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v" Nov 03 02:02:11.848917 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:11.848780 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f-credential-socket\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v\" (UID: \"3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v" Nov 03 02:02:11.848917 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:11.848797 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x7qx\" (UniqueName: \"kubernetes.io/projected/3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f-kube-api-access-4x7qx\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v\" (UID: \"3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v" Nov 03 02:02:11.848917 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:11.848830 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f-istio-data\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v\" (UID: \"3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v" Nov 03 02:02:11.848917 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:11.848865 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f-istio-podinfo\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v\" (UID: \"3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v" Nov 03 02:02:11.848917 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:11.848906 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f-istiod-ca-cert\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v\" (UID: \"3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v" Nov 03 02:02:11.849697 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:11.849628 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f-istiod-ca-cert\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v\" (UID: \"3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v" Nov 03 02:02:11.852578 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:11.852520 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f-credential-socket\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v\" (UID: \"3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v" Nov 03 02:02:11.852768 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:11.852736 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f-workload-socket\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v\" (UID: \"3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v" Nov 03 02:02:11.853111 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:11.852879 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f-workload-certs\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v\" (UID: \"3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v" Nov 03 02:02:11.853111 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:11.853073 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f-istio-data\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v\" (UID: \"3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v" Nov 03 02:02:11.857227 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:11.857199 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f-istio-envoy\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v\" (UID: \"3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v" Nov 03 02:02:11.858708 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:11.858682 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f-istio-podinfo\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v\" (UID: \"3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v" Nov 03 02:02:11.904831 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:11.904792 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f-istio-token\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v\" (UID: \"3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v" Nov 03 02:02:11.905131 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:11.905111 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x7qx\" (UniqueName: \"kubernetes.io/projected/3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f-kube-api-access-4x7qx\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v\" (UID: \"3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v" Nov 03 02:02:12.008610 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:12.008573 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v" Nov 03 02:02:12.368221 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:12.368185 5605 generic.go:356] "Generic (PLEG): container finished" podID="82d1df7e-29ab-422d-94ba-949b56cd4a95" containerID="995fc17348a6f886bf5947bd7d66741767bf40aa7de54355ae6521d6b437c8cd" exitCode=0 Nov 03 02:02:12.368629 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:12.368232 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fbf24ed8-jlrxd" event={"ID":"82d1df7e-29ab-422d-94ba-949b56cd4a95","Type":"ContainerDied","Data":"995fc17348a6f886bf5947bd7d66741767bf40aa7de54355ae6521d6b437c8cd"} Nov 03 02:02:12.425579 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:12.425547 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v"] Nov 03 02:02:12.431509 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:02:12.431483 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e97f1e8_d02b_47f5_9eed_4cab3cdcf44f.slice/crio-37f115b4fb3baf3eb04bbb70e7a3e34128b2700e1c398921b35dfbb106928b01 WatchSource:0}: Error finding container 37f115b4fb3baf3eb04bbb70e7a3e34128b2700e1c398921b35dfbb106928b01: Status 404 returned error can't find the container with id 37f115b4fb3baf3eb04bbb70e7a3e34128b2700e1c398921b35dfbb106928b01 Nov 03 02:02:12.433662 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:12.433630 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:02:12.433739 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:12.433695 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:02:12.433739 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:12.433723 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:02:12.627591 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:12.627547 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7683f606-69d94fc9c9-9npx5" podUID="49fe9781-4bf8-4cde-9461-6e993dcc0725" containerName="payload-pre-processing" containerID="cri-o://4060ffa7fb169399f6029b0d3740a87a06b2548b4442f6b31a13f0b450f77892" gracePeriod=25 Nov 03 02:02:12.874974 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:12.874949 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7683f606-69d94fc9c9-9npx5" Nov 03 02:02:12.957988 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:12.957953 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49fe9781-4bf8-4cde-9461-6e993dcc0725-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49fe9781-4bf8-4cde-9461-6e993dcc0725-plugins-config-volume\") pod \"49fe9781-4bf8-4cde-9461-6e993dcc0725\" (UID: \"49fe9781-4bf8-4cde-9461-6e993dcc0725\") " Nov 03 02:02:12.958155 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:12.958057 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49fe9781-4bf8-4cde-9461-6e993dcc0725-kube-api-access-z5sqg\" (UniqueName: \"kubernetes.io/projected/49fe9781-4bf8-4cde-9461-6e993dcc0725-kube-api-access-z5sqg\") pod \"49fe9781-4bf8-4cde-9461-6e993dcc0725\" (UID: \"49fe9781-4bf8-4cde-9461-6e993dcc0725\") " Nov 03 02:02:12.958394 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:12.958339 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49fe9781-4bf8-4cde-9461-6e993dcc0725-plugins-config-volume" pod "49fe9781-4bf8-4cde-9461-6e993dcc0725" (UID: "49fe9781-4bf8-4cde-9461-6e993dcc0725"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:02:12.960110 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:12.960091 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49fe9781-4bf8-4cde-9461-6e993dcc0725-kube-api-access-z5sqg" pod "49fe9781-4bf8-4cde-9461-6e993dcc0725" (UID: "49fe9781-4bf8-4cde-9461-6e993dcc0725"). InnerVolumeSpecName "kube-api-access-z5sqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:02:13.058796 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:13.058769 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5sqg\" (UniqueName: \"kubernetes.io/projected/49fe9781-4bf8-4cde-9461-6e993dcc0725-kube-api-access-z5sqg\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:13.058796 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:13.058794 5605 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49fe9781-4bf8-4cde-9461-6e993dcc0725-plugins-config-volume\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:13.374511 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:13.374414 5605 generic.go:356] "Generic (PLEG): container finished" podID="49fe9781-4bf8-4cde-9461-6e993dcc0725" containerID="4060ffa7fb169399f6029b0d3740a87a06b2548b4442f6b31a13f0b450f77892" exitCode=0 Nov 03 02:02:13.374511 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:13.374461 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7683f606-69d94fc9c9-9npx5" Nov 03 02:02:13.374511 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:13.374499 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7683f606-69d94fc9c9-9npx5" event={"ID":"49fe9781-4bf8-4cde-9461-6e993dcc0725","Type":"ContainerDied","Data":"4060ffa7fb169399f6029b0d3740a87a06b2548b4442f6b31a13f0b450f77892"} Nov 03 02:02:13.375069 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:13.374555 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7683f606-69d94fc9c9-9npx5" event={"ID":"49fe9781-4bf8-4cde-9461-6e993dcc0725","Type":"ContainerDied","Data":"5b0d7fd53d89ffa9afe28ac57307871c3f69843e2ff4987eb18af6c44e16255f"} Nov 03 02:02:13.375069 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:13.374586 5605 scope.go:122] "RemoveContainer" containerID="4060ffa7fb169399f6029b0d3740a87a06b2548b4442f6b31a13f0b450f77892" Nov 03 02:02:13.376286 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:13.376264 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v" event={"ID":"3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f","Type":"ContainerStarted","Data":"37610810acc57ec8d446db9a2193104731dab7ed2131a1f5df057e3360b7aefc"} Nov 03 02:02:13.376415 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:13.376312 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v" event={"ID":"3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f","Type":"ContainerStarted","Data":"37f115b4fb3baf3eb04bbb70e7a3e34128b2700e1c398921b35dfbb106928b01"} Nov 03 02:02:13.383633 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:13.383601 5605 scope.go:122] "RemoveContainer" containerID="4060ffa7fb169399f6029b0d3740a87a06b2548b4442f6b31a13f0b450f77892" Nov 03 02:02:13.383954 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:02:13.383935 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4060ffa7fb169399f6029b0d3740a87a06b2548b4442f6b31a13f0b450f77892\": container with ID starting with 4060ffa7fb169399f6029b0d3740a87a06b2548b4442f6b31a13f0b450f77892 not found: ID does not exist" containerID="4060ffa7fb169399f6029b0d3740a87a06b2548b4442f6b31a13f0b450f77892" Nov 03 02:02:13.384025 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:13.383962 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4060ffa7fb169399f6029b0d3740a87a06b2548b4442f6b31a13f0b450f77892"} err="failed to get container status \"4060ffa7fb169399f6029b0d3740a87a06b2548b4442f6b31a13f0b450f77892\": rpc error: code = NotFound desc = could not find container \"4060ffa7fb169399f6029b0d3740a87a06b2548b4442f6b31a13f0b450f77892\": container with ID starting with 4060ffa7fb169399f6029b0d3740a87a06b2548b4442f6b31a13f0b450f77892 not found: ID does not exist" Nov 03 02:02:13.501878 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:13.501853 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fbf24ed8-jlrxd" Nov 03 02:02:13.526641 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:13.526611 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-7683f606-69d94fc9c9-9npx5"] Nov 03 02:02:13.528762 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:13.528740 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-7683f606-69d94fc9c9-9npx5"] Nov 03 02:02:13.563432 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:13.563406 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82d1df7e-29ab-422d-94ba-949b56cd4a95-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82d1df7e-29ab-422d-94ba-949b56cd4a95-maas-api-xks-ca\") pod \"82d1df7e-29ab-422d-94ba-949b56cd4a95\" (UID: \"82d1df7e-29ab-422d-94ba-949b56cd4a95\") " Nov 03 02:02:13.563549 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:13.563450 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82d1df7e-29ab-422d-94ba-949b56cd4a95-kube-api-access-qd9h8\" (UniqueName: \"kubernetes.io/projected/82d1df7e-29ab-422d-94ba-949b56cd4a95-kube-api-access-qd9h8\") pod \"82d1df7e-29ab-422d-94ba-949b56cd4a95\" (UID: \"82d1df7e-29ab-422d-94ba-949b56cd4a95\") " Nov 03 02:02:13.563549 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:13.563499 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82d1df7e-29ab-422d-94ba-949b56cd4a95-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82d1df7e-29ab-422d-94ba-949b56cd4a95-maas-api-openshift-ca\") pod \"82d1df7e-29ab-422d-94ba-949b56cd4a95\" (UID: \"82d1df7e-29ab-422d-94ba-949b56cd4a95\") " Nov 03 02:02:13.563890 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:13.563865 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82d1df7e-29ab-422d-94ba-949b56cd4a95-maas-api-openshift-ca" pod "82d1df7e-29ab-422d-94ba-949b56cd4a95" (UID: "82d1df7e-29ab-422d-94ba-949b56cd4a95"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:02:13.565430 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:13.565403 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82d1df7e-29ab-422d-94ba-949b56cd4a95-kube-api-access-qd9h8" pod "82d1df7e-29ab-422d-94ba-949b56cd4a95" (UID: "82d1df7e-29ab-422d-94ba-949b56cd4a95"). InnerVolumeSpecName "kube-api-access-qd9h8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:02:13.565534 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:13.565466 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82d1df7e-29ab-422d-94ba-949b56cd4a95-maas-api-xks-ca" pod "82d1df7e-29ab-422d-94ba-949b56cd4a95" (UID: "82d1df7e-29ab-422d-94ba-949b56cd4a95"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:02:13.635938 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:13.635829 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v" podStartSLOduration=2.635814147 podStartE2EDuration="2.635814147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:02:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:02:13.634944893 +0000 UTC m=+610.894143047" watchObservedRunningTime="2026-11-03 02:02:13.635814147 +0000 UTC m=+610.895012303" Nov 03 02:02:13.664782 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:13.664746 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qd9h8\" (UniqueName: \"kubernetes.io/projected/82d1df7e-29ab-422d-94ba-949b56cd4a95-kube-api-access-qd9h8\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:13.664782 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:13.664778 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82d1df7e-29ab-422d-94ba-949b56cd4a95-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:13.664782 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:13.664788 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82d1df7e-29ab-422d-94ba-949b56cd4a95-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:14.009479 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:14.009380 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v" Nov 03 02:02:14.014504 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:14.014481 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v" Nov 03 02:02:14.382621 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:14.382580 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fbf24ed8-jlrxd" Nov 03 02:02:14.382621 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:14.382609 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fbf24ed8-jlrxd" event={"ID":"82d1df7e-29ab-422d-94ba-949b56cd4a95","Type":"ContainerDied","Data":"9727ed1b51881cf52bb885163a81d896ef7c8e14edf3ef841811502f1689157a"} Nov 03 02:02:14.383104 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:14.382672 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9727ed1b51881cf52bb885163a81d896ef7c8e14edf3ef841811502f1689157a" Nov 03 02:02:14.384029 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:14.384008 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v" Nov 03 02:02:14.385101 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:14.385082 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v" Nov 03 02:02:14.759653 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:14.759575 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:02:14.848706 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:14.848677 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49fe9781-4bf8-4cde-9461-6e993dcc0725" path="/var/lib/kubelet/pods/49fe9781-4bf8-4cde-9461-6e993dcc0725/volumes" Nov 03 02:02:23.911409 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:02:23.911349 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:02:27.858617 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:27.858579 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fbf24ed8-jlrxd"] Nov 03 02:02:27.905007 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:27.904968 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fbf24ed8-jlrxd"] Nov 03 02:02:28.847707 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:28.847674 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82d1df7e-29ab-422d-94ba-949b56cd4a95" path="/var/lib/kubelet/pods/82d1df7e-29ab-422d-94ba-949b56cd4a95/volumes" Nov 03 02:02:29.561228 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:29.561194 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-a0e72e-64b874b665-4k4v9"] Nov 03 02:02:29.561757 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:29.561731 5605 cpu_manager.go:414] "containerMap: removing container" podUID="82d1df7e-29ab-422d-94ba-949b56cd4a95" containerName="revoke-keys" Nov 03 02:02:29.561757 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:29.561753 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82d1df7e-29ab-422d-94ba-949b56cd4a95" containerName="revoke-keys" Nov 03 02:02:29.561874 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:29.561786 5605 cpu_manager.go:414] "containerMap: removing container" podUID="49fe9781-4bf8-4cde-9461-6e993dcc0725" containerName="payload-pre-processing" Nov 03 02:02:29.561874 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:29.561796 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49fe9781-4bf8-4cde-9461-6e993dcc0725" containerName="payload-pre-processing" Nov 03 02:02:29.561971 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:29.561882 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="82d1df7e-29ab-422d-94ba-949b56cd4a95" containerName="revoke-keys" Nov 03 02:02:29.561971 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:29.561899 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="49fe9781-4bf8-4cde-9461-6e993dcc0725" containerName="payload-pre-processing" Nov 03 02:02:29.570249 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:29.570221 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-a0e72e-64b874b665-4k4v9" Nov 03 02:02:29.631505 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:29.631477 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-a0e72e-dockercfg-4p25h\"" Nov 03 02:02:29.679648 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:29.679611 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-a0e72e-64b874b665-4k4v9"] Nov 03 02:02:29.703832 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:29.703806 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-a0e72e\"" Nov 03 02:02:29.737252 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:29.737213 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl448\" (UniqueName: \"kubernetes.io/projected/5c524f2c-3641-4719-b906-2434a016e677-kube-api-access-wl448\") pod \"payload-processing-e2e-worker-w3-a0e72e-64b874b665-4k4v9\" (UID: \"5c524f2c-3641-4719-b906-2434a016e677\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-a0e72e-64b874b665-4k4v9" Nov 03 02:02:29.737424 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:29.737292 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c524f2c-3641-4719-b906-2434a016e677-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-a0e72e-64b874b665-4k4v9\" (UID: \"5c524f2c-3641-4719-b906-2434a016e677\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-a0e72e-64b874b665-4k4v9" Nov 03 02:02:29.838758 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:29.838663 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl448\" (UniqueName: \"kubernetes.io/projected/5c524f2c-3641-4719-b906-2434a016e677-kube-api-access-wl448\") pod \"payload-processing-e2e-worker-w3-a0e72e-64b874b665-4k4v9\" (UID: \"5c524f2c-3641-4719-b906-2434a016e677\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-a0e72e-64b874b665-4k4v9" Nov 03 02:02:29.838758 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:29.838735 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c524f2c-3641-4719-b906-2434a016e677-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-a0e72e-64b874b665-4k4v9\" (UID: \"5c524f2c-3641-4719-b906-2434a016e677\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-a0e72e-64b874b665-4k4v9" Nov 03 02:02:29.839508 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:29.839487 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c524f2c-3641-4719-b906-2434a016e677-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-a0e72e-64b874b665-4k4v9\" (UID: \"5c524f2c-3641-4719-b906-2434a016e677\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-a0e72e-64b874b665-4k4v9" Nov 03 02:02:29.892334 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:29.892294 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl448\" (UniqueName: \"kubernetes.io/projected/5c524f2c-3641-4719-b906-2434a016e677-kube-api-access-wl448\") pod \"payload-processing-e2e-worker-w3-a0e72e-64b874b665-4k4v9\" (UID: \"5c524f2c-3641-4719-b906-2434a016e677\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-a0e72e-64b874b665-4k4v9" Nov 03 02:02:29.895933 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:29.895911 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-a0e72e-64b874b665-4k4v9" Nov 03 02:02:30.259931 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:30.259850 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-a0e72e-64b874b665-4k4v9"] Nov 03 02:02:30.266782 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:02:30.266754 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c524f2c_3641_4719_b906_2434a016e677.slice/crio-6be3eaf7c1c4a29cce68576d2e2c350f56799f317e0a122e45c6683a2484eb1c WatchSource:0}: Error finding container 6be3eaf7c1c4a29cce68576d2e2c350f56799f317e0a122e45c6683a2484eb1c: Status 404 returned error can't find the container with id 6be3eaf7c1c4a29cce68576d2e2c350f56799f317e0a122e45c6683a2484eb1c Nov 03 02:02:30.458978 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:30.458931 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-a0e72e-64b874b665-4k4v9" event={"ID":"5c524f2c-3641-4719-b906-2434a016e677","Type":"ContainerStarted","Data":"fb3e3dd8a330b2f38c9c8bd5d2ac7a81845b0246797e3ea2f62dcc5d68681d56"} Nov 03 02:02:30.459163 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:30.458997 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-a0e72e-64b874b665-4k4v9" event={"ID":"5c524f2c-3641-4719-b906-2434a016e677","Type":"ContainerStarted","Data":"6be3eaf7c1c4a29cce68576d2e2c350f56799f317e0a122e45c6683a2484eb1c"} Nov 03 02:02:30.459163 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:30.459129 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-a0e72e-64b874b665-4k4v9" Nov 03 02:02:30.565608 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:30.565554 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-a0e72e-64b874b665-4k4v9" podStartSLOduration=1.56553891 podStartE2EDuration="1.56553891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:02:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:02:30.564276052 +0000 UTC m=+627.823474205" watchObservedRunningTime="2026-11-03 02:02:30.56553891 +0000 UTC m=+627.824737043" Nov 03 02:02:32.830625 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:32.830591 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:02:40.352053 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:40.352018 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aead6b17-scjlw"] Nov 03 02:02:40.398462 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:40.398428 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aead6b17-scjlw"] Nov 03 02:02:40.849300 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:40.849223 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dfb5119-fdfb-4bf9-9722-fa3e5158af5f" path="/var/lib/kubelet/pods/1dfb5119-fdfb-4bf9-9722-fa3e5158af5f/volumes" Nov 03 02:02:41.481676 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:41.481646 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-a0e72e-64b874b665-4k4v9" Nov 03 02:02:48.458666 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:48.458631 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:02:50.547672 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:50.547641 5605 generic.go:356] "Generic (PLEG): container finished" podID="3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f" containerID="37610810acc57ec8d446db9a2193104731dab7ed2131a1f5df057e3360b7aefc" exitCode=0 Nov 03 02:02:50.548089 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:50.547693 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v" event={"ID":"3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f","Type":"ContainerDied","Data":"37610810acc57ec8d446db9a2193104731dab7ed2131a1f5df057e3360b7aefc"} Nov 03 02:02:50.548089 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:50.548075 5605 scope.go:122] "RemoveContainer" containerID="37610810acc57ec8d446db9a2193104731dab7ed2131a1f5df057e3360b7aefc" Nov 03 02:02:50.549441 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:50.549406 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:02:50.549526 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:50.549494 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:02:50.549564 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:50.549537 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:02:51.051268 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:51.051223 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c7d758-d578b44bc-cccmm"] Nov 03 02:02:51.054787 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:51.054764 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-c7d758-d578b44bc-cccmm" Nov 03 02:02:51.106971 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:51.106939 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-c7d758-dockercfg-xp9kq\"" Nov 03 02:02:51.107723 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:51.107682 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-c7d758\"" Nov 03 02:02:51.109123 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:51.109097 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c7d758-d578b44bc-cccmm"] Nov 03 02:02:51.123271 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:51.123234 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46qld\" (UniqueName: \"kubernetes.io/projected/e2eff9e7-e011-46ce-a9e1-306ca1bf5893-kube-api-access-46qld\") pod \"payload-processing-e2e-worker-w4-c7d758-d578b44bc-cccmm\" (UID: \"e2eff9e7-e011-46ce-a9e1-306ca1bf5893\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c7d758-d578b44bc-cccmm" Nov 03 02:02:51.123422 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:51.123380 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2eff9e7-e011-46ce-a9e1-306ca1bf5893-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-c7d758-d578b44bc-cccmm\" (UID: \"e2eff9e7-e011-46ce-a9e1-306ca1bf5893\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c7d758-d578b44bc-cccmm" Nov 03 02:02:51.224059 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:51.224021 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2eff9e7-e011-46ce-a9e1-306ca1bf5893-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-c7d758-d578b44bc-cccmm\" (UID: \"e2eff9e7-e011-46ce-a9e1-306ca1bf5893\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c7d758-d578b44bc-cccmm" Nov 03 02:02:51.224279 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:51.224097 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46qld\" (UniqueName: \"kubernetes.io/projected/e2eff9e7-e011-46ce-a9e1-306ca1bf5893-kube-api-access-46qld\") pod \"payload-processing-e2e-worker-w4-c7d758-d578b44bc-cccmm\" (UID: \"e2eff9e7-e011-46ce-a9e1-306ca1bf5893\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c7d758-d578b44bc-cccmm" Nov 03 02:02:51.224795 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:51.224771 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2eff9e7-e011-46ce-a9e1-306ca1bf5893-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-c7d758-d578b44bc-cccmm\" (UID: \"e2eff9e7-e011-46ce-a9e1-306ca1bf5893\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c7d758-d578b44bc-cccmm" Nov 03 02:02:51.282226 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:51.282191 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46qld\" (UniqueName: \"kubernetes.io/projected/e2eff9e7-e011-46ce-a9e1-306ca1bf5893-kube-api-access-46qld\") pod \"payload-processing-e2e-worker-w4-c7d758-d578b44bc-cccmm\" (UID: \"e2eff9e7-e011-46ce-a9e1-306ca1bf5893\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c7d758-d578b44bc-cccmm" Nov 03 02:02:51.374436 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:51.372536 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-c7d758-d578b44bc-cccmm" Nov 03 02:02:51.576385 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:51.554534 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v" event={"ID":"3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f","Type":"ContainerStarted","Data":"248e50158d844363bcfcf798e336953ea1c002f02f86cec88e4b766b50249beb"} Nov 03 02:02:51.782581 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:51.782495 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c7d758-d578b44bc-cccmm"] Nov 03 02:02:51.789299 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:02:51.789269 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2eff9e7_e011_46ce_a9e1_306ca1bf5893.slice/crio-fe96814f1d1723daa5984a7ea4e4095f4240cec8f84e865414595d1a651c0643 WatchSource:0}: Error finding container fe96814f1d1723daa5984a7ea4e4095f4240cec8f84e865414595d1a651c0643: Status 404 returned error can't find the container with id fe96814f1d1723daa5984a7ea4e4095f4240cec8f84e865414595d1a651c0643 Nov 03 02:02:52.008862 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:52.008826 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v" Nov 03 02:02:52.009173 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:52.009136 5605 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Nov 03 02:02:52.009223 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:52.009198 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v" podUID="3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Nov 03 02:02:52.581383 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:52.578886 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-c7d758-d578b44bc-cccmm" event={"ID":"e2eff9e7-e011-46ce-a9e1-306ca1bf5893","Type":"ContainerStarted","Data":"1ac18cacaa2401eeb837d21b4ebd191b0603a700887195bcb3343877b4b6bd03"} Nov 03 02:02:52.581383 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:52.578943 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-c7d758-d578b44bc-cccmm" event={"ID":"e2eff9e7-e011-46ce-a9e1-306ca1bf5893","Type":"ContainerStarted","Data":"fe96814f1d1723daa5984a7ea4e4095f4240cec8f84e865414595d1a651c0643"} Nov 03 02:02:52.581383 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:52.578990 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-c7d758-d578b44bc-cccmm" Nov 03 02:02:52.685685 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:52.685626 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-c7d758-d578b44bc-cccmm" podStartSLOduration=1.685605154 podStartE2EDuration="1.685605154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:02:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:02:52.684926522 +0000 UTC m=+649.944124699" watchObservedRunningTime="2026-11-03 02:02:52.685605154 +0000 UTC m=+649.944803309" Nov 03 02:02:53.009216 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:53.009121 5605 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Nov 03 02:02:53.009216 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:53.009183 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v" podUID="3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Nov 03 02:02:53.911120 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:02:53.911086 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:02:54.009422 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:54.009383 5605 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Nov 03 02:02:54.009598 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:54.009441 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v" podUID="3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Nov 03 02:02:55.009894 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:55.009858 5605 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Nov 03 02:02:55.010394 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:55.009917 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v" podUID="3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Nov 03 02:02:56.009378 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:56.009317 5605 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Nov 03 02:02:56.009559 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:56.009390 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v" podUID="3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Nov 03 02:02:57.013922 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:57.013876 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v" Nov 03 02:02:57.014406 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:57.014143 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v" Nov 03 02:02:57.014834 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:57.014815 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v" Nov 03 02:02:57.937253 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:02:57.937219 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:03:04.590927 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:04.590896 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-c7d758-d578b44bc-cccmm" Nov 03 02:03:12.014107 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:12.014072 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-521f1f/simulator-w2-521f1f-kserve-5f47b86f6-mgksf"] Nov 03 02:03:12.019055 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:12.019031 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-521f1f/simulator-w2-521f1f-kserve-5f47b86f6-mgksf" Nov 03 02:03:12.036020 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:12.035996 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e1b7fda-0ade-4042-ad63-5115bc66b454-tls-certs\") pod \"simulator-w2-521f1f-kserve-5f47b86f6-mgksf\" (UID: \"6e1b7fda-0ade-4042-ad63-5115bc66b454\") " pod="e2e-models-e2e-worker-w2-521f1f/simulator-w2-521f1f-kserve-5f47b86f6-mgksf" Nov 03 02:03:12.036151 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:12.036026 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52p49\" (UniqueName: \"kubernetes.io/projected/6e1b7fda-0ade-4042-ad63-5115bc66b454-kube-api-access-52p49\") pod \"simulator-w2-521f1f-kserve-5f47b86f6-mgksf\" (UID: \"6e1b7fda-0ade-4042-ad63-5115bc66b454\") " pod="e2e-models-e2e-worker-w2-521f1f/simulator-w2-521f1f-kserve-5f47b86f6-mgksf" Nov 03 02:03:12.036151 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:12.036062 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e1b7fda-0ade-4042-ad63-5115bc66b454-model-cache\") pod \"simulator-w2-521f1f-kserve-5f47b86f6-mgksf\" (UID: \"6e1b7fda-0ade-4042-ad63-5115bc66b454\") " pod="e2e-models-e2e-worker-w2-521f1f/simulator-w2-521f1f-kserve-5f47b86f6-mgksf" Nov 03 02:03:12.036151 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:12.036139 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e1b7fda-0ade-4042-ad63-5115bc66b454-home\") pod \"simulator-w2-521f1f-kserve-5f47b86f6-mgksf\" (UID: \"6e1b7fda-0ade-4042-ad63-5115bc66b454\") " pod="e2e-models-e2e-worker-w2-521f1f/simulator-w2-521f1f-kserve-5f47b86f6-mgksf" Nov 03 02:03:12.036251 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:12.036177 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e1b7fda-0ade-4042-ad63-5115bc66b454-tmp-dir\") pod \"simulator-w2-521f1f-kserve-5f47b86f6-mgksf\" (UID: \"6e1b7fda-0ade-4042-ad63-5115bc66b454\") " pod="e2e-models-e2e-worker-w2-521f1f/simulator-w2-521f1f-kserve-5f47b86f6-mgksf" Nov 03 02:03:12.036251 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:12.036197 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e1b7fda-0ade-4042-ad63-5115bc66b454-dshm\") pod \"simulator-w2-521f1f-kserve-5f47b86f6-mgksf\" (UID: \"6e1b7fda-0ade-4042-ad63-5115bc66b454\") " pod="e2e-models-e2e-worker-w2-521f1f/simulator-w2-521f1f-kserve-5f47b86f6-mgksf" Nov 03 02:03:12.105831 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:12.105795 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-521f1f\"/\"kube-root-ca.crt\"" Nov 03 02:03:12.116946 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:12.116918 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-521f1f\"/\"openshift-service-ca.crt\"" Nov 03 02:03:12.117111 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:12.117069 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-521f1f\"/\"simulator-w2-521f1f-kserve-self-signed-certs\"" Nov 03 02:03:12.117174 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:12.117134 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-521f1f\"/\"default-dockercfg-dkd2g\"" Nov 03 02:03:12.118232 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:12.118211 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-521f1f/simulator-w2-521f1f-kserve-5f47b86f6-mgksf"] Nov 03 02:03:12.136671 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:12.136642 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e1b7fda-0ade-4042-ad63-5115bc66b454-tls-certs\") pod \"simulator-w2-521f1f-kserve-5f47b86f6-mgksf\" (UID: \"6e1b7fda-0ade-4042-ad63-5115bc66b454\") " pod="e2e-models-e2e-worker-w2-521f1f/simulator-w2-521f1f-kserve-5f47b86f6-mgksf" Nov 03 02:03:12.136780 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:12.136675 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52p49\" (UniqueName: \"kubernetes.io/projected/6e1b7fda-0ade-4042-ad63-5115bc66b454-kube-api-access-52p49\") pod \"simulator-w2-521f1f-kserve-5f47b86f6-mgksf\" (UID: \"6e1b7fda-0ade-4042-ad63-5115bc66b454\") " pod="e2e-models-e2e-worker-w2-521f1f/simulator-w2-521f1f-kserve-5f47b86f6-mgksf" Nov 03 02:03:12.136780 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:12.136722 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e1b7fda-0ade-4042-ad63-5115bc66b454-model-cache\") pod \"simulator-w2-521f1f-kserve-5f47b86f6-mgksf\" (UID: \"6e1b7fda-0ade-4042-ad63-5115bc66b454\") " pod="e2e-models-e2e-worker-w2-521f1f/simulator-w2-521f1f-kserve-5f47b86f6-mgksf" Nov 03 02:03:12.136882 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:12.136842 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e1b7fda-0ade-4042-ad63-5115bc66b454-home\") pod \"simulator-w2-521f1f-kserve-5f47b86f6-mgksf\" (UID: \"6e1b7fda-0ade-4042-ad63-5115bc66b454\") " pod="e2e-models-e2e-worker-w2-521f1f/simulator-w2-521f1f-kserve-5f47b86f6-mgksf" Nov 03 02:03:12.136943 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:12.136889 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e1b7fda-0ade-4042-ad63-5115bc66b454-tmp-dir\") pod \"simulator-w2-521f1f-kserve-5f47b86f6-mgksf\" (UID: \"6e1b7fda-0ade-4042-ad63-5115bc66b454\") " pod="e2e-models-e2e-worker-w2-521f1f/simulator-w2-521f1f-kserve-5f47b86f6-mgksf" Nov 03 02:03:12.136998 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:12.136969 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e1b7fda-0ade-4042-ad63-5115bc66b454-dshm\") pod \"simulator-w2-521f1f-kserve-5f47b86f6-mgksf\" (UID: \"6e1b7fda-0ade-4042-ad63-5115bc66b454\") " pod="e2e-models-e2e-worker-w2-521f1f/simulator-w2-521f1f-kserve-5f47b86f6-mgksf" Nov 03 02:03:12.137100 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:12.137074 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e1b7fda-0ade-4042-ad63-5115bc66b454-model-cache\") pod \"simulator-w2-521f1f-kserve-5f47b86f6-mgksf\" (UID: \"6e1b7fda-0ade-4042-ad63-5115bc66b454\") " pod="e2e-models-e2e-worker-w2-521f1f/simulator-w2-521f1f-kserve-5f47b86f6-mgksf" Nov 03 02:03:12.137202 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:12.137180 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e1b7fda-0ade-4042-ad63-5115bc66b454-home\") pod \"simulator-w2-521f1f-kserve-5f47b86f6-mgksf\" (UID: \"6e1b7fda-0ade-4042-ad63-5115bc66b454\") " pod="e2e-models-e2e-worker-w2-521f1f/simulator-w2-521f1f-kserve-5f47b86f6-mgksf" Nov 03 02:03:12.137242 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:12.137191 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e1b7fda-0ade-4042-ad63-5115bc66b454-tmp-dir\") pod \"simulator-w2-521f1f-kserve-5f47b86f6-mgksf\" (UID: \"6e1b7fda-0ade-4042-ad63-5115bc66b454\") " pod="e2e-models-e2e-worker-w2-521f1f/simulator-w2-521f1f-kserve-5f47b86f6-mgksf" Nov 03 02:03:12.139123 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:12.139102 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e1b7fda-0ade-4042-ad63-5115bc66b454-dshm\") pod \"simulator-w2-521f1f-kserve-5f47b86f6-mgksf\" (UID: \"6e1b7fda-0ade-4042-ad63-5115bc66b454\") " pod="e2e-models-e2e-worker-w2-521f1f/simulator-w2-521f1f-kserve-5f47b86f6-mgksf" Nov 03 02:03:12.139253 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:12.139235 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e1b7fda-0ade-4042-ad63-5115bc66b454-tls-certs\") pod \"simulator-w2-521f1f-kserve-5f47b86f6-mgksf\" (UID: \"6e1b7fda-0ade-4042-ad63-5115bc66b454\") " pod="e2e-models-e2e-worker-w2-521f1f/simulator-w2-521f1f-kserve-5f47b86f6-mgksf" Nov 03 02:03:12.190305 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:12.190271 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52p49\" (UniqueName: \"kubernetes.io/projected/6e1b7fda-0ade-4042-ad63-5115bc66b454-kube-api-access-52p49\") pod \"simulator-w2-521f1f-kserve-5f47b86f6-mgksf\" (UID: \"6e1b7fda-0ade-4042-ad63-5115bc66b454\") " pod="e2e-models-e2e-worker-w2-521f1f/simulator-w2-521f1f-kserve-5f47b86f6-mgksf" Nov 03 02:03:12.330737 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:12.330697 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-521f1f/simulator-w2-521f1f-kserve-5f47b86f6-mgksf" Nov 03 02:03:12.701792 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:12.701702 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-521f1f/simulator-w2-521f1f-kserve-5f47b86f6-mgksf"] Nov 03 02:03:12.707983 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:03:12.707954 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e1b7fda_0ade_4042_ad63_5115bc66b454.slice/crio-58ed42aa6e00103394a5308a75f9dcbfd569f40ef257032b4053d34d98fbb453 WatchSource:0}: Error finding container 58ed42aa6e00103394a5308a75f9dcbfd569f40ef257032b4053d34d98fbb453: Status 404 returned error can't find the container with id 58ed42aa6e00103394a5308a75f9dcbfd569f40ef257032b4053d34d98fbb453 Nov 03 02:03:13.677108 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:13.677067 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-521f1f/simulator-w2-521f1f-kserve-5f47b86f6-mgksf" event={"ID":"6e1b7fda-0ade-4042-ad63-5115bc66b454","Type":"ContainerStarted","Data":"58ed42aa6e00103394a5308a75f9dcbfd569f40ef257032b4053d34d98fbb453"} Nov 03 02:03:15.688582 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:15.688549 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-521f1f/simulator-w2-521f1f-kserve-5f47b86f6-mgksf" event={"ID":"6e1b7fda-0ade-4042-ad63-5115bc66b454","Type":"ContainerStarted","Data":"c59287da96ff090ec1aaca274904ed223bc1b7851441c352b29e106b29a67337"} Nov 03 02:03:15.796273 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:15.796220 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-521f1f/simulator-w2-521f1f-kserve-5f47b86f6-mgksf" podStartSLOduration=2.535217125 podStartE2EDuration="4.796204838s" totalImagesPullingTime="2.260987713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:03:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:03:15.795271046 +0000 UTC m=+673.054469200" watchObservedRunningTime="2026-11-03 02:03:15.796204838 +0000 UTC m=+673.055402991" Nov 03 02:03:16.331748 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:16.331717 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-521f1f/simulator-w2-521f1f-kserve-5f47b86f6-mgksf" Nov 03 02:03:18.667816 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:18.667780 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:03:22.331684 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:22.331643 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-521f1f/simulator-w2-521f1f-kserve-5f47b86f6-mgksf" Nov 03 02:03:22.344681 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:22.344654 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-521f1f/simulator-w2-521f1f-kserve-5f47b86f6-mgksf" Nov 03 02:03:22.734345 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:22.734269 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-521f1f/simulator-w2-521f1f-kserve-5f47b86f6-mgksf" Nov 03 02:03:23.911693 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:03:23.911658 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:03:34.001972 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:34.001930 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-a0e72e/simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq"] Nov 03 02:03:34.006781 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:34.006755 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a0e72e/simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq" Nov 03 02:03:34.103657 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:34.103620 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-a0e72e\"/\"openshift-service-ca.crt\"" Nov 03 02:03:34.103873 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:34.103852 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-a0e72e\"/\"kube-root-ca.crt\"" Nov 03 02:03:34.103947 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:34.103935 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-a0e72e\"/\"simulator-w3-a0e72e-kserve-self-signed-certs\"" Nov 03 02:03:34.104052 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:34.104034 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-a0e72e\"/\"default-dockercfg-5tq55\"" Nov 03 02:03:34.105003 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:34.104984 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-a0e72e/simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq"] Nov 03 02:03:34.129690 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:34.129661 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1901f57b-279b-4cbf-bf0c-054abc5786d4-model-cache\") pod \"simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq\" (UID: \"1901f57b-279b-4cbf-bf0c-054abc5786d4\") " pod="e2e-models-e2e-worker-w3-a0e72e/simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq" Nov 03 02:03:34.129853 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:34.129697 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stlkh\" (UniqueName: \"kubernetes.io/projected/1901f57b-279b-4cbf-bf0c-054abc5786d4-kube-api-access-stlkh\") pod \"simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq\" (UID: \"1901f57b-279b-4cbf-bf0c-054abc5786d4\") " pod="e2e-models-e2e-worker-w3-a0e72e/simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq" Nov 03 02:03:34.129853 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:34.129721 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1901f57b-279b-4cbf-bf0c-054abc5786d4-tmp-dir\") pod \"simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq\" (UID: \"1901f57b-279b-4cbf-bf0c-054abc5786d4\") " pod="e2e-models-e2e-worker-w3-a0e72e/simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq" Nov 03 02:03:34.129853 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:34.129797 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1901f57b-279b-4cbf-bf0c-054abc5786d4-home\") pod \"simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq\" (UID: \"1901f57b-279b-4cbf-bf0c-054abc5786d4\") " pod="e2e-models-e2e-worker-w3-a0e72e/simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq" Nov 03 02:03:34.129853 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:34.129833 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1901f57b-279b-4cbf-bf0c-054abc5786d4-tls-certs\") pod \"simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq\" (UID: \"1901f57b-279b-4cbf-bf0c-054abc5786d4\") " pod="e2e-models-e2e-worker-w3-a0e72e/simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq" Nov 03 02:03:34.130070 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:34.129953 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1901f57b-279b-4cbf-bf0c-054abc5786d4-dshm\") pod \"simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq\" (UID: \"1901f57b-279b-4cbf-bf0c-054abc5786d4\") " pod="e2e-models-e2e-worker-w3-a0e72e/simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq" Nov 03 02:03:34.231082 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:34.231031 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1901f57b-279b-4cbf-bf0c-054abc5786d4-model-cache\") pod \"simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq\" (UID: \"1901f57b-279b-4cbf-bf0c-054abc5786d4\") " pod="e2e-models-e2e-worker-w3-a0e72e/simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq" Nov 03 02:03:34.231272 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:34.231087 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stlkh\" (UniqueName: \"kubernetes.io/projected/1901f57b-279b-4cbf-bf0c-054abc5786d4-kube-api-access-stlkh\") pod \"simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq\" (UID: \"1901f57b-279b-4cbf-bf0c-054abc5786d4\") " pod="e2e-models-e2e-worker-w3-a0e72e/simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq" Nov 03 02:03:34.231272 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:34.231123 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1901f57b-279b-4cbf-bf0c-054abc5786d4-tmp-dir\") pod \"simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq\" (UID: \"1901f57b-279b-4cbf-bf0c-054abc5786d4\") " pod="e2e-models-e2e-worker-w3-a0e72e/simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq" Nov 03 02:03:34.231272 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:34.231148 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1901f57b-279b-4cbf-bf0c-054abc5786d4-home\") pod \"simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq\" (UID: \"1901f57b-279b-4cbf-bf0c-054abc5786d4\") " pod="e2e-models-e2e-worker-w3-a0e72e/simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq" Nov 03 02:03:34.231272 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:34.231196 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1901f57b-279b-4cbf-bf0c-054abc5786d4-tls-certs\") pod \"simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq\" (UID: \"1901f57b-279b-4cbf-bf0c-054abc5786d4\") " pod="e2e-models-e2e-worker-w3-a0e72e/simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq" Nov 03 02:03:34.231272 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:34.231268 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1901f57b-279b-4cbf-bf0c-054abc5786d4-dshm\") pod \"simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq\" (UID: \"1901f57b-279b-4cbf-bf0c-054abc5786d4\") " pod="e2e-models-e2e-worker-w3-a0e72e/simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq" Nov 03 02:03:34.231574 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:34.231548 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1901f57b-279b-4cbf-bf0c-054abc5786d4-tmp-dir\") pod \"simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq\" (UID: \"1901f57b-279b-4cbf-bf0c-054abc5786d4\") " pod="e2e-models-e2e-worker-w3-a0e72e/simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq" Nov 03 02:03:34.231574 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:34.231561 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1901f57b-279b-4cbf-bf0c-054abc5786d4-model-cache\") pod \"simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq\" (UID: \"1901f57b-279b-4cbf-bf0c-054abc5786d4\") " pod="e2e-models-e2e-worker-w3-a0e72e/simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq" Nov 03 02:03:34.231903 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:34.231877 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1901f57b-279b-4cbf-bf0c-054abc5786d4-home\") pod \"simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq\" (UID: \"1901f57b-279b-4cbf-bf0c-054abc5786d4\") " pod="e2e-models-e2e-worker-w3-a0e72e/simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq" Nov 03 02:03:34.233818 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:34.233780 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1901f57b-279b-4cbf-bf0c-054abc5786d4-dshm\") pod \"simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq\" (UID: \"1901f57b-279b-4cbf-bf0c-054abc5786d4\") " pod="e2e-models-e2e-worker-w3-a0e72e/simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq" Nov 03 02:03:34.234348 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:34.234325 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1901f57b-279b-4cbf-bf0c-054abc5786d4-tls-certs\") pod \"simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq\" (UID: \"1901f57b-279b-4cbf-bf0c-054abc5786d4\") " pod="e2e-models-e2e-worker-w3-a0e72e/simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq" Nov 03 02:03:34.284543 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:34.284457 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stlkh\" (UniqueName: \"kubernetes.io/projected/1901f57b-279b-4cbf-bf0c-054abc5786d4-kube-api-access-stlkh\") pod \"simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq\" (UID: \"1901f57b-279b-4cbf-bf0c-054abc5786d4\") " pod="e2e-models-e2e-worker-w3-a0e72e/simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq" Nov 03 02:03:34.318451 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:34.318416 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a0e72e/simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq" Nov 03 02:03:34.685591 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:34.685556 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-a0e72e/simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq"] Nov 03 02:03:34.692122 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:03:34.692091 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1901f57b_279b_4cbf_bf0c_054abc5786d4.slice/crio-4bc681f89142799b7b29b94db7fdbd3d38e04d9b10fd2be959740e529a878426 WatchSource:0}: Error finding container 4bc681f89142799b7b29b94db7fdbd3d38e04d9b10fd2be959740e529a878426: Status 404 returned error can't find the container with id 4bc681f89142799b7b29b94db7fdbd3d38e04d9b10fd2be959740e529a878426 Nov 03 02:03:34.776760 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:34.776718 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a0e72e/simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq" event={"ID":"1901f57b-279b-4cbf-bf0c-054abc5786d4","Type":"ContainerStarted","Data":"83278823364d3ba13b2c0f63ab403df213efbd2f7cd1e611afa52b2c2130f5d8"} Nov 03 02:03:34.776921 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:34.776770 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a0e72e/simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq" event={"ID":"1901f57b-279b-4cbf-bf0c-054abc5786d4","Type":"ContainerStarted","Data":"4bc681f89142799b7b29b94db7fdbd3d38e04d9b10fd2be959740e529a878426"} Nov 03 02:03:34.886344 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:34.886287 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-a0e72e/simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq" podStartSLOduration=1.8862723639999999 podStartE2EDuration="1.886272364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:03:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:03:34.884965493 +0000 UTC m=+692.144163647" watchObservedRunningTime="2026-11-03 02:03:34.886272364 +0000 UTC m=+692.145470518" Nov 03 02:03:35.319580 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:35.319549 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a0e72e/simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq" Nov 03 02:03:36.146713 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:36.146680 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:03:44.319411 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:44.319366 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-a0e72e/simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq" Nov 03 02:03:44.331950 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:44.331923 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-a0e72e/simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq" Nov 03 02:03:44.837218 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:44.837183 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a0e72e/simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq" Nov 03 02:03:45.962431 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:03:45.962393 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:03:53.911708 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:03:53.911663 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:04:03.855371 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:03.855321 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-521f1f/premium-w2-521f1f-kserve-7578fb7957-89wfb"] Nov 03 02:04:03.860519 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:03.860494 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-521f1f/premium-w2-521f1f-kserve-7578fb7957-89wfb" Nov 03 02:04:03.912058 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:03.912020 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d988573-d8b1-43ba-a423-51a63ca12337-model-cache\") pod \"premium-w2-521f1f-kserve-7578fb7957-89wfb\" (UID: \"3d988573-d8b1-43ba-a423-51a63ca12337\") " pod="e2e-models-e2e-worker-w2-521f1f/premium-w2-521f1f-kserve-7578fb7957-89wfb" Nov 03 02:04:03.912513 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:03.912481 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d988573-d8b1-43ba-a423-51a63ca12337-home\") pod \"premium-w2-521f1f-kserve-7578fb7957-89wfb\" (UID: \"3d988573-d8b1-43ba-a423-51a63ca12337\") " pod="e2e-models-e2e-worker-w2-521f1f/premium-w2-521f1f-kserve-7578fb7957-89wfb" Nov 03 02:04:03.912660 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:03.912521 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d988573-d8b1-43ba-a423-51a63ca12337-tmp-dir\") pod \"premium-w2-521f1f-kserve-7578fb7957-89wfb\" (UID: \"3d988573-d8b1-43ba-a423-51a63ca12337\") " pod="e2e-models-e2e-worker-w2-521f1f/premium-w2-521f1f-kserve-7578fb7957-89wfb" Nov 03 02:04:03.912660 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:03.912550 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d988573-d8b1-43ba-a423-51a63ca12337-dshm\") pod \"premium-w2-521f1f-kserve-7578fb7957-89wfb\" (UID: \"3d988573-d8b1-43ba-a423-51a63ca12337\") " pod="e2e-models-e2e-worker-w2-521f1f/premium-w2-521f1f-kserve-7578fb7957-89wfb" Nov 03 02:04:03.912660 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:03.912575 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d988573-d8b1-43ba-a423-51a63ca12337-tls-certs\") pod \"premium-w2-521f1f-kserve-7578fb7957-89wfb\" (UID: \"3d988573-d8b1-43ba-a423-51a63ca12337\") " pod="e2e-models-e2e-worker-w2-521f1f/premium-w2-521f1f-kserve-7578fb7957-89wfb" Nov 03 02:04:03.912660 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:03.912598 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxbg2\" (UniqueName: \"kubernetes.io/projected/3d988573-d8b1-43ba-a423-51a63ca12337-kube-api-access-dxbg2\") pod \"premium-w2-521f1f-kserve-7578fb7957-89wfb\" (UID: \"3d988573-d8b1-43ba-a423-51a63ca12337\") " pod="e2e-models-e2e-worker-w2-521f1f/premium-w2-521f1f-kserve-7578fb7957-89wfb" Nov 03 02:04:03.936028 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:03.935993 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-521f1f/premium-w2-521f1f-kserve-7578fb7957-89wfb"] Nov 03 02:04:03.958134 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:03.958098 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-521f1f\"/\"premium-w2-521f1f-kserve-self-signed-certs\"" Nov 03 02:04:04.020385 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:04.017016 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d988573-d8b1-43ba-a423-51a63ca12337-home\") pod \"premium-w2-521f1f-kserve-7578fb7957-89wfb\" (UID: \"3d988573-d8b1-43ba-a423-51a63ca12337\") " pod="e2e-models-e2e-worker-w2-521f1f/premium-w2-521f1f-kserve-7578fb7957-89wfb" Nov 03 02:04:04.020385 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:04.018620 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d988573-d8b1-43ba-a423-51a63ca12337-tmp-dir\") pod \"premium-w2-521f1f-kserve-7578fb7957-89wfb\" (UID: \"3d988573-d8b1-43ba-a423-51a63ca12337\") " pod="e2e-models-e2e-worker-w2-521f1f/premium-w2-521f1f-kserve-7578fb7957-89wfb" Nov 03 02:04:04.020385 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:04.018667 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d988573-d8b1-43ba-a423-51a63ca12337-dshm\") pod \"premium-w2-521f1f-kserve-7578fb7957-89wfb\" (UID: \"3d988573-d8b1-43ba-a423-51a63ca12337\") " pod="e2e-models-e2e-worker-w2-521f1f/premium-w2-521f1f-kserve-7578fb7957-89wfb" Nov 03 02:04:04.020385 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:04.018696 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d988573-d8b1-43ba-a423-51a63ca12337-tls-certs\") pod \"premium-w2-521f1f-kserve-7578fb7957-89wfb\" (UID: \"3d988573-d8b1-43ba-a423-51a63ca12337\") " pod="e2e-models-e2e-worker-w2-521f1f/premium-w2-521f1f-kserve-7578fb7957-89wfb" Nov 03 02:04:04.020385 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:04.018723 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxbg2\" (UniqueName: \"kubernetes.io/projected/3d988573-d8b1-43ba-a423-51a63ca12337-kube-api-access-dxbg2\") pod \"premium-w2-521f1f-kserve-7578fb7957-89wfb\" (UID: \"3d988573-d8b1-43ba-a423-51a63ca12337\") " pod="e2e-models-e2e-worker-w2-521f1f/premium-w2-521f1f-kserve-7578fb7957-89wfb" Nov 03 02:04:04.020385 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:04.018784 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d988573-d8b1-43ba-a423-51a63ca12337-model-cache\") pod \"premium-w2-521f1f-kserve-7578fb7957-89wfb\" (UID: \"3d988573-d8b1-43ba-a423-51a63ca12337\") " pod="e2e-models-e2e-worker-w2-521f1f/premium-w2-521f1f-kserve-7578fb7957-89wfb" Nov 03 02:04:04.020385 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:04.018004 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d988573-d8b1-43ba-a423-51a63ca12337-home\") pod \"premium-w2-521f1f-kserve-7578fb7957-89wfb\" (UID: \"3d988573-d8b1-43ba-a423-51a63ca12337\") " pod="e2e-models-e2e-worker-w2-521f1f/premium-w2-521f1f-kserve-7578fb7957-89wfb" Nov 03 02:04:04.020385 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:04.019412 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d988573-d8b1-43ba-a423-51a63ca12337-model-cache\") pod \"premium-w2-521f1f-kserve-7578fb7957-89wfb\" (UID: \"3d988573-d8b1-43ba-a423-51a63ca12337\") " pod="e2e-models-e2e-worker-w2-521f1f/premium-w2-521f1f-kserve-7578fb7957-89wfb" Nov 03 02:04:04.020385 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:04.019705 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d988573-d8b1-43ba-a423-51a63ca12337-tmp-dir\") pod \"premium-w2-521f1f-kserve-7578fb7957-89wfb\" (UID: \"3d988573-d8b1-43ba-a423-51a63ca12337\") " pod="e2e-models-e2e-worker-w2-521f1f/premium-w2-521f1f-kserve-7578fb7957-89wfb" Nov 03 02:04:04.022142 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:04.022118 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d988573-d8b1-43ba-a423-51a63ca12337-dshm\") pod \"premium-w2-521f1f-kserve-7578fb7957-89wfb\" (UID: \"3d988573-d8b1-43ba-a423-51a63ca12337\") " pod="e2e-models-e2e-worker-w2-521f1f/premium-w2-521f1f-kserve-7578fb7957-89wfb" Nov 03 02:04:04.022436 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:04.022416 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d988573-d8b1-43ba-a423-51a63ca12337-tls-certs\") pod \"premium-w2-521f1f-kserve-7578fb7957-89wfb\" (UID: \"3d988573-d8b1-43ba-a423-51a63ca12337\") " pod="e2e-models-e2e-worker-w2-521f1f/premium-w2-521f1f-kserve-7578fb7957-89wfb" Nov 03 02:04:04.072600 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:04.072564 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxbg2\" (UniqueName: \"kubernetes.io/projected/3d988573-d8b1-43ba-a423-51a63ca12337-kube-api-access-dxbg2\") pod \"premium-w2-521f1f-kserve-7578fb7957-89wfb\" (UID: \"3d988573-d8b1-43ba-a423-51a63ca12337\") " pod="e2e-models-e2e-worker-w2-521f1f/premium-w2-521f1f-kserve-7578fb7957-89wfb" Nov 03 02:04:04.171638 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:04.171539 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-521f1f/premium-w2-521f1f-kserve-7578fb7957-89wfb" Nov 03 02:04:04.541031 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:04.540937 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-521f1f/premium-w2-521f1f-kserve-7578fb7957-89wfb"] Nov 03 02:04:04.547838 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:04:04.547801 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d988573_d8b1_43ba_a423_51a63ca12337.slice/crio-395898ff4fb26a6dc43f8c8c285ac2149c776476c94395e6e9982a100273e691 WatchSource:0}: Error finding container 395898ff4fb26a6dc43f8c8c285ac2149c776476c94395e6e9982a100273e691: Status 404 returned error can't find the container with id 395898ff4fb26a6dc43f8c8c285ac2149c776476c94395e6e9982a100273e691 Nov 03 02:04:04.922155 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:04.922118 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-521f1f/premium-w2-521f1f-kserve-7578fb7957-89wfb" event={"ID":"3d988573-d8b1-43ba-a423-51a63ca12337","Type":"ContainerStarted","Data":"5a4a1e2f703733cb114e00e6a72cc2efdcf3f7c20aa0fd52ed7f7a9346ad1600"} Nov 03 02:04:04.922678 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:04.922163 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-521f1f/premium-w2-521f1f-kserve-7578fb7957-89wfb" event={"ID":"3d988573-d8b1-43ba-a423-51a63ca12337","Type":"ContainerStarted","Data":"395898ff4fb26a6dc43f8c8c285ac2149c776476c94395e6e9982a100273e691"} Nov 03 02:04:05.029525 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:05.029461 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-521f1f/premium-w2-521f1f-kserve-7578fb7957-89wfb" podStartSLOduration=2.029441813 podStartE2EDuration="2.029441813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:04:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:04:05.028390644 +0000 UTC m=+722.287588798" watchObservedRunningTime="2026-11-03 02:04:05.029441813 +0000 UTC m=+722.288639971" Nov 03 02:04:05.172609 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:05.172526 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-521f1f/premium-w2-521f1f-kserve-7578fb7957-89wfb" Nov 03 02:04:09.467758 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:09.467719 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:04:14.172043 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:14.171968 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-521f1f/premium-w2-521f1f-kserve-7578fb7957-89wfb" Nov 03 02:04:14.184158 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:14.184131 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-521f1f/premium-w2-521f1f-kserve-7578fb7957-89wfb" Nov 03 02:04:14.987514 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:14.987484 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-521f1f/premium-w2-521f1f-kserve-7578fb7957-89wfb" Nov 03 02:04:23.119154 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:23.119113 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:04:23.911306 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:04:23.911271 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:04:30.045929 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:30.045895 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-a0e72e/premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv"] Nov 03 02:04:30.049459 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:30.049436 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a0e72e/premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv" Nov 03 02:04:30.147539 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:30.147492 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-a0e72e\"/\"premium-w3-a0e72e-kserve-self-signed-certs\"" Nov 03 02:04:30.148810 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:30.148788 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-a0e72e/premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv"] Nov 03 02:04:30.171711 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:30.171674 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9b9af44-ab6d-4caf-89af-c986ea25b4c9-home\") pod \"premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv\" (UID: \"a9b9af44-ab6d-4caf-89af-c986ea25b4c9\") " pod="e2e-models-e2e-worker-w3-a0e72e/premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv" Nov 03 02:04:30.171879 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:30.171720 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9b9af44-ab6d-4caf-89af-c986ea25b4c9-model-cache\") pod \"premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv\" (UID: \"a9b9af44-ab6d-4caf-89af-c986ea25b4c9\") " pod="e2e-models-e2e-worker-w3-a0e72e/premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv" Nov 03 02:04:30.171879 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:30.171744 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfpdh\" (UniqueName: \"kubernetes.io/projected/a9b9af44-ab6d-4caf-89af-c986ea25b4c9-kube-api-access-mfpdh\") pod \"premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv\" (UID: \"a9b9af44-ab6d-4caf-89af-c986ea25b4c9\") " pod="e2e-models-e2e-worker-w3-a0e72e/premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv" Nov 03 02:04:30.171879 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:30.171823 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9b9af44-ab6d-4caf-89af-c986ea25b4c9-dshm\") pod \"premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv\" (UID: \"a9b9af44-ab6d-4caf-89af-c986ea25b4c9\") " pod="e2e-models-e2e-worker-w3-a0e72e/premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv" Nov 03 02:04:30.171979 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:30.171884 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9b9af44-ab6d-4caf-89af-c986ea25b4c9-tmp-dir\") pod \"premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv\" (UID: \"a9b9af44-ab6d-4caf-89af-c986ea25b4c9\") " pod="e2e-models-e2e-worker-w3-a0e72e/premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv" Nov 03 02:04:30.171979 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:30.171908 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9b9af44-ab6d-4caf-89af-c986ea25b4c9-tls-certs\") pod \"premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv\" (UID: \"a9b9af44-ab6d-4caf-89af-c986ea25b4c9\") " pod="e2e-models-e2e-worker-w3-a0e72e/premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv" Nov 03 02:04:30.273154 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:30.273114 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9b9af44-ab6d-4caf-89af-c986ea25b4c9-home\") pod \"premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv\" (UID: \"a9b9af44-ab6d-4caf-89af-c986ea25b4c9\") " pod="e2e-models-e2e-worker-w3-a0e72e/premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv" Nov 03 02:04:30.273154 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:30.273162 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9b9af44-ab6d-4caf-89af-c986ea25b4c9-model-cache\") pod \"premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv\" (UID: \"a9b9af44-ab6d-4caf-89af-c986ea25b4c9\") " pod="e2e-models-e2e-worker-w3-a0e72e/premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv" Nov 03 02:04:30.273450 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:30.273180 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfpdh\" (UniqueName: \"kubernetes.io/projected/a9b9af44-ab6d-4caf-89af-c986ea25b4c9-kube-api-access-mfpdh\") pod \"premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv\" (UID: \"a9b9af44-ab6d-4caf-89af-c986ea25b4c9\") " pod="e2e-models-e2e-worker-w3-a0e72e/premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv" Nov 03 02:04:30.273450 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:30.273204 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9b9af44-ab6d-4caf-89af-c986ea25b4c9-dshm\") pod \"premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv\" (UID: \"a9b9af44-ab6d-4caf-89af-c986ea25b4c9\") " pod="e2e-models-e2e-worker-w3-a0e72e/premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv" Nov 03 02:04:30.273450 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:30.273234 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9b9af44-ab6d-4caf-89af-c986ea25b4c9-tmp-dir\") pod \"premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv\" (UID: \"a9b9af44-ab6d-4caf-89af-c986ea25b4c9\") " pod="e2e-models-e2e-worker-w3-a0e72e/premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv" Nov 03 02:04:30.273450 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:30.273417 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9b9af44-ab6d-4caf-89af-c986ea25b4c9-tls-certs\") pod \"premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv\" (UID: \"a9b9af44-ab6d-4caf-89af-c986ea25b4c9\") " pod="e2e-models-e2e-worker-w3-a0e72e/premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv" Nov 03 02:04:30.273706 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:30.273680 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9b9af44-ab6d-4caf-89af-c986ea25b4c9-home\") pod \"premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv\" (UID: \"a9b9af44-ab6d-4caf-89af-c986ea25b4c9\") " pod="e2e-models-e2e-worker-w3-a0e72e/premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv" Nov 03 02:04:30.273771 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:30.273728 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9b9af44-ab6d-4caf-89af-c986ea25b4c9-model-cache\") pod \"premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv\" (UID: \"a9b9af44-ab6d-4caf-89af-c986ea25b4c9\") " pod="e2e-models-e2e-worker-w3-a0e72e/premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv" Nov 03 02:04:30.273817 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:30.273782 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9b9af44-ab6d-4caf-89af-c986ea25b4c9-tmp-dir\") pod \"premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv\" (UID: \"a9b9af44-ab6d-4caf-89af-c986ea25b4c9\") " pod="e2e-models-e2e-worker-w3-a0e72e/premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv" Nov 03 02:04:30.275782 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:30.275755 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9b9af44-ab6d-4caf-89af-c986ea25b4c9-dshm\") pod \"premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv\" (UID: \"a9b9af44-ab6d-4caf-89af-c986ea25b4c9\") " pod="e2e-models-e2e-worker-w3-a0e72e/premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv" Nov 03 02:04:30.275896 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:30.275867 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9b9af44-ab6d-4caf-89af-c986ea25b4c9-tls-certs\") pod \"premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv\" (UID: \"a9b9af44-ab6d-4caf-89af-c986ea25b4c9\") " pod="e2e-models-e2e-worker-w3-a0e72e/premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv" Nov 03 02:04:30.327090 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:30.327046 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfpdh\" (UniqueName: \"kubernetes.io/projected/a9b9af44-ab6d-4caf-89af-c986ea25b4c9-kube-api-access-mfpdh\") pod \"premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv\" (UID: \"a9b9af44-ab6d-4caf-89af-c986ea25b4c9\") " pod="e2e-models-e2e-worker-w3-a0e72e/premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv" Nov 03 02:04:30.361817 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:30.361778 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a0e72e/premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv" Nov 03 02:04:30.733981 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:30.733881 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-a0e72e/premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv"] Nov 03 02:04:30.740988 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:04:30.740959 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda9b9af44_ab6d_4caf_89af_c986ea25b4c9.slice/crio-0392f35f9cb3eca01159741a9db4ac42c87ec836e0048f986a9983ea98ce8468 WatchSource:0}: Error finding container 0392f35f9cb3eca01159741a9db4ac42c87ec836e0048f986a9983ea98ce8468: Status 404 returned error can't find the container with id 0392f35f9cb3eca01159741a9db4ac42c87ec836e0048f986a9983ea98ce8468 Nov 03 02:04:31.052767 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:31.052728 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a0e72e/premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv" event={"ID":"a9b9af44-ab6d-4caf-89af-c986ea25b4c9","Type":"ContainerStarted","Data":"9221409c6e9a68a85e610d4a12f79d98e717f4cc7a63ba3d6f60d794b1f21b67"} Nov 03 02:04:31.053282 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:31.052782 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a0e72e/premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv" event={"ID":"a9b9af44-ab6d-4caf-89af-c986ea25b4c9","Type":"ContainerStarted","Data":"0392f35f9cb3eca01159741a9db4ac42c87ec836e0048f986a9983ea98ce8468"} Nov 03 02:04:31.159829 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:31.159776 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-a0e72e/premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv" podStartSLOduration=1.159760007 podStartE2EDuration="1.159760007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:04:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:04:31.158320907 +0000 UTC m=+748.417519060" watchObservedRunningTime="2026-11-03 02:04:31.159760007 +0000 UTC m=+748.418958160" Nov 03 02:04:31.362963 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:31.362867 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a0e72e/premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv" Nov 03 02:04:38.857961 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:38.857926 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:04:40.362839 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:40.362798 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-a0e72e/premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv" Nov 03 02:04:40.375482 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:40.375454 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-a0e72e/premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv" Nov 03 02:04:41.110282 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:41.110256 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a0e72e/premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv" Nov 03 02:04:41.434108 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:41.434021 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f7d9bb/simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls"] Nov 03 02:04:41.437688 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:41.437667 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f7d9bb/simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls" Nov 03 02:04:41.486930 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:41.486899 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f7d9bb\"/\"kube-root-ca.crt\"" Nov 03 02:04:41.487102 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:41.487005 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f7d9bb\"/\"openshift-service-ca.crt\"" Nov 03 02:04:41.487102 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:41.487045 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f7d9bb\"/\"default-dockercfg-57ngd\"" Nov 03 02:04:41.487192 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:41.487119 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f7d9bb\"/\"simulator-w0-f7d9bb-kserve-self-signed-certs\"" Nov 03 02:04:41.491980 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:41.491955 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f7d9bb/simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls"] Nov 03 02:04:41.594048 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:41.594008 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9173364c-14ba-4287-8587-45f25118e3e4-dshm\") pod \"simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls\" (UID: \"9173364c-14ba-4287-8587-45f25118e3e4\") " pod="e2e-models-e2e-worker-w0-f7d9bb/simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls" Nov 03 02:04:41.594292 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:41.594271 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9173364c-14ba-4287-8587-45f25118e3e4-tls-certs\") pod \"simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls\" (UID: \"9173364c-14ba-4287-8587-45f25118e3e4\") " pod="e2e-models-e2e-worker-w0-f7d9bb/simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls" Nov 03 02:04:41.594518 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:41.594488 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9173364c-14ba-4287-8587-45f25118e3e4-tmp-dir\") pod \"simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls\" (UID: \"9173364c-14ba-4287-8587-45f25118e3e4\") " pod="e2e-models-e2e-worker-w0-f7d9bb/simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls" Nov 03 02:04:41.594611 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:41.594524 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5c5v\" (UniqueName: \"kubernetes.io/projected/9173364c-14ba-4287-8587-45f25118e3e4-kube-api-access-d5c5v\") pod \"simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls\" (UID: \"9173364c-14ba-4287-8587-45f25118e3e4\") " pod="e2e-models-e2e-worker-w0-f7d9bb/simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls" Nov 03 02:04:41.594611 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:41.594551 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9173364c-14ba-4287-8587-45f25118e3e4-model-cache\") pod \"simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls\" (UID: \"9173364c-14ba-4287-8587-45f25118e3e4\") " pod="e2e-models-e2e-worker-w0-f7d9bb/simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls" Nov 03 02:04:41.594611 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:41.594593 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9173364c-14ba-4287-8587-45f25118e3e4-home\") pod \"simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls\" (UID: \"9173364c-14ba-4287-8587-45f25118e3e4\") " pod="e2e-models-e2e-worker-w0-f7d9bb/simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls" Nov 03 02:04:41.695919 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:41.695831 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9173364c-14ba-4287-8587-45f25118e3e4-dshm\") pod \"simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls\" (UID: \"9173364c-14ba-4287-8587-45f25118e3e4\") " pod="e2e-models-e2e-worker-w0-f7d9bb/simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls" Nov 03 02:04:41.695919 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:41.695884 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9173364c-14ba-4287-8587-45f25118e3e4-tls-certs\") pod \"simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls\" (UID: \"9173364c-14ba-4287-8587-45f25118e3e4\") " pod="e2e-models-e2e-worker-w0-f7d9bb/simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls" Nov 03 02:04:41.696146 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:41.695935 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9173364c-14ba-4287-8587-45f25118e3e4-tmp-dir\") pod \"simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls\" (UID: \"9173364c-14ba-4287-8587-45f25118e3e4\") " pod="e2e-models-e2e-worker-w0-f7d9bb/simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls" Nov 03 02:04:41.696146 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:41.695960 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5c5v\" (UniqueName: \"kubernetes.io/projected/9173364c-14ba-4287-8587-45f25118e3e4-kube-api-access-d5c5v\") pod \"simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls\" (UID: \"9173364c-14ba-4287-8587-45f25118e3e4\") " pod="e2e-models-e2e-worker-w0-f7d9bb/simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls" Nov 03 02:04:41.696146 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:41.695985 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9173364c-14ba-4287-8587-45f25118e3e4-model-cache\") pod \"simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls\" (UID: \"9173364c-14ba-4287-8587-45f25118e3e4\") " pod="e2e-models-e2e-worker-w0-f7d9bb/simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls" Nov 03 02:04:41.696146 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:41.696020 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9173364c-14ba-4287-8587-45f25118e3e4-home\") pod \"simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls\" (UID: \"9173364c-14ba-4287-8587-45f25118e3e4\") " pod="e2e-models-e2e-worker-w0-f7d9bb/simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls" Nov 03 02:04:41.696349 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:41.696328 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9173364c-14ba-4287-8587-45f25118e3e4-tmp-dir\") pod \"simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls\" (UID: \"9173364c-14ba-4287-8587-45f25118e3e4\") " pod="e2e-models-e2e-worker-w0-f7d9bb/simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls" Nov 03 02:04:41.696442 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:41.696416 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9173364c-14ba-4287-8587-45f25118e3e4-home\") pod \"simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls\" (UID: \"9173364c-14ba-4287-8587-45f25118e3e4\") " pod="e2e-models-e2e-worker-w0-f7d9bb/simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls" Nov 03 02:04:41.696509 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:41.696485 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9173364c-14ba-4287-8587-45f25118e3e4-model-cache\") pod \"simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls\" (UID: \"9173364c-14ba-4287-8587-45f25118e3e4\") " pod="e2e-models-e2e-worker-w0-f7d9bb/simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls" Nov 03 02:04:41.698267 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:41.698233 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9173364c-14ba-4287-8587-45f25118e3e4-dshm\") pod \"simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls\" (UID: \"9173364c-14ba-4287-8587-45f25118e3e4\") " pod="e2e-models-e2e-worker-w0-f7d9bb/simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls" Nov 03 02:04:41.698509 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:41.698487 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9173364c-14ba-4287-8587-45f25118e3e4-tls-certs\") pod \"simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls\" (UID: \"9173364c-14ba-4287-8587-45f25118e3e4\") " pod="e2e-models-e2e-worker-w0-f7d9bb/simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls" Nov 03 02:04:41.749007 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:41.748967 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5c5v\" (UniqueName: \"kubernetes.io/projected/9173364c-14ba-4287-8587-45f25118e3e4-kube-api-access-d5c5v\") pod \"simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls\" (UID: \"9173364c-14ba-4287-8587-45f25118e3e4\") " pod="e2e-models-e2e-worker-w0-f7d9bb/simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls" Nov 03 02:04:41.749223 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:41.749206 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f7d9bb/simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls" Nov 03 02:04:42.137130 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:42.137094 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f7d9bb/simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls"] Nov 03 02:04:42.143473 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:04:42.143438 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9173364c_14ba_4287_8587_45f25118e3e4.slice/crio-0b47d7fc7966b0833b713c5a694705501f0ed4e82d42fe3c9202d2a1a0839852 WatchSource:0}: Error finding container 0b47d7fc7966b0833b713c5a694705501f0ed4e82d42fe3c9202d2a1a0839852: Status 404 returned error can't find the container with id 0b47d7fc7966b0833b713c5a694705501f0ed4e82d42fe3c9202d2a1a0839852 Nov 03 02:04:43.112916 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:43.112876 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f7d9bb/simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls" event={"ID":"9173364c-14ba-4287-8587-45f25118e3e4","Type":"ContainerStarted","Data":"da057a3221ca59a939aed416d0fbc48f65c67c1cc1ab277bb4d75939b0b9c09b"} Nov 03 02:04:43.113305 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:43.112927 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f7d9bb/simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls" event={"ID":"9173364c-14ba-4287-8587-45f25118e3e4","Type":"ContainerStarted","Data":"0b47d7fc7966b0833b713c5a694705501f0ed4e82d42fe3c9202d2a1a0839852"} Nov 03 02:04:43.220556 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:43.220498 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f7d9bb/simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls" podStartSLOduration=2.220483601 podStartE2EDuration="2.220483601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:04:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:04:43.219524789 +0000 UTC m=+760.478722955" watchObservedRunningTime="2026-11-03 02:04:43.220483601 +0000 UTC m=+760.479681748" Nov 03 02:04:43.749767 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:43.749729 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f7d9bb/simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls" Nov 03 02:04:51.749491 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:51.749439 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f7d9bb/simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls" Nov 03 02:04:51.762593 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:51.762563 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f7d9bb/simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls" Nov 03 02:04:52.163481 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:04:52.163452 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f7d9bb/simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls" Nov 03 02:04:53.912786 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:04:53.912742 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:05:01.298404 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:01.298369 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:05:04.302249 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:04.302213 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2"] Nov 03 02:05:04.305839 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:04.305813 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2" Nov 03 02:05:04.354748 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:04.354713 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-a0e72e-openshift-default-dockercfg-cskrh\"" Nov 03 02:05:04.403108 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:04.403074 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2"] Nov 03 02:05:04.409842 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:04.409812 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/868119ea-b49f-4566-8a46-bb3990a1e6f6-istio-envoy\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2\" (UID: \"868119ea-b49f-4566-8a46-bb3990a1e6f6\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2" Nov 03 02:05:04.410012 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:04.409865 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/868119ea-b49f-4566-8a46-bb3990a1e6f6-istio-data\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2\" (UID: \"868119ea-b49f-4566-8a46-bb3990a1e6f6\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2" Nov 03 02:05:04.410012 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:04.409988 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/868119ea-b49f-4566-8a46-bb3990a1e6f6-istiod-ca-cert\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2\" (UID: \"868119ea-b49f-4566-8a46-bb3990a1e6f6\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2" Nov 03 02:05:04.410136 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:04.410030 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/868119ea-b49f-4566-8a46-bb3990a1e6f6-workload-socket\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2\" (UID: \"868119ea-b49f-4566-8a46-bb3990a1e6f6\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2" Nov 03 02:05:04.410136 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:04.410112 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/868119ea-b49f-4566-8a46-bb3990a1e6f6-workload-certs\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2\" (UID: \"868119ea-b49f-4566-8a46-bb3990a1e6f6\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2" Nov 03 02:05:04.410235 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:04.410143 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/868119ea-b49f-4566-8a46-bb3990a1e6f6-istio-token\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2\" (UID: \"868119ea-b49f-4566-8a46-bb3990a1e6f6\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2" Nov 03 02:05:04.410235 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:04.410162 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4vdn\" (UniqueName: \"kubernetes.io/projected/868119ea-b49f-4566-8a46-bb3990a1e6f6-kube-api-access-l4vdn\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2\" (UID: \"868119ea-b49f-4566-8a46-bb3990a1e6f6\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2" Nov 03 02:05:04.410235 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:04.410178 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/868119ea-b49f-4566-8a46-bb3990a1e6f6-credential-socket\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2\" (UID: \"868119ea-b49f-4566-8a46-bb3990a1e6f6\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2" Nov 03 02:05:04.410330 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:04.410265 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/868119ea-b49f-4566-8a46-bb3990a1e6f6-istio-podinfo\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2\" (UID: \"868119ea-b49f-4566-8a46-bb3990a1e6f6\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2" Nov 03 02:05:04.511734 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:04.511690 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/868119ea-b49f-4566-8a46-bb3990a1e6f6-istiod-ca-cert\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2\" (UID: \"868119ea-b49f-4566-8a46-bb3990a1e6f6\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2" Nov 03 02:05:04.511734 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:04.511741 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/868119ea-b49f-4566-8a46-bb3990a1e6f6-workload-socket\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2\" (UID: \"868119ea-b49f-4566-8a46-bb3990a1e6f6\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2" Nov 03 02:05:04.511995 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:04.511773 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/868119ea-b49f-4566-8a46-bb3990a1e6f6-workload-certs\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2\" (UID: \"868119ea-b49f-4566-8a46-bb3990a1e6f6\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2" Nov 03 02:05:04.511995 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:04.511808 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/868119ea-b49f-4566-8a46-bb3990a1e6f6-istio-token\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2\" (UID: \"868119ea-b49f-4566-8a46-bb3990a1e6f6\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2" Nov 03 02:05:04.511995 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:04.511834 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4vdn\" (UniqueName: \"kubernetes.io/projected/868119ea-b49f-4566-8a46-bb3990a1e6f6-kube-api-access-l4vdn\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2\" (UID: \"868119ea-b49f-4566-8a46-bb3990a1e6f6\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2" Nov 03 02:05:04.511995 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:04.511861 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/868119ea-b49f-4566-8a46-bb3990a1e6f6-credential-socket\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2\" (UID: \"868119ea-b49f-4566-8a46-bb3990a1e6f6\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2" Nov 03 02:05:04.511995 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:04.511897 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/868119ea-b49f-4566-8a46-bb3990a1e6f6-istio-podinfo\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2\" (UID: \"868119ea-b49f-4566-8a46-bb3990a1e6f6\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2" Nov 03 02:05:04.511995 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:04.511974 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/868119ea-b49f-4566-8a46-bb3990a1e6f6-istio-envoy\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2\" (UID: \"868119ea-b49f-4566-8a46-bb3990a1e6f6\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2" Nov 03 02:05:04.512271 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:04.512041 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/868119ea-b49f-4566-8a46-bb3990a1e6f6-istio-data\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2\" (UID: \"868119ea-b49f-4566-8a46-bb3990a1e6f6\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2" Nov 03 02:05:04.512380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:04.512330 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/868119ea-b49f-4566-8a46-bb3990a1e6f6-credential-socket\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2\" (UID: \"868119ea-b49f-4566-8a46-bb3990a1e6f6\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2" Nov 03 02:05:04.512474 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:04.512450 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/868119ea-b49f-4566-8a46-bb3990a1e6f6-istio-data\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2\" (UID: \"868119ea-b49f-4566-8a46-bb3990a1e6f6\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2" Nov 03 02:05:04.512523 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:04.512510 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/868119ea-b49f-4566-8a46-bb3990a1e6f6-workload-certs\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2\" (UID: \"868119ea-b49f-4566-8a46-bb3990a1e6f6\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2" Nov 03 02:05:04.512557 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:04.512537 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/868119ea-b49f-4566-8a46-bb3990a1e6f6-workload-socket\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2\" (UID: \"868119ea-b49f-4566-8a46-bb3990a1e6f6\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2" Nov 03 02:05:04.512643 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:04.512616 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/868119ea-b49f-4566-8a46-bb3990a1e6f6-istiod-ca-cert\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2\" (UID: \"868119ea-b49f-4566-8a46-bb3990a1e6f6\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2" Nov 03 02:05:04.514391 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:04.514352 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/868119ea-b49f-4566-8a46-bb3990a1e6f6-istio-envoy\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2\" (UID: \"868119ea-b49f-4566-8a46-bb3990a1e6f6\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2" Nov 03 02:05:04.514607 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:04.514586 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/868119ea-b49f-4566-8a46-bb3990a1e6f6-istio-podinfo\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2\" (UID: \"868119ea-b49f-4566-8a46-bb3990a1e6f6\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2" Nov 03 02:05:04.565200 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:04.565127 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/868119ea-b49f-4566-8a46-bb3990a1e6f6-istio-token\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2\" (UID: \"868119ea-b49f-4566-8a46-bb3990a1e6f6\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2" Nov 03 02:05:04.565341 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:04.565265 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4vdn\" (UniqueName: \"kubernetes.io/projected/868119ea-b49f-4566-8a46-bb3990a1e6f6-kube-api-access-l4vdn\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2\" (UID: \"868119ea-b49f-4566-8a46-bb3990a1e6f6\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2" Nov 03 02:05:04.618621 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:04.618583 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2" Nov 03 02:05:05.035454 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:05.035369 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2"] Nov 03 02:05:05.041751 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:05:05.041720 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod868119ea_b49f_4566_8a46_bb3990a1e6f6.slice/crio-f25ca3af36c6193b6373bbc6ad8a4b0da5abd44124d7935a870501a02c1f07b5 WatchSource:0}: Error finding container f25ca3af36c6193b6373bbc6ad8a4b0da5abd44124d7935a870501a02c1f07b5: Status 404 returned error can't find the container with id f25ca3af36c6193b6373bbc6ad8a4b0da5abd44124d7935a870501a02c1f07b5 Nov 03 02:05:05.043987 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:05.043952 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:05:05.044067 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:05.044020 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:05:05.044067 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:05.044046 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:05:05.214927 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:05.214894 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2" event={"ID":"868119ea-b49f-4566-8a46-bb3990a1e6f6","Type":"ContainerStarted","Data":"baaf9d99689962b25fee4f5f220b168b506a4210f39550c37999e78a23f86a78"} Nov 03 02:05:05.215079 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:05.214944 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2" event={"ID":"868119ea-b49f-4566-8a46-bb3990a1e6f6","Type":"ContainerStarted","Data":"f25ca3af36c6193b6373bbc6ad8a4b0da5abd44124d7935a870501a02c1f07b5"} Nov 03 02:05:05.325250 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:05.325193 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2" podStartSLOduration=1.3251798670000001 podStartE2EDuration="1.325179867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:05:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:05:05.323515056 +0000 UTC m=+782.582713233" watchObservedRunningTime="2026-11-03 02:05:05.325179867 +0000 UTC m=+782.584378020" Nov 03 02:05:05.619328 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:05.619245 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2" Nov 03 02:05:06.621214 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:06.621178 5605 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 03 02:05:06.621632 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:06.621235 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2" podUID="868119ea-b49f-4566-8a46-bb3990a1e6f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 03 02:05:07.619254 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:07.619218 5605 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 03 02:05:07.619439 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:07.619283 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2" podUID="868119ea-b49f-4566-8a46-bb3990a1e6f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 03 02:05:08.618986 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:08.618950 5605 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 03 02:05:08.619414 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:08.619011 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2" podUID="868119ea-b49f-4566-8a46-bb3990a1e6f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 03 02:05:09.619211 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:09.619175 5605 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 03 02:05:09.619624 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:09.619234 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2" podUID="868119ea-b49f-4566-8a46-bb3990a1e6f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 03 02:05:10.619686 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:10.619646 5605 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 03 02:05:10.620074 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:10.619715 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2" podUID="868119ea-b49f-4566-8a46-bb3990a1e6f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 03 02:05:11.736629 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:11.735863 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2" Nov 03 02:05:11.737102 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:11.736781 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2" Nov 03 02:05:11.737619 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:11.737591 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2" Nov 03 02:05:20.413195 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:20.413162 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:05:23.912547 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:05:23.912500 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:05:34.895592 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:34.895550 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-l6wzt"] Nov 03 02:05:34.896078 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:34.895832 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-l6wzt" podUID="c1ba0fd6-b847-4fda-92f9-a9b29eb4e881" containerName="istio-proxy" containerID="cri-o://e7f4bf9f1864e7ce89492bb637110b3b21ad958ec41a4ef5555f0ff882078fa9" gracePeriod=30 Nov 03 02:05:36.424117 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:36.424082 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fb9aaea-f46v2"] Nov 03 02:05:36.427650 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:36.427630 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb9aaea-f46v2" Nov 03 02:05:36.468645 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:36.468615 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8c56fa6-4d99-4622-b342-dd39dd949145-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fb9aaea-f46v2\" (UID: \"d8c56fa6-4d99-4622-b342-dd39dd949145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb9aaea-f46v2" Nov 03 02:05:36.468767 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:36.468689 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8c56fa6-4d99-4622-b342-dd39dd949145-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fb9aaea-f46v2\" (UID: \"d8c56fa6-4d99-4622-b342-dd39dd949145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb9aaea-f46v2" Nov 03 02:05:36.468822 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:36.468803 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hjn8\" (UniqueName: \"kubernetes.io/projected/d8c56fa6-4d99-4622-b342-dd39dd949145-kube-api-access-4hjn8\") pod \"maas-api-revoke-sub-5fb9aaea-f46v2\" (UID: \"d8c56fa6-4d99-4622-b342-dd39dd949145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb9aaea-f46v2" Nov 03 02:05:36.483184 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:36.480883 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fb9aaea-f46v2"] Nov 03 02:05:36.525802 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:36.525771 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:05:36.525917 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:36.525772 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:05:36.525917 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:36.525809 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:05:36.525917 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:36.525880 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:05:36.569589 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:36.569555 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8c56fa6-4d99-4622-b342-dd39dd949145-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fb9aaea-f46v2\" (UID: \"d8c56fa6-4d99-4622-b342-dd39dd949145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb9aaea-f46v2" Nov 03 02:05:36.569718 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:36.569611 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8c56fa6-4d99-4622-b342-dd39dd949145-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fb9aaea-f46v2\" (UID: \"d8c56fa6-4d99-4622-b342-dd39dd949145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb9aaea-f46v2" Nov 03 02:05:36.569718 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:36.569644 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hjn8\" (UniqueName: \"kubernetes.io/projected/d8c56fa6-4d99-4622-b342-dd39dd949145-kube-api-access-4hjn8\") pod \"maas-api-revoke-sub-5fb9aaea-f46v2\" (UID: \"d8c56fa6-4d99-4622-b342-dd39dd949145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb9aaea-f46v2" Nov 03 02:05:36.570299 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:36.570275 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8c56fa6-4d99-4622-b342-dd39dd949145-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fb9aaea-f46v2\" (UID: \"d8c56fa6-4d99-4622-b342-dd39dd949145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb9aaea-f46v2" Nov 03 02:05:36.571920 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:36.571899 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8c56fa6-4d99-4622-b342-dd39dd949145-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fb9aaea-f46v2\" (UID: \"d8c56fa6-4d99-4622-b342-dd39dd949145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb9aaea-f46v2" Nov 03 02:05:36.622300 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:36.622263 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hjn8\" (UniqueName: \"kubernetes.io/projected/d8c56fa6-4d99-4622-b342-dd39dd949145-kube-api-access-4hjn8\") pod \"maas-api-revoke-sub-5fb9aaea-f46v2\" (UID: \"d8c56fa6-4d99-4622-b342-dd39dd949145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb9aaea-f46v2" Nov 03 02:05:36.761385 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:36.761280 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb9aaea-f46v2" Nov 03 02:05:37.126199 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:37.126164 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fb9aaea-f46v2"] Nov 03 02:05:37.132987 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:05:37.132954 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8c56fa6_4d99_4622_b342_dd39dd949145.slice/crio-3961727cd9ccef9f76abfa63de015ff6d9ca21e51aad3659de1f2868f9b00dcf WatchSource:0}: Error finding container 3961727cd9ccef9f76abfa63de015ff6d9ca21e51aad3659de1f2868f9b00dcf: Status 404 returned error can't find the container with id 3961727cd9ccef9f76abfa63de015ff6d9ca21e51aad3659de1f2868f9b00dcf Nov 03 02:05:37.374040 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:37.374004 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb9aaea-f46v2" event={"ID":"d8c56fa6-4d99-4622-b342-dd39dd949145","Type":"ContainerStarted","Data":"3961727cd9ccef9f76abfa63de015ff6d9ca21e51aad3659de1f2868f9b00dcf"} Nov 03 02:05:38.379529 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:38.379492 5605 generic.go:356] "Generic (PLEG): container finished" podID="d8c56fa6-4d99-4622-b342-dd39dd949145" containerID="d041fac9df564313f4d6bc8a288173ba1843859075498a5fdd281a8bec6f98f8" exitCode=0 Nov 03 02:05:38.379967 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:38.379602 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb9aaea-f46v2" event={"ID":"d8c56fa6-4d99-4622-b342-dd39dd949145","Type":"ContainerDied","Data":"d041fac9df564313f4d6bc8a288173ba1843859075498a5fdd281a8bec6f98f8"} Nov 03 02:05:39.515696 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:39.515670 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb9aaea-f46v2" Nov 03 02:05:39.600648 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:39.600617 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8c56fa6-4d99-4622-b342-dd39dd949145-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8c56fa6-4d99-4622-b342-dd39dd949145-maas-api-openshift-ca\") pod \"d8c56fa6-4d99-4622-b342-dd39dd949145\" (UID: \"d8c56fa6-4d99-4622-b342-dd39dd949145\") " Nov 03 02:05:39.600809 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:39.600722 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8c56fa6-4d99-4622-b342-dd39dd949145-kube-api-access-4hjn8\" (UniqueName: \"kubernetes.io/projected/d8c56fa6-4d99-4622-b342-dd39dd949145-kube-api-access-4hjn8\") pod \"d8c56fa6-4d99-4622-b342-dd39dd949145\" (UID: \"d8c56fa6-4d99-4622-b342-dd39dd949145\") " Nov 03 02:05:39.600809 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:39.600761 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8c56fa6-4d99-4622-b342-dd39dd949145-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8c56fa6-4d99-4622-b342-dd39dd949145-maas-api-xks-ca\") pod \"d8c56fa6-4d99-4622-b342-dd39dd949145\" (UID: \"d8c56fa6-4d99-4622-b342-dd39dd949145\") " Nov 03 02:05:39.601020 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:39.600993 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8c56fa6-4d99-4622-b342-dd39dd949145-maas-api-openshift-ca" pod "d8c56fa6-4d99-4622-b342-dd39dd949145" (UID: "d8c56fa6-4d99-4622-b342-dd39dd949145"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:05:39.602815 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:39.602785 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8c56fa6-4d99-4622-b342-dd39dd949145-maas-api-xks-ca" pod "d8c56fa6-4d99-4622-b342-dd39dd949145" (UID: "d8c56fa6-4d99-4622-b342-dd39dd949145"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:05:39.602917 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:39.602842 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8c56fa6-4d99-4622-b342-dd39dd949145-kube-api-access-4hjn8" pod "d8c56fa6-4d99-4622-b342-dd39dd949145" (UID: "d8c56fa6-4d99-4622-b342-dd39dd949145"). InnerVolumeSpecName "kube-api-access-4hjn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:05:39.701946 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:39.701853 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8c56fa6-4d99-4622-b342-dd39dd949145-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:05:39.701946 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:39.701892 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hjn8\" (UniqueName: \"kubernetes.io/projected/d8c56fa6-4d99-4622-b342-dd39dd949145-kube-api-access-4hjn8\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:05:39.701946 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:39.701902 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8c56fa6-4d99-4622-b342-dd39dd949145-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:05:40.132116 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:40.132087 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-l6wzt" Nov 03 02:05:40.207333 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:40.207301 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1ba0fd6-b847-4fda-92f9-a9b29eb4e881-kube-api-access-flc98\" (UniqueName: \"kubernetes.io/projected/c1ba0fd6-b847-4fda-92f9-a9b29eb4e881-kube-api-access-flc98\") pod \"c1ba0fd6-b847-4fda-92f9-a9b29eb4e881\" (UID: \"c1ba0fd6-b847-4fda-92f9-a9b29eb4e881\") " Nov 03 02:05:40.207491 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:40.207344 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1ba0fd6-b847-4fda-92f9-a9b29eb4e881-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1ba0fd6-b847-4fda-92f9-a9b29eb4e881-istio-data\") pod \"c1ba0fd6-b847-4fda-92f9-a9b29eb4e881\" (UID: \"c1ba0fd6-b847-4fda-92f9-a9b29eb4e881\") " Nov 03 02:05:40.207491 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:40.207395 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1ba0fd6-b847-4fda-92f9-a9b29eb4e881-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1ba0fd6-b847-4fda-92f9-a9b29eb4e881-workload-certs\") pod \"c1ba0fd6-b847-4fda-92f9-a9b29eb4e881\" (UID: \"c1ba0fd6-b847-4fda-92f9-a9b29eb4e881\") " Nov 03 02:05:40.207491 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:40.207427 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c1ba0fd6-b847-4fda-92f9-a9b29eb4e881-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1ba0fd6-b847-4fda-92f9-a9b29eb4e881-istio-podinfo\") pod \"c1ba0fd6-b847-4fda-92f9-a9b29eb4e881\" (UID: \"c1ba0fd6-b847-4fda-92f9-a9b29eb4e881\") " Nov 03 02:05:40.207663 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:40.207505 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1ba0fd6-b847-4fda-92f9-a9b29eb4e881-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1ba0fd6-b847-4fda-92f9-a9b29eb4e881-istio-envoy\") pod \"c1ba0fd6-b847-4fda-92f9-a9b29eb4e881\" (UID: \"c1ba0fd6-b847-4fda-92f9-a9b29eb4e881\") " Nov 03 02:05:40.207663 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:40.207564 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1ba0fd6-b847-4fda-92f9-a9b29eb4e881-istio-token\" (UniqueName: \"kubernetes.io/projected/c1ba0fd6-b847-4fda-92f9-a9b29eb4e881-istio-token\") pod \"c1ba0fd6-b847-4fda-92f9-a9b29eb4e881\" (UID: \"c1ba0fd6-b847-4fda-92f9-a9b29eb4e881\") " Nov 03 02:05:40.207663 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:40.207609 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1ba0fd6-b847-4fda-92f9-a9b29eb4e881-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1ba0fd6-b847-4fda-92f9-a9b29eb4e881-workload-socket\") pod \"c1ba0fd6-b847-4fda-92f9-a9b29eb4e881\" (UID: \"c1ba0fd6-b847-4fda-92f9-a9b29eb4e881\") " Nov 03 02:05:40.207663 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:40.207643 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1ba0fd6-b847-4fda-92f9-a9b29eb4e881-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1ba0fd6-b847-4fda-92f9-a9b29eb4e881-istiod-ca-cert\") pod \"c1ba0fd6-b847-4fda-92f9-a9b29eb4e881\" (UID: \"c1ba0fd6-b847-4fda-92f9-a9b29eb4e881\") " Nov 03 02:05:40.207881 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:40.207671 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1ba0fd6-b847-4fda-92f9-a9b29eb4e881-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1ba0fd6-b847-4fda-92f9-a9b29eb4e881-credential-socket\") pod \"c1ba0fd6-b847-4fda-92f9-a9b29eb4e881\" (UID: \"c1ba0fd6-b847-4fda-92f9-a9b29eb4e881\") " Nov 03 02:05:40.207881 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:40.207693 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1ba0fd6-b847-4fda-92f9-a9b29eb4e881-istio-data" pod "c1ba0fd6-b847-4fda-92f9-a9b29eb4e881" (UID: "c1ba0fd6-b847-4fda-92f9-a9b29eb4e881"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:05:40.207881 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:40.207794 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1ba0fd6-b847-4fda-92f9-a9b29eb4e881-workload-certs" pod "c1ba0fd6-b847-4fda-92f9-a9b29eb4e881" (UID: "c1ba0fd6-b847-4fda-92f9-a9b29eb4e881"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:05:40.208056 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:40.208034 5605 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1ba0fd6-b847-4fda-92f9-a9b29eb4e881-istio-data\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:05:40.208242 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:40.208059 5605 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1ba0fd6-b847-4fda-92f9-a9b29eb4e881-workload-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:05:40.208242 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:40.208224 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1ba0fd6-b847-4fda-92f9-a9b29eb4e881-workload-socket" pod "c1ba0fd6-b847-4fda-92f9-a9b29eb4e881" (UID: "c1ba0fd6-b847-4fda-92f9-a9b29eb4e881"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:05:40.208459 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:40.208426 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1ba0fd6-b847-4fda-92f9-a9b29eb4e881-credential-socket" pod "c1ba0fd6-b847-4fda-92f9-a9b29eb4e881" (UID: "c1ba0fd6-b847-4fda-92f9-a9b29eb4e881"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:05:40.208546 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:40.208477 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1ba0fd6-b847-4fda-92f9-a9b29eb4e881-istiod-ca-cert" pod "c1ba0fd6-b847-4fda-92f9-a9b29eb4e881" (UID: "c1ba0fd6-b847-4fda-92f9-a9b29eb4e881"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:05:40.209730 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:40.209706 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1ba0fd6-b847-4fda-92f9-a9b29eb4e881-istio-token" pod "c1ba0fd6-b847-4fda-92f9-a9b29eb4e881" (UID: "c1ba0fd6-b847-4fda-92f9-a9b29eb4e881"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:05:40.210026 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:40.210001 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1ba0fd6-b847-4fda-92f9-a9b29eb4e881-kube-api-access-flc98" pod "c1ba0fd6-b847-4fda-92f9-a9b29eb4e881" (UID: "c1ba0fd6-b847-4fda-92f9-a9b29eb4e881"). InnerVolumeSpecName "kube-api-access-flc98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:05:40.210234 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:40.210215 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c1ba0fd6-b847-4fda-92f9-a9b29eb4e881-istio-podinfo" pod "c1ba0fd6-b847-4fda-92f9-a9b29eb4e881" (UID: "c1ba0fd6-b847-4fda-92f9-a9b29eb4e881"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:05:40.210234 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:40.210223 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1ba0fd6-b847-4fda-92f9-a9b29eb4e881-istio-envoy" pod "c1ba0fd6-b847-4fda-92f9-a9b29eb4e881" (UID: "c1ba0fd6-b847-4fda-92f9-a9b29eb4e881"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:05:40.309003 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:40.308963 5605 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1ba0fd6-b847-4fda-92f9-a9b29eb4e881-istio-envoy\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:05:40.309141 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:40.309010 5605 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1ba0fd6-b847-4fda-92f9-a9b29eb4e881-istio-token\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:05:40.309141 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:40.309027 5605 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1ba0fd6-b847-4fda-92f9-a9b29eb4e881-workload-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:05:40.309141 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:40.309042 5605 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1ba0fd6-b847-4fda-92f9-a9b29eb4e881-istiod-ca-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:05:40.309141 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:40.309057 5605 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1ba0fd6-b847-4fda-92f9-a9b29eb4e881-credential-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:05:40.309141 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:40.309071 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flc98\" (UniqueName: \"kubernetes.io/projected/c1ba0fd6-b847-4fda-92f9-a9b29eb4e881-kube-api-access-flc98\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:05:40.309141 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:40.309086 5605 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1ba0fd6-b847-4fda-92f9-a9b29eb4e881-istio-podinfo\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:05:40.391433 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:40.391401 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb9aaea-f46v2" Nov 03 02:05:40.391647 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:40.391402 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb9aaea-f46v2" event={"ID":"d8c56fa6-4d99-4622-b342-dd39dd949145","Type":"ContainerDied","Data":"3961727cd9ccef9f76abfa63de015ff6d9ca21e51aad3659de1f2868f9b00dcf"} Nov 03 02:05:40.391647 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:40.391551 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3961727cd9ccef9f76abfa63de015ff6d9ca21e51aad3659de1f2868f9b00dcf" Nov 03 02:05:40.392830 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:40.392804 5605 generic.go:356] "Generic (PLEG): container finished" podID="c1ba0fd6-b847-4fda-92f9-a9b29eb4e881" containerID="e7f4bf9f1864e7ce89492bb637110b3b21ad958ec41a4ef5555f0ff882078fa9" exitCode=0 Nov 03 02:05:40.392971 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:40.392838 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-l6wzt" event={"ID":"c1ba0fd6-b847-4fda-92f9-a9b29eb4e881","Type":"ContainerDied","Data":"e7f4bf9f1864e7ce89492bb637110b3b21ad958ec41a4ef5555f0ff882078fa9"} Nov 03 02:05:40.392971 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:40.392861 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-l6wzt" Nov 03 02:05:40.392971 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:40.392883 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-l6wzt" event={"ID":"c1ba0fd6-b847-4fda-92f9-a9b29eb4e881","Type":"ContainerDied","Data":"1eec86d28eae5ab243e996560ae0d2bb5dac79215ef755c5360fcd6bf2251c21"} Nov 03 02:05:40.392971 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:40.392905 5605 scope.go:122] "RemoveContainer" containerID="e7f4bf9f1864e7ce89492bb637110b3b21ad958ec41a4ef5555f0ff882078fa9" Nov 03 02:05:40.401647 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:40.401629 5605 scope.go:122] "RemoveContainer" containerID="e7f4bf9f1864e7ce89492bb637110b3b21ad958ec41a4ef5555f0ff882078fa9" Nov 03 02:05:40.401996 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:05:40.401972 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7f4bf9f1864e7ce89492bb637110b3b21ad958ec41a4ef5555f0ff882078fa9\": container with ID starting with e7f4bf9f1864e7ce89492bb637110b3b21ad958ec41a4ef5555f0ff882078fa9 not found: ID does not exist" containerID="e7f4bf9f1864e7ce89492bb637110b3b21ad958ec41a4ef5555f0ff882078fa9" Nov 03 02:05:40.402097 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:40.402008 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7f4bf9f1864e7ce89492bb637110b3b21ad958ec41a4ef5555f0ff882078fa9"} err="failed to get container status \"e7f4bf9f1864e7ce89492bb637110b3b21ad958ec41a4ef5555f0ff882078fa9\": rpc error: code = NotFound desc = could not find container \"e7f4bf9f1864e7ce89492bb637110b3b21ad958ec41a4ef5555f0ff882078fa9\": container with ID starting with e7f4bf9f1864e7ce89492bb637110b3b21ad958ec41a4ef5555f0ff882078fa9 not found: ID does not exist" Nov 03 02:05:40.599498 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:40.599402 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-l6wzt"] Nov 03 02:05:40.606894 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:40.606861 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-l6wzt"] Nov 03 02:05:40.850754 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:40.850629 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1ba0fd6-b847-4fda-92f9-a9b29eb4e881" path="/var/lib/kubelet/pods/c1ba0fd6-b847-4fda-92f9-a9b29eb4e881/volumes" Nov 03 02:05:51.392886 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:51.392848 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-641c7353-fwvxm"] Nov 03 02:05:51.393475 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:51.393313 5605 cpu_manager.go:414] "containerMap: removing container" podUID="c1ba0fd6-b847-4fda-92f9-a9b29eb4e881" containerName="istio-proxy" Nov 03 02:05:51.393475 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:51.393333 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1ba0fd6-b847-4fda-92f9-a9b29eb4e881" containerName="istio-proxy" Nov 03 02:05:51.393475 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:51.393371 5605 cpu_manager.go:414] "containerMap: removing container" podUID="d8c56fa6-4d99-4622-b342-dd39dd949145" containerName="revoke-keys" Nov 03 02:05:51.393475 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:51.393380 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8c56fa6-4d99-4622-b342-dd39dd949145" containerName="revoke-keys" Nov 03 02:05:51.393475 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:51.393458 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8c56fa6-4d99-4622-b342-dd39dd949145" containerName="revoke-keys" Nov 03 02:05:51.393475 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:51.393470 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1ba0fd6-b847-4fda-92f9-a9b29eb4e881" containerName="istio-proxy" Nov 03 02:05:51.396407 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:51.396383 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-641c7353-fwvxm" Nov 03 02:05:51.450250 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:51.450226 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-641c7353-fwvxm"] Nov 03 02:05:51.494790 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:51.494754 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:05:51.494790 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:51.494754 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:05:51.495001 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:51.494803 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:05:51.495001 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:51.494803 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:05:51.520580 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:51.520542 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f01f063e-88ed-4dcd-850a-4ac3723f7472-maas-api-xks-ca\") pod \"maas-api-revoke-sub-641c7353-fwvxm\" (UID: \"f01f063e-88ed-4dcd-850a-4ac3723f7472\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-641c7353-fwvxm" Nov 03 02:05:51.520759 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:51.520619 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f01f063e-88ed-4dcd-850a-4ac3723f7472-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-641c7353-fwvxm\" (UID: \"f01f063e-88ed-4dcd-850a-4ac3723f7472\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-641c7353-fwvxm" Nov 03 02:05:51.520759 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:51.520721 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4f8t\" (UniqueName: \"kubernetes.io/projected/f01f063e-88ed-4dcd-850a-4ac3723f7472-kube-api-access-k4f8t\") pod \"maas-api-revoke-sub-641c7353-fwvxm\" (UID: \"f01f063e-88ed-4dcd-850a-4ac3723f7472\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-641c7353-fwvxm" Nov 03 02:05:51.622128 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:51.622092 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4f8t\" (UniqueName: \"kubernetes.io/projected/f01f063e-88ed-4dcd-850a-4ac3723f7472-kube-api-access-k4f8t\") pod \"maas-api-revoke-sub-641c7353-fwvxm\" (UID: \"f01f063e-88ed-4dcd-850a-4ac3723f7472\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-641c7353-fwvxm" Nov 03 02:05:51.622333 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:51.622141 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f01f063e-88ed-4dcd-850a-4ac3723f7472-maas-api-xks-ca\") pod \"maas-api-revoke-sub-641c7353-fwvxm\" (UID: \"f01f063e-88ed-4dcd-850a-4ac3723f7472\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-641c7353-fwvxm" Nov 03 02:05:51.622333 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:51.622190 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f01f063e-88ed-4dcd-850a-4ac3723f7472-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-641c7353-fwvxm\" (UID: \"f01f063e-88ed-4dcd-850a-4ac3723f7472\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-641c7353-fwvxm" Nov 03 02:05:51.622841 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:51.622820 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f01f063e-88ed-4dcd-850a-4ac3723f7472-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-641c7353-fwvxm\" (UID: \"f01f063e-88ed-4dcd-850a-4ac3723f7472\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-641c7353-fwvxm" Nov 03 02:05:51.624568 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:51.624547 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f01f063e-88ed-4dcd-850a-4ac3723f7472-maas-api-xks-ca\") pod \"maas-api-revoke-sub-641c7353-fwvxm\" (UID: \"f01f063e-88ed-4dcd-850a-4ac3723f7472\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-641c7353-fwvxm" Nov 03 02:05:51.675198 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:51.675147 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4f8t\" (UniqueName: \"kubernetes.io/projected/f01f063e-88ed-4dcd-850a-4ac3723f7472-kube-api-access-k4f8t\") pod \"maas-api-revoke-sub-641c7353-fwvxm\" (UID: \"f01f063e-88ed-4dcd-850a-4ac3723f7472\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-641c7353-fwvxm" Nov 03 02:05:51.709063 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:51.709036 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-641c7353-fwvxm" Nov 03 02:05:52.073879 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:52.073846 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-641c7353-fwvxm"] Nov 03 02:05:52.080034 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:05:52.080001 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf01f063e_88ed_4dcd_850a_4ac3723f7472.slice/crio-dea2305fd1d9595fd66d32833506333140a7259622da5e10e58fd0699043ae83 WatchSource:0}: Error finding container dea2305fd1d9595fd66d32833506333140a7259622da5e10e58fd0699043ae83: Status 404 returned error can't find the container with id dea2305fd1d9595fd66d32833506333140a7259622da5e10e58fd0699043ae83 Nov 03 02:05:52.451044 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:52.450949 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-641c7353-fwvxm" event={"ID":"f01f063e-88ed-4dcd-850a-4ac3723f7472","Type":"ContainerStarted","Data":"f7fc02c64e453c882bd3c2cddfe88ed1b3936e4732e5c7f2767aee9db4ff2bec"} Nov 03 02:05:52.451044 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:52.451008 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-641c7353-fwvxm" event={"ID":"f01f063e-88ed-4dcd-850a-4ac3723f7472","Type":"ContainerStarted","Data":"dea2305fd1d9595fd66d32833506333140a7259622da5e10e58fd0699043ae83"} Nov 03 02:05:52.558217 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:52.558154 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-641c7353-fwvxm" podStartSLOduration=1.558132683 podStartE2EDuration="1.558132683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:05:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:05:52.555831088 +0000 UTC m=+829.815029281" watchObservedRunningTime="2026-11-03 02:05:52.558132683 +0000 UTC m=+829.817330838" Nov 03 02:05:53.457191 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:53.457151 5605 generic.go:356] "Generic (PLEG): container finished" podID="f01f063e-88ed-4dcd-850a-4ac3723f7472" containerID="f7fc02c64e453c882bd3c2cddfe88ed1b3936e4732e5c7f2767aee9db4ff2bec" exitCode=0 Nov 03 02:05:53.457627 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:53.457237 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-641c7353-fwvxm" event={"ID":"f01f063e-88ed-4dcd-850a-4ac3723f7472","Type":"ContainerDied","Data":"f7fc02c64e453c882bd3c2cddfe88ed1b3936e4732e5c7f2767aee9db4ff2bec"} Nov 03 02:05:53.913529 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:05:53.913493 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:05:54.590274 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:54.590250 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-641c7353-fwvxm" Nov 03 02:05:54.760872 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:54.760784 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f01f063e-88ed-4dcd-850a-4ac3723f7472-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f01f063e-88ed-4dcd-850a-4ac3723f7472-maas-api-xks-ca\") pod \"f01f063e-88ed-4dcd-850a-4ac3723f7472\" (UID: \"f01f063e-88ed-4dcd-850a-4ac3723f7472\") " Nov 03 02:05:54.761047 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:54.760883 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f01f063e-88ed-4dcd-850a-4ac3723f7472-kube-api-access-k4f8t\" (UniqueName: \"kubernetes.io/projected/f01f063e-88ed-4dcd-850a-4ac3723f7472-kube-api-access-k4f8t\") pod \"f01f063e-88ed-4dcd-850a-4ac3723f7472\" (UID: \"f01f063e-88ed-4dcd-850a-4ac3723f7472\") " Nov 03 02:05:54.761047 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:54.760916 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f01f063e-88ed-4dcd-850a-4ac3723f7472-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f01f063e-88ed-4dcd-850a-4ac3723f7472-maas-api-openshift-ca\") pod \"f01f063e-88ed-4dcd-850a-4ac3723f7472\" (UID: \"f01f063e-88ed-4dcd-850a-4ac3723f7472\") " Nov 03 02:05:54.761309 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:54.761281 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f01f063e-88ed-4dcd-850a-4ac3723f7472-maas-api-openshift-ca" pod "f01f063e-88ed-4dcd-850a-4ac3723f7472" (UID: "f01f063e-88ed-4dcd-850a-4ac3723f7472"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:05:54.762946 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:54.762920 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f01f063e-88ed-4dcd-850a-4ac3723f7472-maas-api-xks-ca" pod "f01f063e-88ed-4dcd-850a-4ac3723f7472" (UID: "f01f063e-88ed-4dcd-850a-4ac3723f7472"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:05:54.763030 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:54.762961 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f01f063e-88ed-4dcd-850a-4ac3723f7472-kube-api-access-k4f8t" pod "f01f063e-88ed-4dcd-850a-4ac3723f7472" (UID: "f01f063e-88ed-4dcd-850a-4ac3723f7472"). InnerVolumeSpecName "kube-api-access-k4f8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:05:54.861853 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:54.861825 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4f8t\" (UniqueName: \"kubernetes.io/projected/f01f063e-88ed-4dcd-850a-4ac3723f7472-kube-api-access-k4f8t\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:05:54.861853 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:54.861852 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f01f063e-88ed-4dcd-850a-4ac3723f7472-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:05:54.862018 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:54.861863 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f01f063e-88ed-4dcd-850a-4ac3723f7472-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:05:55.018994 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:55.018917 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:05:55.468596 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:55.468564 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-641c7353-fwvxm" Nov 03 02:05:55.468772 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:55.468598 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-641c7353-fwvxm" event={"ID":"f01f063e-88ed-4dcd-850a-4ac3723f7472","Type":"ContainerDied","Data":"dea2305fd1d9595fd66d32833506333140a7259622da5e10e58fd0699043ae83"} Nov 03 02:05:55.468772 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:05:55.468642 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dea2305fd1d9595fd66d32833506333140a7259622da5e10e58fd0699043ae83" Nov 03 02:06:00.880019 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:00.879977 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz"] Nov 03 02:06:00.880470 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:00.880393 5605 cpu_manager.go:414] "containerMap: removing container" podUID="f01f063e-88ed-4dcd-850a-4ac3723f7472" containerName="revoke-keys" Nov 03 02:06:00.880470 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:00.880406 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f01f063e-88ed-4dcd-850a-4ac3723f7472" containerName="revoke-keys" Nov 03 02:06:00.880551 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:00.880481 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="f01f063e-88ed-4dcd-850a-4ac3723f7472" containerName="revoke-keys" Nov 03 02:06:00.883779 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:00.883758 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz" Nov 03 02:06:00.932538 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:00.932502 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f014c618-b1ce-44b0-b042-f1234c247228-istio-podinfo\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz\" (UID: \"f014c618-b1ce-44b0-b042-f1234c247228\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz" Nov 03 02:06:00.932756 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:00.932583 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f014c618-b1ce-44b0-b042-f1234c247228-istio-data\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz\" (UID: \"f014c618-b1ce-44b0-b042-f1234c247228\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz" Nov 03 02:06:00.932756 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:00.932613 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f014c618-b1ce-44b0-b042-f1234c247228-istiod-ca-cert\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz\" (UID: \"f014c618-b1ce-44b0-b042-f1234c247228\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz" Nov 03 02:06:00.932756 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:00.932631 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f014c618-b1ce-44b0-b042-f1234c247228-istio-token\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz\" (UID: \"f014c618-b1ce-44b0-b042-f1234c247228\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz" Nov 03 02:06:00.932899 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:00.932816 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f014c618-b1ce-44b0-b042-f1234c247228-workload-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz\" (UID: \"f014c618-b1ce-44b0-b042-f1234c247228\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz" Nov 03 02:06:00.932899 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:00.932824 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-1bccbd-openshift-default-dockercfg-c57w7\"" Nov 03 02:06:00.932899 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:00.932836 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjzf7\" (UniqueName: \"kubernetes.io/projected/f014c618-b1ce-44b0-b042-f1234c247228-kube-api-access-jjzf7\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz\" (UID: \"f014c618-b1ce-44b0-b042-f1234c247228\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz" Nov 03 02:06:00.933021 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:00.932908 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f014c618-b1ce-44b0-b042-f1234c247228-credential-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz\" (UID: \"f014c618-b1ce-44b0-b042-f1234c247228\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz" Nov 03 02:06:00.933021 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:00.932936 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f014c618-b1ce-44b0-b042-f1234c247228-istio-envoy\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz\" (UID: \"f014c618-b1ce-44b0-b042-f1234c247228\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz" Nov 03 02:06:00.933096 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:00.933022 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f014c618-b1ce-44b0-b042-f1234c247228-workload-certs\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz\" (UID: \"f014c618-b1ce-44b0-b042-f1234c247228\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz" Nov 03 02:06:00.981329 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:00.981295 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz"] Nov 03 02:06:01.033692 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:01.033654 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f014c618-b1ce-44b0-b042-f1234c247228-workload-certs\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz\" (UID: \"f014c618-b1ce-44b0-b042-f1234c247228\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz" Nov 03 02:06:01.033692 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:01.033705 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f014c618-b1ce-44b0-b042-f1234c247228-istio-podinfo\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz\" (UID: \"f014c618-b1ce-44b0-b042-f1234c247228\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz" Nov 03 02:06:01.033906 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:01.033822 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f014c618-b1ce-44b0-b042-f1234c247228-istio-data\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz\" (UID: \"f014c618-b1ce-44b0-b042-f1234c247228\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz" Nov 03 02:06:01.033906 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:01.033853 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f014c618-b1ce-44b0-b042-f1234c247228-istiod-ca-cert\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz\" (UID: \"f014c618-b1ce-44b0-b042-f1234c247228\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz" Nov 03 02:06:01.033906 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:01.033872 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f014c618-b1ce-44b0-b042-f1234c247228-istio-token\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz\" (UID: \"f014c618-b1ce-44b0-b042-f1234c247228\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz" Nov 03 02:06:01.034040 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:01.033961 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f014c618-b1ce-44b0-b042-f1234c247228-workload-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz\" (UID: \"f014c618-b1ce-44b0-b042-f1234c247228\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz" Nov 03 02:06:01.034040 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:01.033985 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjzf7\" (UniqueName: \"kubernetes.io/projected/f014c618-b1ce-44b0-b042-f1234c247228-kube-api-access-jjzf7\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz\" (UID: \"f014c618-b1ce-44b0-b042-f1234c247228\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz" Nov 03 02:06:01.034176 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:01.034145 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f014c618-b1ce-44b0-b042-f1234c247228-workload-certs\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz\" (UID: \"f014c618-b1ce-44b0-b042-f1234c247228\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz" Nov 03 02:06:01.034310 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:01.034171 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f014c618-b1ce-44b0-b042-f1234c247228-credential-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz\" (UID: \"f014c618-b1ce-44b0-b042-f1234c247228\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz" Nov 03 02:06:01.034401 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:01.034309 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f014c618-b1ce-44b0-b042-f1234c247228-istio-envoy\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz\" (UID: \"f014c618-b1ce-44b0-b042-f1234c247228\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz" Nov 03 02:06:01.034401 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:01.034350 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f014c618-b1ce-44b0-b042-f1234c247228-istio-data\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz\" (UID: \"f014c618-b1ce-44b0-b042-f1234c247228\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz" Nov 03 02:06:01.034401 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:01.034345 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f014c618-b1ce-44b0-b042-f1234c247228-workload-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz\" (UID: \"f014c618-b1ce-44b0-b042-f1234c247228\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz" Nov 03 02:06:01.034602 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:01.034583 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f014c618-b1ce-44b0-b042-f1234c247228-credential-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz\" (UID: \"f014c618-b1ce-44b0-b042-f1234c247228\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz" Nov 03 02:06:01.034859 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:01.034838 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f014c618-b1ce-44b0-b042-f1234c247228-istiod-ca-cert\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz\" (UID: \"f014c618-b1ce-44b0-b042-f1234c247228\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz" Nov 03 02:06:01.036288 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:01.036269 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f014c618-b1ce-44b0-b042-f1234c247228-istio-podinfo\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz\" (UID: \"f014c618-b1ce-44b0-b042-f1234c247228\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz" Nov 03 02:06:01.036505 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:01.036487 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f014c618-b1ce-44b0-b042-f1234c247228-istio-envoy\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz\" (UID: \"f014c618-b1ce-44b0-b042-f1234c247228\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz" Nov 03 02:06:01.087635 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:01.087597 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f014c618-b1ce-44b0-b042-f1234c247228-istio-token\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz\" (UID: \"f014c618-b1ce-44b0-b042-f1234c247228\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz" Nov 03 02:06:01.087794 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:01.087713 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjzf7\" (UniqueName: \"kubernetes.io/projected/f014c618-b1ce-44b0-b042-f1234c247228-kube-api-access-jjzf7\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz\" (UID: \"f014c618-b1ce-44b0-b042-f1234c247228\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz" Nov 03 02:06:01.196152 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:01.196057 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz" Nov 03 02:06:01.618717 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:01.618683 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz"] Nov 03 02:06:01.625121 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:06:01.625092 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf014c618_b1ce_44b0_b042_f1234c247228.slice/crio-9d70cda70786af9f333fbbdef619e3dbc281427a6556b61e636956ef0f3c8b5d WatchSource:0}: Error finding container 9d70cda70786af9f333fbbdef619e3dbc281427a6556b61e636956ef0f3c8b5d: Status 404 returned error can't find the container with id 9d70cda70786af9f333fbbdef619e3dbc281427a6556b61e636956ef0f3c8b5d Nov 03 02:06:01.627373 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:01.627325 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:06:01.627486 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:01.627426 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:06:01.627552 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:01.627501 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:06:02.507507 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:02.507458 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz" event={"ID":"f014c618-b1ce-44b0-b042-f1234c247228","Type":"ContainerStarted","Data":"b9928e7eb339922c3dd179c1542692e0e9f79374d5ba19da140c7c3bceaa757f"} Nov 03 02:06:02.507909 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:02.507565 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz" event={"ID":"f014c618-b1ce-44b0-b042-f1234c247228","Type":"ContainerStarted","Data":"9d70cda70786af9f333fbbdef619e3dbc281427a6556b61e636956ef0f3c8b5d"} Nov 03 02:06:02.729536 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:02.729474 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz" podStartSLOduration=2.729458256 podStartE2EDuration="2.729458256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:06:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:06:02.681915922 +0000 UTC m=+839.941114077" watchObservedRunningTime="2026-11-03 02:06:02.729458256 +0000 UTC m=+839.988656409" Nov 03 02:06:03.196252 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:03.196163 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz" Nov 03 02:06:03.198008 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:03.197972 5605 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 03 02:06:03.198170 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:03.198029 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz" podUID="f014c618-b1ce-44b0-b042-f1234c247228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 03 02:06:04.196869 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:04.196832 5605 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 03 02:06:04.197353 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:04.196893 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz" podUID="f014c618-b1ce-44b0-b042-f1234c247228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 03 02:06:05.196981 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:05.196944 5605 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 03 02:06:05.197409 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:05.197002 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz" podUID="f014c618-b1ce-44b0-b042-f1234c247228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 03 02:06:06.197275 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:06.197240 5605 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 03 02:06:06.197703 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:06.197306 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz" podUID="f014c618-b1ce-44b0-b042-f1234c247228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 03 02:06:07.196634 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:07.196600 5605 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 03 02:06:07.196816 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:07.196663 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz" podUID="f014c618-b1ce-44b0-b042-f1234c247228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 03 02:06:08.383871 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:08.383835 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz" Nov 03 02:06:08.385890 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:08.385868 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz" Nov 03 02:06:08.388783 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:08.388762 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz" Nov 03 02:06:10.682651 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:10.682619 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt"] Nov 03 02:06:10.687286 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:10.687269 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt" Nov 03 02:06:10.736054 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:10.736026 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-f7d9bb-openshift-default-dockercfg-dp5lm\"" Nov 03 02:06:10.760503 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:10.760472 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f563fffd-fb25-4763-98f7-fdec204a5637-istio-envoy\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt\" (UID: \"f563fffd-fb25-4763-98f7-fdec204a5637\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt" Nov 03 02:06:10.760641 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:10.760518 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf8ht\" (UniqueName: \"kubernetes.io/projected/f563fffd-fb25-4763-98f7-fdec204a5637-kube-api-access-jf8ht\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt\" (UID: \"f563fffd-fb25-4763-98f7-fdec204a5637\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt" Nov 03 02:06:10.760709 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:10.760661 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f563fffd-fb25-4763-98f7-fdec204a5637-istio-token\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt\" (UID: \"f563fffd-fb25-4763-98f7-fdec204a5637\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt" Nov 03 02:06:10.760709 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:10.760700 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f563fffd-fb25-4763-98f7-fdec204a5637-istio-podinfo\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt\" (UID: \"f563fffd-fb25-4763-98f7-fdec204a5637\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt" Nov 03 02:06:10.760819 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:10.760755 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f563fffd-fb25-4763-98f7-fdec204a5637-workload-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt\" (UID: \"f563fffd-fb25-4763-98f7-fdec204a5637\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt" Nov 03 02:06:10.760856 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:10.760831 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f563fffd-fb25-4763-98f7-fdec204a5637-credential-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt\" (UID: \"f563fffd-fb25-4763-98f7-fdec204a5637\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt" Nov 03 02:06:10.760912 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:10.760897 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f563fffd-fb25-4763-98f7-fdec204a5637-workload-certs\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt\" (UID: \"f563fffd-fb25-4763-98f7-fdec204a5637\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt" Nov 03 02:06:10.760948 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:10.760939 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f563fffd-fb25-4763-98f7-fdec204a5637-istiod-ca-cert\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt\" (UID: \"f563fffd-fb25-4763-98f7-fdec204a5637\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt" Nov 03 02:06:10.761005 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:10.760959 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f563fffd-fb25-4763-98f7-fdec204a5637-istio-data\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt\" (UID: \"f563fffd-fb25-4763-98f7-fdec204a5637\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt" Nov 03 02:06:10.784410 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:10.784368 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt"] Nov 03 02:06:10.862083 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:10.862053 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f563fffd-fb25-4763-98f7-fdec204a5637-istio-token\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt\" (UID: \"f563fffd-fb25-4763-98f7-fdec204a5637\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt" Nov 03 02:06:10.862083 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:10.862085 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f563fffd-fb25-4763-98f7-fdec204a5637-istio-podinfo\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt\" (UID: \"f563fffd-fb25-4763-98f7-fdec204a5637\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt" Nov 03 02:06:10.862321 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:10.862116 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f563fffd-fb25-4763-98f7-fdec204a5637-workload-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt\" (UID: \"f563fffd-fb25-4763-98f7-fdec204a5637\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt" Nov 03 02:06:10.862321 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:10.862137 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f563fffd-fb25-4763-98f7-fdec204a5637-credential-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt\" (UID: \"f563fffd-fb25-4763-98f7-fdec204a5637\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt" Nov 03 02:06:10.862321 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:10.862194 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f563fffd-fb25-4763-98f7-fdec204a5637-workload-certs\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt\" (UID: \"f563fffd-fb25-4763-98f7-fdec204a5637\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt" Nov 03 02:06:10.862321 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:10.862243 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f563fffd-fb25-4763-98f7-fdec204a5637-istiod-ca-cert\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt\" (UID: \"f563fffd-fb25-4763-98f7-fdec204a5637\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt" Nov 03 02:06:10.862321 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:10.862275 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f563fffd-fb25-4763-98f7-fdec204a5637-istio-data\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt\" (UID: \"f563fffd-fb25-4763-98f7-fdec204a5637\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt" Nov 03 02:06:10.862321 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:10.862315 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f563fffd-fb25-4763-98f7-fdec204a5637-istio-envoy\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt\" (UID: \"f563fffd-fb25-4763-98f7-fdec204a5637\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt" Nov 03 02:06:10.862668 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:10.862381 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf8ht\" (UniqueName: \"kubernetes.io/projected/f563fffd-fb25-4763-98f7-fdec204a5637-kube-api-access-jf8ht\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt\" (UID: \"f563fffd-fb25-4763-98f7-fdec204a5637\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt" Nov 03 02:06:10.862668 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:10.862584 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f563fffd-fb25-4763-98f7-fdec204a5637-workload-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt\" (UID: \"f563fffd-fb25-4763-98f7-fdec204a5637\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt" Nov 03 02:06:10.862668 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:10.862626 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f563fffd-fb25-4763-98f7-fdec204a5637-credential-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt\" (UID: \"f563fffd-fb25-4763-98f7-fdec204a5637\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt" Nov 03 02:06:10.862806 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:10.862765 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f563fffd-fb25-4763-98f7-fdec204a5637-workload-certs\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt\" (UID: \"f563fffd-fb25-4763-98f7-fdec204a5637\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt" Nov 03 02:06:10.862868 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:10.862822 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f563fffd-fb25-4763-98f7-fdec204a5637-istio-data\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt\" (UID: \"f563fffd-fb25-4763-98f7-fdec204a5637\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt" Nov 03 02:06:10.862998 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:10.862979 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f563fffd-fb25-4763-98f7-fdec204a5637-istiod-ca-cert\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt\" (UID: \"f563fffd-fb25-4763-98f7-fdec204a5637\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt" Nov 03 02:06:10.864601 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:10.864585 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f563fffd-fb25-4763-98f7-fdec204a5637-istio-podinfo\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt\" (UID: \"f563fffd-fb25-4763-98f7-fdec204a5637\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt" Nov 03 02:06:10.864660 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:10.864625 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f563fffd-fb25-4763-98f7-fdec204a5637-istio-envoy\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt\" (UID: \"f563fffd-fb25-4763-98f7-fdec204a5637\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt" Nov 03 02:06:10.914753 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:10.914728 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f563fffd-fb25-4763-98f7-fdec204a5637-istio-token\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt\" (UID: \"f563fffd-fb25-4763-98f7-fdec204a5637\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt" Nov 03 02:06:10.914889 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:10.914858 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf8ht\" (UniqueName: \"kubernetes.io/projected/f563fffd-fb25-4763-98f7-fdec204a5637-kube-api-access-jf8ht\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt\" (UID: \"f563fffd-fb25-4763-98f7-fdec204a5637\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt" Nov 03 02:06:10.999128 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:10.999058 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt" Nov 03 02:06:11.416189 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:11.416154 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt"] Nov 03 02:06:11.423229 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:06:11.423200 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf563fffd_fb25_4763_98f7_fdec204a5637.slice/crio-7b6dec0065781bd1ef27a8fbfbc707a16b5b03b1d20895b36d76f3ae03a79bb3 WatchSource:0}: Error finding container 7b6dec0065781bd1ef27a8fbfbc707a16b5b03b1d20895b36d76f3ae03a79bb3: Status 404 returned error can't find the container with id 7b6dec0065781bd1ef27a8fbfbc707a16b5b03b1d20895b36d76f3ae03a79bb3 Nov 03 02:06:11.425416 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:11.425381 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:06:11.425479 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:11.425454 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:06:11.425515 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:11.425486 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:06:11.553179 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:11.553129 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt" event={"ID":"f563fffd-fb25-4763-98f7-fdec204a5637","Type":"ContainerStarted","Data":"d68d37c09aa82fbbacd883e8ddc8b0f128922599bb32494745c7d246118ff3fb"} Nov 03 02:06:11.553341 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:11.553203 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt" event={"ID":"f563fffd-fb25-4763-98f7-fdec204a5637","Type":"ContainerStarted","Data":"7b6dec0065781bd1ef27a8fbfbc707a16b5b03b1d20895b36d76f3ae03a79bb3"} Nov 03 02:06:11.662142 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:11.662083 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt" podStartSLOduration=1.66206746 podStartE2EDuration="1.66206746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:06:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:06:11.661787008 +0000 UTC m=+848.920985162" watchObservedRunningTime="2026-11-03 02:06:11.66206746 +0000 UTC m=+848.921265672" Nov 03 02:06:12.000143 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:12.000110 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt" Nov 03 02:06:12.001688 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:12.001663 5605 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 03 02:06:12.001808 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:12.001708 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt" podUID="f563fffd-fb25-4763-98f7-fdec204a5637" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 03 02:06:13.000821 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:13.000776 5605 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 03 02:06:13.001476 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:13.001430 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt" podUID="f563fffd-fb25-4763-98f7-fdec204a5637" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 03 02:06:14.000222 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:14.000190 5605 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 03 02:06:14.000398 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:14.000236 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt" podUID="f563fffd-fb25-4763-98f7-fdec204a5637" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 03 02:06:14.999454 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:14.999418 5605 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 03 02:06:14.999916 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:14.999478 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt" podUID="f563fffd-fb25-4763-98f7-fdec204a5637" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 03 02:06:16.000766 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:16.000722 5605 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 03 02:06:16.001238 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:16.000791 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt" podUID="f563fffd-fb25-4763-98f7-fdec204a5637" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 03 02:06:16.999452 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:16.999419 5605 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 03 02:06:16.999617 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:16.999482 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt" podUID="f563fffd-fb25-4763-98f7-fdec204a5637" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 03 02:06:18.128717 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:18.128671 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt" Nov 03 02:06:18.129281 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:18.129072 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt" Nov 03 02:06:18.130205 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:18.130186 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt" Nov 03 02:06:20.216380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:20.216326 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:06:23.911760 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:06:23.911724 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:06:35.230899 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.230867 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4gq8z"] Nov 03 02:06:35.234610 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.234589 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4gq8z" Nov 03 02:06:35.277551 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.277524 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jl8k2"] Nov 03 02:06:35.280797 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.280781 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jl8k2" Nov 03 02:06:35.319457 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.319431 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d83861f-abc5-452e-a1d9-3148c6cd3774-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4gq8z\" (UID: \"1d83861f-abc5-452e-a1d9-3148c6cd3774\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4gq8z" Nov 03 02:06:35.319578 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.319468 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d83861f-abc5-452e-a1d9-3148c6cd3774-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4gq8z\" (UID: \"1d83861f-abc5-452e-a1d9-3148c6cd3774\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4gq8z" Nov 03 02:06:35.319578 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.319507 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d83861f-abc5-452e-a1d9-3148c6cd3774-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4gq8z\" (UID: \"1d83861f-abc5-452e-a1d9-3148c6cd3774\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4gq8z" Nov 03 02:06:35.319668 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.319614 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d83861f-abc5-452e-a1d9-3148c6cd3774-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4gq8z\" (UID: \"1d83861f-abc5-452e-a1d9-3148c6cd3774\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4gq8z" Nov 03 02:06:35.319668 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.319648 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d83861f-abc5-452e-a1d9-3148c6cd3774-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4gq8z\" (UID: \"1d83861f-abc5-452e-a1d9-3148c6cd3774\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4gq8z" Nov 03 02:06:35.319737 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.319678 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d83861f-abc5-452e-a1d9-3148c6cd3774-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4gq8z\" (UID: \"1d83861f-abc5-452e-a1d9-3148c6cd3774\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4gq8z" Nov 03 02:06:35.319737 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.319697 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d83861f-abc5-452e-a1d9-3148c6cd3774-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4gq8z\" (UID: \"1d83861f-abc5-452e-a1d9-3148c6cd3774\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4gq8z" Nov 03 02:06:35.319816 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.319739 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb7v5\" (UniqueName: \"kubernetes.io/projected/1d83861f-abc5-452e-a1d9-3148c6cd3774-kube-api-access-fb7v5\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4gq8z\" (UID: \"1d83861f-abc5-452e-a1d9-3148c6cd3774\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4gq8z" Nov 03 02:06:35.319816 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.319756 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d83861f-abc5-452e-a1d9-3148c6cd3774-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4gq8z\" (UID: \"1d83861f-abc5-452e-a1d9-3148c6cd3774\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4gq8z" Nov 03 02:06:35.368644 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.368610 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jl8k2"] Nov 03 02:06:35.420787 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.420761 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb7v5\" (UniqueName: \"kubernetes.io/projected/1d83861f-abc5-452e-a1d9-3148c6cd3774-kube-api-access-fb7v5\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4gq8z\" (UID: \"1d83861f-abc5-452e-a1d9-3148c6cd3774\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4gq8z" Nov 03 02:06:35.420921 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.420792 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d83861f-abc5-452e-a1d9-3148c6cd3774-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4gq8z\" (UID: \"1d83861f-abc5-452e-a1d9-3148c6cd3774\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4gq8z" Nov 03 02:06:35.420921 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.420822 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92507c2a-dbae-47e0-96f7-d50a7ccbd312-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jl8k2\" (UID: \"92507c2a-dbae-47e0-96f7-d50a7ccbd312\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jl8k2" Nov 03 02:06:35.420921 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.420856 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92507c2a-dbae-47e0-96f7-d50a7ccbd312-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jl8k2\" (UID: \"92507c2a-dbae-47e0-96f7-d50a7ccbd312\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jl8k2" Nov 03 02:06:35.421026 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.420981 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d83861f-abc5-452e-a1d9-3148c6cd3774-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4gq8z\" (UID: \"1d83861f-abc5-452e-a1d9-3148c6cd3774\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4gq8z" Nov 03 02:06:35.421026 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.421015 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d83861f-abc5-452e-a1d9-3148c6cd3774-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4gq8z\" (UID: \"1d83861f-abc5-452e-a1d9-3148c6cd3774\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4gq8z" Nov 03 02:06:35.421105 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.421055 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d83861f-abc5-452e-a1d9-3148c6cd3774-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4gq8z\" (UID: \"1d83861f-abc5-452e-a1d9-3148c6cd3774\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4gq8z" Nov 03 02:06:35.421105 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.421084 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92507c2a-dbae-47e0-96f7-d50a7ccbd312-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jl8k2\" (UID: \"92507c2a-dbae-47e0-96f7-d50a7ccbd312\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jl8k2" Nov 03 02:06:35.421191 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.421120 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92507c2a-dbae-47e0-96f7-d50a7ccbd312-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jl8k2\" (UID: \"92507c2a-dbae-47e0-96f7-d50a7ccbd312\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jl8k2" Nov 03 02:06:35.421191 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.421159 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92507c2a-dbae-47e0-96f7-d50a7ccbd312-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jl8k2\" (UID: \"92507c2a-dbae-47e0-96f7-d50a7ccbd312\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jl8k2" Nov 03 02:06:35.421380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.421328 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frwwm\" (UniqueName: \"kubernetes.io/projected/92507c2a-dbae-47e0-96f7-d50a7ccbd312-kube-api-access-frwwm\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jl8k2\" (UID: \"92507c2a-dbae-47e0-96f7-d50a7ccbd312\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jl8k2" Nov 03 02:06:35.421489 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.421437 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d83861f-abc5-452e-a1d9-3148c6cd3774-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4gq8z\" (UID: \"1d83861f-abc5-452e-a1d9-3148c6cd3774\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4gq8z" Nov 03 02:06:35.421489 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.421461 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d83861f-abc5-452e-a1d9-3148c6cd3774-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4gq8z\" (UID: \"1d83861f-abc5-452e-a1d9-3148c6cd3774\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4gq8z" Nov 03 02:06:35.421489 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.421486 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92507c2a-dbae-47e0-96f7-d50a7ccbd312-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jl8k2\" (UID: \"92507c2a-dbae-47e0-96f7-d50a7ccbd312\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jl8k2" Nov 03 02:06:35.421622 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.421515 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92507c2a-dbae-47e0-96f7-d50a7ccbd312-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jl8k2\" (UID: \"92507c2a-dbae-47e0-96f7-d50a7ccbd312\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jl8k2" Nov 03 02:06:35.421622 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.421588 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d83861f-abc5-452e-a1d9-3148c6cd3774-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4gq8z\" (UID: \"1d83861f-abc5-452e-a1d9-3148c6cd3774\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4gq8z" Nov 03 02:06:35.421721 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.421642 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d83861f-abc5-452e-a1d9-3148c6cd3774-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4gq8z\" (UID: \"1d83861f-abc5-452e-a1d9-3148c6cd3774\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4gq8z" Nov 03 02:06:35.421721 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.421686 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d83861f-abc5-452e-a1d9-3148c6cd3774-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4gq8z\" (UID: \"1d83861f-abc5-452e-a1d9-3148c6cd3774\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4gq8z" Nov 03 02:06:35.421841 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.421717 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92507c2a-dbae-47e0-96f7-d50a7ccbd312-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jl8k2\" (UID: \"92507c2a-dbae-47e0-96f7-d50a7ccbd312\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jl8k2" Nov 03 02:06:35.421841 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.421741 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d83861f-abc5-452e-a1d9-3148c6cd3774-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4gq8z\" (UID: \"1d83861f-abc5-452e-a1d9-3148c6cd3774\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4gq8z" Nov 03 02:06:35.422007 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.421838 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d83861f-abc5-452e-a1d9-3148c6cd3774-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4gq8z\" (UID: \"1d83861f-abc5-452e-a1d9-3148c6cd3774\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4gq8z" Nov 03 02:06:35.422007 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.421938 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d83861f-abc5-452e-a1d9-3148c6cd3774-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4gq8z\" (UID: \"1d83861f-abc5-452e-a1d9-3148c6cd3774\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4gq8z" Nov 03 02:06:35.427975 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.422334 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d83861f-abc5-452e-a1d9-3148c6cd3774-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4gq8z\" (UID: \"1d83861f-abc5-452e-a1d9-3148c6cd3774\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4gq8z" Nov 03 02:06:35.428085 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.427991 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4gq8z"] Nov 03 02:06:35.433329 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.433303 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d83861f-abc5-452e-a1d9-3148c6cd3774-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4gq8z\" (UID: \"1d83861f-abc5-452e-a1d9-3148c6cd3774\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4gq8z" Nov 03 02:06:35.433811 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.433791 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d83861f-abc5-452e-a1d9-3148c6cd3774-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4gq8z\" (UID: \"1d83861f-abc5-452e-a1d9-3148c6cd3774\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4gq8z" Nov 03 02:06:35.474040 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.474015 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb7v5\" (UniqueName: \"kubernetes.io/projected/1d83861f-abc5-452e-a1d9-3148c6cd3774-kube-api-access-fb7v5\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4gq8z\" (UID: \"1d83861f-abc5-452e-a1d9-3148c6cd3774\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4gq8z" Nov 03 02:06:35.474307 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.474290 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d83861f-abc5-452e-a1d9-3148c6cd3774-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4gq8z\" (UID: \"1d83861f-abc5-452e-a1d9-3148c6cd3774\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4gq8z" Nov 03 02:06:35.522952 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.522902 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92507c2a-dbae-47e0-96f7-d50a7ccbd312-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jl8k2\" (UID: \"92507c2a-dbae-47e0-96f7-d50a7ccbd312\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jl8k2" Nov 03 02:06:35.523038 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.522957 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92507c2a-dbae-47e0-96f7-d50a7ccbd312-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jl8k2\" (UID: \"92507c2a-dbae-47e0-96f7-d50a7ccbd312\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jl8k2" Nov 03 02:06:35.523038 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.522989 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92507c2a-dbae-47e0-96f7-d50a7ccbd312-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jl8k2\" (UID: \"92507c2a-dbae-47e0-96f7-d50a7ccbd312\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jl8k2" Nov 03 02:06:35.523143 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.523038 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92507c2a-dbae-47e0-96f7-d50a7ccbd312-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jl8k2\" (UID: \"92507c2a-dbae-47e0-96f7-d50a7ccbd312\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jl8k2" Nov 03 02:06:35.523143 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.523068 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92507c2a-dbae-47e0-96f7-d50a7ccbd312-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jl8k2\" (UID: \"92507c2a-dbae-47e0-96f7-d50a7ccbd312\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jl8k2" Nov 03 02:06:35.523143 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.523100 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92507c2a-dbae-47e0-96f7-d50a7ccbd312-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jl8k2\" (UID: \"92507c2a-dbae-47e0-96f7-d50a7ccbd312\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jl8k2" Nov 03 02:06:35.523143 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.523128 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frwwm\" (UniqueName: \"kubernetes.io/projected/92507c2a-dbae-47e0-96f7-d50a7ccbd312-kube-api-access-frwwm\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jl8k2\" (UID: \"92507c2a-dbae-47e0-96f7-d50a7ccbd312\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jl8k2" Nov 03 02:06:35.523346 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.523155 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92507c2a-dbae-47e0-96f7-d50a7ccbd312-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jl8k2\" (UID: \"92507c2a-dbae-47e0-96f7-d50a7ccbd312\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jl8k2" Nov 03 02:06:35.523346 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.523250 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92507c2a-dbae-47e0-96f7-d50a7ccbd312-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jl8k2\" (UID: \"92507c2a-dbae-47e0-96f7-d50a7ccbd312\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jl8k2" Nov 03 02:06:35.523510 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.523461 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92507c2a-dbae-47e0-96f7-d50a7ccbd312-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jl8k2\" (UID: \"92507c2a-dbae-47e0-96f7-d50a7ccbd312\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jl8k2" Nov 03 02:06:35.523652 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.523629 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92507c2a-dbae-47e0-96f7-d50a7ccbd312-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jl8k2\" (UID: \"92507c2a-dbae-47e0-96f7-d50a7ccbd312\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jl8k2" Nov 03 02:06:35.523715 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.523663 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92507c2a-dbae-47e0-96f7-d50a7ccbd312-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jl8k2\" (UID: \"92507c2a-dbae-47e0-96f7-d50a7ccbd312\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jl8k2" Nov 03 02:06:35.523715 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.523688 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92507c2a-dbae-47e0-96f7-d50a7ccbd312-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jl8k2\" (UID: \"92507c2a-dbae-47e0-96f7-d50a7ccbd312\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jl8k2" Nov 03 02:06:35.523828 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.523806 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92507c2a-dbae-47e0-96f7-d50a7ccbd312-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jl8k2\" (UID: \"92507c2a-dbae-47e0-96f7-d50a7ccbd312\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jl8k2" Nov 03 02:06:35.525053 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.525036 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92507c2a-dbae-47e0-96f7-d50a7ccbd312-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jl8k2\" (UID: \"92507c2a-dbae-47e0-96f7-d50a7ccbd312\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jl8k2" Nov 03 02:06:35.525517 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.525501 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92507c2a-dbae-47e0-96f7-d50a7ccbd312-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jl8k2\" (UID: \"92507c2a-dbae-47e0-96f7-d50a7ccbd312\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jl8k2" Nov 03 02:06:35.547462 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.547442 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4gq8z" Nov 03 02:06:35.575858 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.575825 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92507c2a-dbae-47e0-96f7-d50a7ccbd312-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jl8k2\" (UID: \"92507c2a-dbae-47e0-96f7-d50a7ccbd312\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jl8k2" Nov 03 02:06:35.576009 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.575990 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frwwm\" (UniqueName: \"kubernetes.io/projected/92507c2a-dbae-47e0-96f7-d50a7ccbd312-kube-api-access-frwwm\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jl8k2\" (UID: \"92507c2a-dbae-47e0-96f7-d50a7ccbd312\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jl8k2" Nov 03 02:06:35.592840 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.592816 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jl8k2" Nov 03 02:06:35.943835 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.943807 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4gq8z"] Nov 03 02:06:35.949208 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:06:35.949180 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d83861f_abc5_452e_a1d9_3148c6cd3774.slice/crio-3ef26fb668d20b73c481bb6e525e40cb3afb9f5dad0727a00603ab5d825af592 WatchSource:0}: Error finding container 3ef26fb668d20b73c481bb6e525e40cb3afb9f5dad0727a00603ab5d825af592: Status 404 returned error can't find the container with id 3ef26fb668d20b73c481bb6e525e40cb3afb9f5dad0727a00603ab5d825af592 Nov 03 02:06:35.951410 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.951376 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:06:35.951504 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.951456 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:06:35.951504 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.951488 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:06:35.994277 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:35.994254 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jl8k2"] Nov 03 02:06:36.005313 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:06:36.005284 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92507c2a_dbae_47e0_96f7_d50a7ccbd312.slice/crio-f304fe693881b91ac4ef6bf0dd59002963d71b406b5573dc0b073e279cd28e3a WatchSource:0}: Error finding container f304fe693881b91ac4ef6bf0dd59002963d71b406b5573dc0b073e279cd28e3a: Status 404 returned error can't find the container with id f304fe693881b91ac4ef6bf0dd59002963d71b406b5573dc0b073e279cd28e3a Nov 03 02:06:36.007429 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:36.007396 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:06:36.007526 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:36.007474 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:06:36.007589 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:36.007523 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:06:36.712575 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:36.712532 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jl8k2" event={"ID":"92507c2a-dbae-47e0-96f7-d50a7ccbd312","Type":"ContainerStarted","Data":"5d6a51e34152092feb4e7da6b0ca6e8c717b620de6ffe073271b83e1bed9eb0c"} Nov 03 02:06:36.712941 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:36.712602 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jl8k2" event={"ID":"92507c2a-dbae-47e0-96f7-d50a7ccbd312","Type":"ContainerStarted","Data":"f304fe693881b91ac4ef6bf0dd59002963d71b406b5573dc0b073e279cd28e3a"} Nov 03 02:06:36.715165 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:36.715096 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4gq8z" event={"ID":"1d83861f-abc5-452e-a1d9-3148c6cd3774","Type":"ContainerStarted","Data":"52987e50d164950fed2ae55ed3a5a64e4e1443e07d9fb4ab698587690ca7825d"} Nov 03 02:06:36.715165 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:36.715137 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4gq8z" event={"ID":"1d83861f-abc5-452e-a1d9-3148c6cd3774","Type":"ContainerStarted","Data":"3ef26fb668d20b73c481bb6e525e40cb3afb9f5dad0727a00603ab5d825af592"} Nov 03 02:06:36.826727 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:36.826674 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jl8k2" podStartSLOduration=1.826659101 podStartE2EDuration="1.826659101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:06:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:06:36.825024122 +0000 UTC m=+874.084222276" watchObservedRunningTime="2026-11-03 02:06:36.826659101 +0000 UTC m=+874.085857255" Nov 03 02:06:36.937157 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:36.937086 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4gq8z" podStartSLOduration=1.937069294 podStartE2EDuration="1.937069294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:06:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:06:36.935028157 +0000 UTC m=+874.194226343" watchObservedRunningTime="2026-11-03 02:06:36.937069294 +0000 UTC m=+874.196267646" Nov 03 02:06:37.547896 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:37.547860 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4gq8z" Nov 03 02:06:37.549172 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:37.549149 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-4gq8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 03 02:06:37.549229 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:37.549197 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4gq8z" podUID="1d83861f-abc5-452e-a1d9-3148c6cd3774" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 03 02:06:37.598381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:37.597418 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jl8k2" Nov 03 02:06:37.603387 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:37.601781 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-jl8k2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 03 02:06:37.603387 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:37.601829 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jl8k2" podUID="92507c2a-dbae-47e0-96f7-d50a7ccbd312" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 03 02:06:38.548304 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:38.548264 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-4gq8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 03 02:06:38.548742 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:38.548330 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4gq8z" podUID="1d83861f-abc5-452e-a1d9-3148c6cd3774" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 03 02:06:38.604383 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:38.602344 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-jl8k2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 03 02:06:38.604383 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:38.602440 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jl8k2" podUID="92507c2a-dbae-47e0-96f7-d50a7ccbd312" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 03 02:06:39.550662 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:39.550631 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-4gq8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 03 02:06:39.551035 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:39.550689 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4gq8z" podUID="1d83861f-abc5-452e-a1d9-3148c6cd3774" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 03 02:06:39.595791 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:39.595751 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-jl8k2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 03 02:06:39.596081 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:39.596046 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jl8k2" podUID="92507c2a-dbae-47e0-96f7-d50a7ccbd312" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 03 02:06:40.548245 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:40.548210 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-4gq8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 03 02:06:40.548466 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:40.548262 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4gq8z" podUID="1d83861f-abc5-452e-a1d9-3148c6cd3774" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 03 02:06:40.594381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:40.594102 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-jl8k2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 03 02:06:40.594381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:40.594163 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jl8k2" podUID="92507c2a-dbae-47e0-96f7-d50a7ccbd312" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 03 02:06:41.547968 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:41.547921 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-4gq8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 03 02:06:41.548170 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:41.547980 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4gq8z" podUID="1d83861f-abc5-452e-a1d9-3148c6cd3774" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 03 02:06:41.593775 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:41.593732 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-jl8k2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 03 02:06:41.594047 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:41.594017 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jl8k2" podUID="92507c2a-dbae-47e0-96f7-d50a7ccbd312" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 03 02:06:42.548253 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:42.548215 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-4gq8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 03 02:06:42.548689 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:42.548279 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4gq8z" podUID="1d83861f-abc5-452e-a1d9-3148c6cd3774" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 03 02:06:42.593681 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:42.593630 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-jl8k2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 03 02:06:42.593817 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:42.593703 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jl8k2" podUID="92507c2a-dbae-47e0-96f7-d50a7ccbd312" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 03 02:06:42.859706 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:42.859662 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:06:43.550769 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:43.550729 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-4gq8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 03 02:06:43.551386 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:43.551332 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4gq8z" podUID="1d83861f-abc5-452e-a1d9-3148c6cd3774" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 03 02:06:43.594132 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:43.594097 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-jl8k2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 03 02:06:43.594414 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:43.594377 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jl8k2" podUID="92507c2a-dbae-47e0-96f7-d50a7ccbd312" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 03 02:06:45.044210 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:45.044177 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jl8k2" Nov 03 02:06:45.045193 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:45.045161 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jl8k2" Nov 03 02:06:45.045966 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:45.045943 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jl8k2" Nov 03 02:06:45.124698 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:45.124658 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4gq8z" Nov 03 02:06:45.125111 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:45.125085 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4gq8z" Nov 03 02:06:45.125941 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:06:45.125922 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4gq8z" Nov 03 02:06:53.913521 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:06:53.913486 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:07:07.773056 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:07.773017 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:07:15.293910 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:15.293819 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g"] Nov 03 02:07:15.298865 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:15.298836 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g" Nov 03 02:07:15.326420 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:15.326376 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frlq4\" (UniqueName: \"kubernetes.io/projected/f05a0c36-8c90-4813-b84c-0825be1947a6-kube-api-access-frlq4\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g\" (UID: \"f05a0c36-8c90-4813-b84c-0825be1947a6\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g" Nov 03 02:07:15.326607 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:15.326446 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f05a0c36-8c90-4813-b84c-0825be1947a6-istio-data\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g\" (UID: \"f05a0c36-8c90-4813-b84c-0825be1947a6\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g" Nov 03 02:07:15.326607 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:15.326481 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f05a0c36-8c90-4813-b84c-0825be1947a6-istio-podinfo\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g\" (UID: \"f05a0c36-8c90-4813-b84c-0825be1947a6\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g" Nov 03 02:07:15.326607 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:15.326508 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f05a0c36-8c90-4813-b84c-0825be1947a6-workload-certs\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g\" (UID: \"f05a0c36-8c90-4813-b84c-0825be1947a6\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g" Nov 03 02:07:15.326607 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:15.326534 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f05a0c36-8c90-4813-b84c-0825be1947a6-istio-token\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g\" (UID: \"f05a0c36-8c90-4813-b84c-0825be1947a6\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g" Nov 03 02:07:15.326760 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:15.326632 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f05a0c36-8c90-4813-b84c-0825be1947a6-istio-envoy\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g\" (UID: \"f05a0c36-8c90-4813-b84c-0825be1947a6\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g" Nov 03 02:07:15.326760 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:15.326743 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f05a0c36-8c90-4813-b84c-0825be1947a6-workload-socket\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g\" (UID: \"f05a0c36-8c90-4813-b84c-0825be1947a6\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g" Nov 03 02:07:15.326836 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:15.326764 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f05a0c36-8c90-4813-b84c-0825be1947a6-credential-socket\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g\" (UID: \"f05a0c36-8c90-4813-b84c-0825be1947a6\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g" Nov 03 02:07:15.326836 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:15.326781 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f05a0c36-8c90-4813-b84c-0825be1947a6-istiod-ca-cert\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g\" (UID: \"f05a0c36-8c90-4813-b84c-0825be1947a6\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g" Nov 03 02:07:15.347608 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:15.347570 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-c7d758-openshift-default-dockercfg-kst7d\"" Nov 03 02:07:15.396584 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:15.396541 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g"] Nov 03 02:07:15.428387 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:15.428303 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f05a0c36-8c90-4813-b84c-0825be1947a6-istio-data\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g\" (UID: \"f05a0c36-8c90-4813-b84c-0825be1947a6\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g" Nov 03 02:07:15.428577 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:15.428420 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f05a0c36-8c90-4813-b84c-0825be1947a6-istio-podinfo\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g\" (UID: \"f05a0c36-8c90-4813-b84c-0825be1947a6\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g" Nov 03 02:07:15.428577 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:15.428469 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f05a0c36-8c90-4813-b84c-0825be1947a6-workload-certs\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g\" (UID: \"f05a0c36-8c90-4813-b84c-0825be1947a6\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g" Nov 03 02:07:15.428577 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:15.428498 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f05a0c36-8c90-4813-b84c-0825be1947a6-istio-token\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g\" (UID: \"f05a0c36-8c90-4813-b84c-0825be1947a6\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g" Nov 03 02:07:15.428577 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:15.428556 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f05a0c36-8c90-4813-b84c-0825be1947a6-istio-envoy\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g\" (UID: \"f05a0c36-8c90-4813-b84c-0825be1947a6\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g" Nov 03 02:07:15.428789 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:15.428667 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f05a0c36-8c90-4813-b84c-0825be1947a6-workload-socket\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g\" (UID: \"f05a0c36-8c90-4813-b84c-0825be1947a6\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g" Nov 03 02:07:15.428789 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:15.428693 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f05a0c36-8c90-4813-b84c-0825be1947a6-credential-socket\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g\" (UID: \"f05a0c36-8c90-4813-b84c-0825be1947a6\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g" Nov 03 02:07:15.428789 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:15.428719 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f05a0c36-8c90-4813-b84c-0825be1947a6-istiod-ca-cert\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g\" (UID: \"f05a0c36-8c90-4813-b84c-0825be1947a6\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g" Nov 03 02:07:15.428930 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:15.428836 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f05a0c36-8c90-4813-b84c-0825be1947a6-workload-certs\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g\" (UID: \"f05a0c36-8c90-4813-b84c-0825be1947a6\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g" Nov 03 02:07:15.429043 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:15.428938 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frlq4\" (UniqueName: \"kubernetes.io/projected/f05a0c36-8c90-4813-b84c-0825be1947a6-kube-api-access-frlq4\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g\" (UID: \"f05a0c36-8c90-4813-b84c-0825be1947a6\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g" Nov 03 02:07:15.429043 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:15.429013 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f05a0c36-8c90-4813-b84c-0825be1947a6-istio-data\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g\" (UID: \"f05a0c36-8c90-4813-b84c-0825be1947a6\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g" Nov 03 02:07:15.429132 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:15.429013 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f05a0c36-8c90-4813-b84c-0825be1947a6-credential-socket\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g\" (UID: \"f05a0c36-8c90-4813-b84c-0825be1947a6\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g" Nov 03 02:07:15.429339 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:15.429308 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f05a0c36-8c90-4813-b84c-0825be1947a6-workload-socket\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g\" (UID: \"f05a0c36-8c90-4813-b84c-0825be1947a6\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g" Nov 03 02:07:15.429547 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:15.429523 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f05a0c36-8c90-4813-b84c-0825be1947a6-istiod-ca-cert\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g\" (UID: \"f05a0c36-8c90-4813-b84c-0825be1947a6\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g" Nov 03 02:07:15.430948 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:15.430919 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f05a0c36-8c90-4813-b84c-0825be1947a6-istio-envoy\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g\" (UID: \"f05a0c36-8c90-4813-b84c-0825be1947a6\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g" Nov 03 02:07:15.430948 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:15.430943 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f05a0c36-8c90-4813-b84c-0825be1947a6-istio-podinfo\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g\" (UID: \"f05a0c36-8c90-4813-b84c-0825be1947a6\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g" Nov 03 02:07:15.482075 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:15.482033 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f05a0c36-8c90-4813-b84c-0825be1947a6-istio-token\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g\" (UID: \"f05a0c36-8c90-4813-b84c-0825be1947a6\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g" Nov 03 02:07:15.482207 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:15.482086 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frlq4\" (UniqueName: \"kubernetes.io/projected/f05a0c36-8c90-4813-b84c-0825be1947a6-kube-api-access-frlq4\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g\" (UID: \"f05a0c36-8c90-4813-b84c-0825be1947a6\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g" Nov 03 02:07:15.611665 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:15.611618 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g" Nov 03 02:07:16.035405 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:16.035301 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g"] Nov 03 02:07:16.040980 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:07:16.040939 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf05a0c36_8c90_4813_b84c_0825be1947a6.slice/crio-4a661a2ff4b673c7ec09c6a439a71f8a1fd269aa47aa2f58e4b056a7eab00938 WatchSource:0}: Error finding container 4a661a2ff4b673c7ec09c6a439a71f8a1fd269aa47aa2f58e4b056a7eab00938: Status 404 returned error can't find the container with id 4a661a2ff4b673c7ec09c6a439a71f8a1fd269aa47aa2f58e4b056a7eab00938 Nov 03 02:07:16.043382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:16.043327 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:07:16.043456 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:16.043419 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:07:16.043456 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:16.043451 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:07:16.929556 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:16.929506 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g" event={"ID":"f05a0c36-8c90-4813-b84c-0825be1947a6","Type":"ContainerStarted","Data":"5524a870ee429e5a3c4d6dce6fe687c7d70e14239f347e681228972ad5e05016"} Nov 03 02:07:16.930982 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:16.930953 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g" event={"ID":"f05a0c36-8c90-4813-b84c-0825be1947a6","Type":"ContainerStarted","Data":"4a661a2ff4b673c7ec09c6a439a71f8a1fd269aa47aa2f58e4b056a7eab00938"} Nov 03 02:07:17.137992 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:17.137935 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g" podStartSLOduration=2.137916877 podStartE2EDuration="2.137916877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:07:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:07:17.089824705 +0000 UTC m=+914.349022888" watchObservedRunningTime="2026-11-03 02:07:17.137916877 +0000 UTC m=+914.397115033" Nov 03 02:07:17.611838 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:17.611794 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g" Nov 03 02:07:17.613120 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:17.613095 5605 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 03 02:07:17.613186 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:17.613148 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g" podUID="f05a0c36-8c90-4813-b84c-0825be1947a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 03 02:07:18.612800 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:18.612759 5605 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 03 02:07:18.613208 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:18.612817 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g" podUID="f05a0c36-8c90-4813-b84c-0825be1947a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 03 02:07:19.612948 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:19.612898 5605 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 03 02:07:19.613503 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:19.612981 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g" podUID="f05a0c36-8c90-4813-b84c-0825be1947a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 03 02:07:20.612576 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:20.612531 5605 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 03 02:07:20.612755 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:20.612609 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g" podUID="f05a0c36-8c90-4813-b84c-0825be1947a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 03 02:07:21.612451 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:21.612340 5605 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 03 02:07:21.612851 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:21.612482 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g" podUID="f05a0c36-8c90-4813-b84c-0825be1947a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 03 02:07:22.821162 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:22.821123 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g" Nov 03 02:07:22.821684 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:22.821656 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g" Nov 03 02:07:22.823425 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:22.823402 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g" Nov 03 02:07:23.912588 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:07:23.912548 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:07:29.001383 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:28.992475 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-s9dp7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:07:29.001383 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:28.992564 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s9dp7" podUID="c3ef2131-720c-4db4-8c4f-e618edf4b5e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:07:36.094417 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:36.094379 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:07:39.689634 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:39.689598 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-1bccbd/distinct-w1-1bccbd-kserve-5dfc85555f-tkvg6"] Nov 03 02:07:39.693409 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:39.693382 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1bccbd/distinct-w1-1bccbd-kserve-5dfc85555f-tkvg6" Nov 03 02:07:39.753345 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:39.753304 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd4a9d44-c63e-4f67-bcac-a4f98ecf1796-dshm\") pod \"distinct-w1-1bccbd-kserve-5dfc85555f-tkvg6\" (UID: \"dd4a9d44-c63e-4f67-bcac-a4f98ecf1796\") " pod="e2e-models-e2e-worker-w1-1bccbd/distinct-w1-1bccbd-kserve-5dfc85555f-tkvg6" Nov 03 02:07:39.753345 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:39.753374 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd4a9d44-c63e-4f67-bcac-a4f98ecf1796-home\") pod \"distinct-w1-1bccbd-kserve-5dfc85555f-tkvg6\" (UID: \"dd4a9d44-c63e-4f67-bcac-a4f98ecf1796\") " pod="e2e-models-e2e-worker-w1-1bccbd/distinct-w1-1bccbd-kserve-5dfc85555f-tkvg6" Nov 03 02:07:39.753608 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:39.753409 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd4a9d44-c63e-4f67-bcac-a4f98ecf1796-model-cache\") pod \"distinct-w1-1bccbd-kserve-5dfc85555f-tkvg6\" (UID: \"dd4a9d44-c63e-4f67-bcac-a4f98ecf1796\") " pod="e2e-models-e2e-worker-w1-1bccbd/distinct-w1-1bccbd-kserve-5dfc85555f-tkvg6" Nov 03 02:07:39.753608 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:39.753429 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd4a9d44-c63e-4f67-bcac-a4f98ecf1796-tmp-dir\") pod \"distinct-w1-1bccbd-kserve-5dfc85555f-tkvg6\" (UID: \"dd4a9d44-c63e-4f67-bcac-a4f98ecf1796\") " pod="e2e-models-e2e-worker-w1-1bccbd/distinct-w1-1bccbd-kserve-5dfc85555f-tkvg6" Nov 03 02:07:39.753688 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:39.753616 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd4a9d44-c63e-4f67-bcac-a4f98ecf1796-tls-certs\") pod \"distinct-w1-1bccbd-kserve-5dfc85555f-tkvg6\" (UID: \"dd4a9d44-c63e-4f67-bcac-a4f98ecf1796\") " pod="e2e-models-e2e-worker-w1-1bccbd/distinct-w1-1bccbd-kserve-5dfc85555f-tkvg6" Nov 03 02:07:39.753688 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:39.753672 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfg2z\" (UniqueName: \"kubernetes.io/projected/dd4a9d44-c63e-4f67-bcac-a4f98ecf1796-kube-api-access-cfg2z\") pod \"distinct-w1-1bccbd-kserve-5dfc85555f-tkvg6\" (UID: \"dd4a9d44-c63e-4f67-bcac-a4f98ecf1796\") " pod="e2e-models-e2e-worker-w1-1bccbd/distinct-w1-1bccbd-kserve-5dfc85555f-tkvg6" Nov 03 02:07:39.791276 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:39.791238 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-1bccbd\"/\"openshift-service-ca.crt\"" Nov 03 02:07:39.791276 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:39.791276 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-1bccbd\"/\"default-dockercfg-zdpps\"" Nov 03 02:07:39.791557 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:39.791334 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-1bccbd\"/\"kube-root-ca.crt\"" Nov 03 02:07:39.791557 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:39.791401 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-1bccbd\"/\"distinct-w1-1bccbd-kserve-self-signed-certs\"" Nov 03 02:07:39.793075 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:39.793053 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-1bccbd/distinct-w1-1bccbd-kserve-5dfc85555f-tkvg6"] Nov 03 02:07:39.858380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:39.856015 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd4a9d44-c63e-4f67-bcac-a4f98ecf1796-dshm\") pod \"distinct-w1-1bccbd-kserve-5dfc85555f-tkvg6\" (UID: \"dd4a9d44-c63e-4f67-bcac-a4f98ecf1796\") " pod="e2e-models-e2e-worker-w1-1bccbd/distinct-w1-1bccbd-kserve-5dfc85555f-tkvg6" Nov 03 02:07:39.858380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:39.856090 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd4a9d44-c63e-4f67-bcac-a4f98ecf1796-home\") pod \"distinct-w1-1bccbd-kserve-5dfc85555f-tkvg6\" (UID: \"dd4a9d44-c63e-4f67-bcac-a4f98ecf1796\") " pod="e2e-models-e2e-worker-w1-1bccbd/distinct-w1-1bccbd-kserve-5dfc85555f-tkvg6" Nov 03 02:07:39.858380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:39.856141 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd4a9d44-c63e-4f67-bcac-a4f98ecf1796-model-cache\") pod \"distinct-w1-1bccbd-kserve-5dfc85555f-tkvg6\" (UID: \"dd4a9d44-c63e-4f67-bcac-a4f98ecf1796\") " pod="e2e-models-e2e-worker-w1-1bccbd/distinct-w1-1bccbd-kserve-5dfc85555f-tkvg6" Nov 03 02:07:39.858380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:39.856173 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd4a9d44-c63e-4f67-bcac-a4f98ecf1796-tmp-dir\") pod \"distinct-w1-1bccbd-kserve-5dfc85555f-tkvg6\" (UID: \"dd4a9d44-c63e-4f67-bcac-a4f98ecf1796\") " pod="e2e-models-e2e-worker-w1-1bccbd/distinct-w1-1bccbd-kserve-5dfc85555f-tkvg6" Nov 03 02:07:39.858380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:39.856231 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd4a9d44-c63e-4f67-bcac-a4f98ecf1796-tls-certs\") pod \"distinct-w1-1bccbd-kserve-5dfc85555f-tkvg6\" (UID: \"dd4a9d44-c63e-4f67-bcac-a4f98ecf1796\") " pod="e2e-models-e2e-worker-w1-1bccbd/distinct-w1-1bccbd-kserve-5dfc85555f-tkvg6" Nov 03 02:07:39.858380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:39.856257 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfg2z\" (UniqueName: \"kubernetes.io/projected/dd4a9d44-c63e-4f67-bcac-a4f98ecf1796-kube-api-access-cfg2z\") pod \"distinct-w1-1bccbd-kserve-5dfc85555f-tkvg6\" (UID: \"dd4a9d44-c63e-4f67-bcac-a4f98ecf1796\") " pod="e2e-models-e2e-worker-w1-1bccbd/distinct-w1-1bccbd-kserve-5dfc85555f-tkvg6" Nov 03 02:07:39.858380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:39.857280 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd4a9d44-c63e-4f67-bcac-a4f98ecf1796-home\") pod \"distinct-w1-1bccbd-kserve-5dfc85555f-tkvg6\" (UID: \"dd4a9d44-c63e-4f67-bcac-a4f98ecf1796\") " pod="e2e-models-e2e-worker-w1-1bccbd/distinct-w1-1bccbd-kserve-5dfc85555f-tkvg6" Nov 03 02:07:39.870381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:39.865740 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd4a9d44-c63e-4f67-bcac-a4f98ecf1796-tmp-dir\") pod \"distinct-w1-1bccbd-kserve-5dfc85555f-tkvg6\" (UID: \"dd4a9d44-c63e-4f67-bcac-a4f98ecf1796\") " pod="e2e-models-e2e-worker-w1-1bccbd/distinct-w1-1bccbd-kserve-5dfc85555f-tkvg6" Nov 03 02:07:39.887274 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:39.875660 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd4a9d44-c63e-4f67-bcac-a4f98ecf1796-tls-certs\") pod \"distinct-w1-1bccbd-kserve-5dfc85555f-tkvg6\" (UID: \"dd4a9d44-c63e-4f67-bcac-a4f98ecf1796\") " pod="e2e-models-e2e-worker-w1-1bccbd/distinct-w1-1bccbd-kserve-5dfc85555f-tkvg6" Nov 03 02:07:39.887274 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:39.878661 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd4a9d44-c63e-4f67-bcac-a4f98ecf1796-model-cache\") pod \"distinct-w1-1bccbd-kserve-5dfc85555f-tkvg6\" (UID: \"dd4a9d44-c63e-4f67-bcac-a4f98ecf1796\") " pod="e2e-models-e2e-worker-w1-1bccbd/distinct-w1-1bccbd-kserve-5dfc85555f-tkvg6" Nov 03 02:07:39.887274 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:39.887200 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd4a9d44-c63e-4f67-bcac-a4f98ecf1796-dshm\") pod \"distinct-w1-1bccbd-kserve-5dfc85555f-tkvg6\" (UID: \"dd4a9d44-c63e-4f67-bcac-a4f98ecf1796\") " pod="e2e-models-e2e-worker-w1-1bccbd/distinct-w1-1bccbd-kserve-5dfc85555f-tkvg6" Nov 03 02:07:39.910113 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:39.910070 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfg2z\" (UniqueName: \"kubernetes.io/projected/dd4a9d44-c63e-4f67-bcac-a4f98ecf1796-kube-api-access-cfg2z\") pod \"distinct-w1-1bccbd-kserve-5dfc85555f-tkvg6\" (UID: \"dd4a9d44-c63e-4f67-bcac-a4f98ecf1796\") " pod="e2e-models-e2e-worker-w1-1bccbd/distinct-w1-1bccbd-kserve-5dfc85555f-tkvg6" Nov 03 02:07:40.006538 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:40.006447 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1bccbd/distinct-w1-1bccbd-kserve-5dfc85555f-tkvg6" Nov 03 02:07:40.409253 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:40.409218 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-1bccbd/distinct-w1-1bccbd-kserve-5dfc85555f-tkvg6"] Nov 03 02:07:40.415200 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:07:40.415171 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddd4a9d44_c63e_4f67_bcac_a4f98ecf1796.slice/crio-b5078dae62b1d5c8cbe557dfb99d179b013b733b96a9a2e2fb3ac7a3bb99cb7b WatchSource:0}: Error finding container b5078dae62b1d5c8cbe557dfb99d179b013b733b96a9a2e2fb3ac7a3bb99cb7b: Status 404 returned error can't find the container with id b5078dae62b1d5c8cbe557dfb99d179b013b733b96a9a2e2fb3ac7a3bb99cb7b Nov 03 02:07:41.126621 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:41.126588 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1bccbd/distinct-w1-1bccbd-kserve-5dfc85555f-tkvg6" event={"ID":"dd4a9d44-c63e-4f67-bcac-a4f98ecf1796","Type":"ContainerStarted","Data":"55a2db3860d5cfe05593d9fc454af1847915448d03c9d747e72ae4acaa9a4c32"} Nov 03 02:07:41.127030 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:41.126632 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1bccbd/distinct-w1-1bccbd-kserve-5dfc85555f-tkvg6" event={"ID":"dd4a9d44-c63e-4f67-bcac-a4f98ecf1796","Type":"ContainerStarted","Data":"b5078dae62b1d5c8cbe557dfb99d179b013b733b96a9a2e2fb3ac7a3bb99cb7b"} Nov 03 02:07:41.233636 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:41.233573 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-1bccbd/distinct-w1-1bccbd-kserve-5dfc85555f-tkvg6" podStartSLOduration=2.233556674 podStartE2EDuration="2.233556674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:07:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:07:41.232461326 +0000 UTC m=+938.491659474" watchObservedRunningTime="2026-11-03 02:07:41.233556674 +0000 UTC m=+938.492754828" Nov 03 02:07:42.007048 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:42.007007 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1bccbd/distinct-w1-1bccbd-kserve-5dfc85555f-tkvg6" Nov 03 02:07:50.007180 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:50.007133 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-1bccbd/distinct-w1-1bccbd-kserve-5dfc85555f-tkvg6" Nov 03 02:07:50.019948 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:50.019834 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-1bccbd/distinct-w1-1bccbd-kserve-5dfc85555f-tkvg6" Nov 03 02:07:50.207729 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:07:50.207685 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1bccbd/distinct-w1-1bccbd-kserve-5dfc85555f-tkvg6" Nov 03 02:07:53.914604 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:07:53.914566 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:08:02.948162 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:08:02.948117 5605 scope.go:122] "RemoveContainer" containerID="8d63b207c57462451c6bbf4621064f18b1cee3c22077b418b70864741b0f236c" Nov 03 02:08:04.697600 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:08:04.697557 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:08:12.297054 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:08:12.297018 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-b-7e89b72d-kserve-7c88c79ddc-p9ttd"] Nov 03 02:08:12.302013 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:08:12.301983 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-b-7e89b72d-kserve-7c88c79ddc-p9ttd" Nov 03 02:08:12.388466 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:08:12.388413 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xmk9\" (UniqueName: \"kubernetes.io/projected/ee3127db-b893-45d9-abc3-582b458aa4c3-kube-api-access-7xmk9\") pod \"e2e-conflict-b-7e89b72d-kserve-7c88c79ddc-p9ttd\" (UID: \"ee3127db-b893-45d9-abc3-582b458aa4c3\") " pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-b-7e89b72d-kserve-7c88c79ddc-p9ttd" Nov 03 02:08:12.388676 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:08:12.388490 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee3127db-b893-45d9-abc3-582b458aa4c3-home\") pod \"e2e-conflict-b-7e89b72d-kserve-7c88c79ddc-p9ttd\" (UID: \"ee3127db-b893-45d9-abc3-582b458aa4c3\") " pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-b-7e89b72d-kserve-7c88c79ddc-p9ttd" Nov 03 02:08:12.388676 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:08:12.388579 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee3127db-b893-45d9-abc3-582b458aa4c3-tmp-dir\") pod \"e2e-conflict-b-7e89b72d-kserve-7c88c79ddc-p9ttd\" (UID: \"ee3127db-b893-45d9-abc3-582b458aa4c3\") " pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-b-7e89b72d-kserve-7c88c79ddc-p9ttd" Nov 03 02:08:12.388676 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:08:12.388615 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee3127db-b893-45d9-abc3-582b458aa4c3-tls-certs\") pod \"e2e-conflict-b-7e89b72d-kserve-7c88c79ddc-p9ttd\" (UID: \"ee3127db-b893-45d9-abc3-582b458aa4c3\") " pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-b-7e89b72d-kserve-7c88c79ddc-p9ttd" Nov 03 02:08:12.388676 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:08:12.388638 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee3127db-b893-45d9-abc3-582b458aa4c3-dshm\") pod \"e2e-conflict-b-7e89b72d-kserve-7c88c79ddc-p9ttd\" (UID: \"ee3127db-b893-45d9-abc3-582b458aa4c3\") " pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-b-7e89b72d-kserve-7c88c79ddc-p9ttd" Nov 03 02:08:12.388908 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:08:12.388722 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee3127db-b893-45d9-abc3-582b458aa4c3-model-cache\") pod \"e2e-conflict-b-7e89b72d-kserve-7c88c79ddc-p9ttd\" (UID: \"ee3127db-b893-45d9-abc3-582b458aa4c3\") " pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-b-7e89b72d-kserve-7c88c79ddc-p9ttd" Nov 03 02:08:12.398688 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:08:12.398660 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f7d9bb\"/\"e2e-conflict-b-7e89b72d-kserve-self-signed-certs\"" Nov 03 02:08:12.400533 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:08:12.400510 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-b-7e89b72d-kserve-7c88c79ddc-p9ttd"] Nov 03 02:08:12.498380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:08:12.496039 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xmk9\" (UniqueName: \"kubernetes.io/projected/ee3127db-b893-45d9-abc3-582b458aa4c3-kube-api-access-7xmk9\") pod \"e2e-conflict-b-7e89b72d-kserve-7c88c79ddc-p9ttd\" (UID: \"ee3127db-b893-45d9-abc3-582b458aa4c3\") " pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-b-7e89b72d-kserve-7c88c79ddc-p9ttd" Nov 03 02:08:12.498380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:08:12.496124 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee3127db-b893-45d9-abc3-582b458aa4c3-home\") pod \"e2e-conflict-b-7e89b72d-kserve-7c88c79ddc-p9ttd\" (UID: \"ee3127db-b893-45d9-abc3-582b458aa4c3\") " pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-b-7e89b72d-kserve-7c88c79ddc-p9ttd" Nov 03 02:08:12.498380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:08:12.496204 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee3127db-b893-45d9-abc3-582b458aa4c3-tmp-dir\") pod \"e2e-conflict-b-7e89b72d-kserve-7c88c79ddc-p9ttd\" (UID: \"ee3127db-b893-45d9-abc3-582b458aa4c3\") " pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-b-7e89b72d-kserve-7c88c79ddc-p9ttd" Nov 03 02:08:12.498380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:08:12.496254 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee3127db-b893-45d9-abc3-582b458aa4c3-tls-certs\") pod \"e2e-conflict-b-7e89b72d-kserve-7c88c79ddc-p9ttd\" (UID: \"ee3127db-b893-45d9-abc3-582b458aa4c3\") " pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-b-7e89b72d-kserve-7c88c79ddc-p9ttd" Nov 03 02:08:12.498380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:08:12.496288 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee3127db-b893-45d9-abc3-582b458aa4c3-dshm\") pod \"e2e-conflict-b-7e89b72d-kserve-7c88c79ddc-p9ttd\" (UID: \"ee3127db-b893-45d9-abc3-582b458aa4c3\") " pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-b-7e89b72d-kserve-7c88c79ddc-p9ttd" Nov 03 02:08:12.498380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:08:12.496324 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee3127db-b893-45d9-abc3-582b458aa4c3-model-cache\") pod \"e2e-conflict-b-7e89b72d-kserve-7c88c79ddc-p9ttd\" (UID: \"ee3127db-b893-45d9-abc3-582b458aa4c3\") " pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-b-7e89b72d-kserve-7c88c79ddc-p9ttd" Nov 03 02:08:12.498380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:08:12.496837 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee3127db-b893-45d9-abc3-582b458aa4c3-model-cache\") pod \"e2e-conflict-b-7e89b72d-kserve-7c88c79ddc-p9ttd\" (UID: \"ee3127db-b893-45d9-abc3-582b458aa4c3\") " pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-b-7e89b72d-kserve-7c88c79ddc-p9ttd" Nov 03 02:08:12.498380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:08:12.497462 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee3127db-b893-45d9-abc3-582b458aa4c3-home\") pod \"e2e-conflict-b-7e89b72d-kserve-7c88c79ddc-p9ttd\" (UID: \"ee3127db-b893-45d9-abc3-582b458aa4c3\") " pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-b-7e89b72d-kserve-7c88c79ddc-p9ttd" Nov 03 02:08:12.509238 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:08:12.509191 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee3127db-b893-45d9-abc3-582b458aa4c3-tmp-dir\") pod \"e2e-conflict-b-7e89b72d-kserve-7c88c79ddc-p9ttd\" (UID: \"ee3127db-b893-45d9-abc3-582b458aa4c3\") " pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-b-7e89b72d-kserve-7c88c79ddc-p9ttd" Nov 03 02:08:12.516849 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:08:12.516807 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee3127db-b893-45d9-abc3-582b458aa4c3-tls-certs\") pod \"e2e-conflict-b-7e89b72d-kserve-7c88c79ddc-p9ttd\" (UID: \"ee3127db-b893-45d9-abc3-582b458aa4c3\") " pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-b-7e89b72d-kserve-7c88c79ddc-p9ttd" Nov 03 02:08:12.521251 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:08:12.521218 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee3127db-b893-45d9-abc3-582b458aa4c3-dshm\") pod \"e2e-conflict-b-7e89b72d-kserve-7c88c79ddc-p9ttd\" (UID: \"ee3127db-b893-45d9-abc3-582b458aa4c3\") " pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-b-7e89b72d-kserve-7c88c79ddc-p9ttd" Nov 03 02:08:12.550190 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:08:12.550159 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xmk9\" (UniqueName: \"kubernetes.io/projected/ee3127db-b893-45d9-abc3-582b458aa4c3-kube-api-access-7xmk9\") pod \"e2e-conflict-b-7e89b72d-kserve-7c88c79ddc-p9ttd\" (UID: \"ee3127db-b893-45d9-abc3-582b458aa4c3\") " pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-b-7e89b72d-kserve-7c88c79ddc-p9ttd" Nov 03 02:08:12.613666 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:08:12.613624 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-b-7e89b72d-kserve-7c88c79ddc-p9ttd" Nov 03 02:08:12.993498 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:08:12.993411 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-b-7e89b72d-kserve-7c88c79ddc-p9ttd"] Nov 03 02:08:13.000486 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:08:13.000446 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee3127db_b893_45d9_abc3_582b458aa4c3.slice/crio-d6369a1093e25d8e5a2e13c3a1537b683baf8104579ad8a56e0c90a0249673f8 WatchSource:0}: Error finding container d6369a1093e25d8e5a2e13c3a1537b683baf8104579ad8a56e0c90a0249673f8: Status 404 returned error can't find the container with id d6369a1093e25d8e5a2e13c3a1537b683baf8104579ad8a56e0c90a0249673f8 Nov 03 02:08:13.335366 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:08:13.335318 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-b-7e89b72d-kserve-7c88c79ddc-p9ttd" event={"ID":"ee3127db-b893-45d9-abc3-582b458aa4c3","Type":"ContainerStarted","Data":"530fc5f1030541c4f19a8e734b0c3e465c233f474363e5cd9abfba31402705d3"} Nov 03 02:08:13.335774 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:08:13.335392 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-b-7e89b72d-kserve-7c88c79ddc-p9ttd" event={"ID":"ee3127db-b893-45d9-abc3-582b458aa4c3","Type":"ContainerStarted","Data":"d6369a1093e25d8e5a2e13c3a1537b683baf8104579ad8a56e0c90a0249673f8"} Nov 03 02:08:13.442429 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:08:13.442373 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-b-7e89b72d-kserve-7c88c79ddc-p9ttd" podStartSLOduration=1.4423435119999999 podStartE2EDuration="1.442343512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:08:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:08:13.440659766 +0000 UTC m=+970.699857920" watchObservedRunningTime="2026-11-03 02:08:13.442343512 +0000 UTC m=+970.701541687" Nov 03 02:08:13.614511 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:08:13.614421 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-b-7e89b72d-kserve-7c88c79ddc-p9ttd" Nov 03 02:08:22.614695 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:08:22.614649 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-b-7e89b72d-kserve-7c88c79ddc-p9ttd" Nov 03 02:08:22.628104 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:08:22.628075 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-b-7e89b72d-kserve-7c88c79ddc-p9ttd" Nov 03 02:08:23.396879 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:08:23.396846 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-b-7e89b72d-kserve-7c88c79ddc-p9ttd" Nov 03 02:08:23.913898 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:08:23.913859 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:08:35.110057 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:08:35.110009 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:08:52.835272 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:08:52.835186 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-c7d758/distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5"] Nov 03 02:08:52.838823 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:08:52.838801 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c7d758/distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5" Nov 03 02:08:52.934476 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:08:52.934438 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c7d758\"/\"distinct-2-w4-c7d758-kserve-self-signed-certs\"" Nov 03 02:08:52.934476 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:08:52.934438 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-c7d758\"/\"openshift-service-ca.crt\"" Nov 03 02:08:52.934734 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:08:52.934540 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-c7d758\"/\"kube-root-ca.crt\"" Nov 03 02:08:52.934734 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:08:52.934662 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c7d758\"/\"default-dockercfg-7k4bc\"" Nov 03 02:08:52.936494 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:08:52.936463 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c7d758/distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5"] Nov 03 02:08:53.022381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:08:53.018021 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fffc3fbb-097a-4a5d-82c8-b1e221d0e710-tls-certs\") pod \"distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5\" (UID: \"fffc3fbb-097a-4a5d-82c8-b1e221d0e710\") " pod="e2e-models-e2e-worker-w4-c7d758/distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5" Nov 03 02:08:53.022381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:08:53.018098 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fffc3fbb-097a-4a5d-82c8-b1e221d0e710-tmp-dir\") pod \"distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5\" (UID: \"fffc3fbb-097a-4a5d-82c8-b1e221d0e710\") " pod="e2e-models-e2e-worker-w4-c7d758/distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5" Nov 03 02:08:53.022381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:08:53.018148 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srb5t\" (UniqueName: \"kubernetes.io/projected/fffc3fbb-097a-4a5d-82c8-b1e221d0e710-kube-api-access-srb5t\") pod \"distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5\" (UID: \"fffc3fbb-097a-4a5d-82c8-b1e221d0e710\") " pod="e2e-models-e2e-worker-w4-c7d758/distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5" Nov 03 02:08:53.022381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:08:53.018177 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fffc3fbb-097a-4a5d-82c8-b1e221d0e710-home\") pod \"distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5\" (UID: \"fffc3fbb-097a-4a5d-82c8-b1e221d0e710\") " pod="e2e-models-e2e-worker-w4-c7d758/distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5" Nov 03 02:08:53.022381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:08:53.018231 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fffc3fbb-097a-4a5d-82c8-b1e221d0e710-model-cache\") pod \"distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5\" (UID: \"fffc3fbb-097a-4a5d-82c8-b1e221d0e710\") " pod="e2e-models-e2e-worker-w4-c7d758/distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5" Nov 03 02:08:53.022381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:08:53.018303 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fffc3fbb-097a-4a5d-82c8-b1e221d0e710-dshm\") pod \"distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5\" (UID: \"fffc3fbb-097a-4a5d-82c8-b1e221d0e710\") " pod="e2e-models-e2e-worker-w4-c7d758/distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5" Nov 03 02:08:53.119455 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:08:53.119323 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fffc3fbb-097a-4a5d-82c8-b1e221d0e710-tmp-dir\") pod \"distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5\" (UID: \"fffc3fbb-097a-4a5d-82c8-b1e221d0e710\") " pod="e2e-models-e2e-worker-w4-c7d758/distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5" Nov 03 02:08:53.119455 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:08:53.119421 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srb5t\" (UniqueName: \"kubernetes.io/projected/fffc3fbb-097a-4a5d-82c8-b1e221d0e710-kube-api-access-srb5t\") pod \"distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5\" (UID: \"fffc3fbb-097a-4a5d-82c8-b1e221d0e710\") " pod="e2e-models-e2e-worker-w4-c7d758/distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5" Nov 03 02:08:53.119710 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:08:53.119458 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fffc3fbb-097a-4a5d-82c8-b1e221d0e710-home\") pod \"distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5\" (UID: \"fffc3fbb-097a-4a5d-82c8-b1e221d0e710\") " pod="e2e-models-e2e-worker-w4-c7d758/distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5" Nov 03 02:08:53.119710 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:08:53.119516 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fffc3fbb-097a-4a5d-82c8-b1e221d0e710-model-cache\") pod \"distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5\" (UID: \"fffc3fbb-097a-4a5d-82c8-b1e221d0e710\") " pod="e2e-models-e2e-worker-w4-c7d758/distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5" Nov 03 02:08:53.119710 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:08:53.119578 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fffc3fbb-097a-4a5d-82c8-b1e221d0e710-dshm\") pod \"distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5\" (UID: \"fffc3fbb-097a-4a5d-82c8-b1e221d0e710\") " pod="e2e-models-e2e-worker-w4-c7d758/distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5" Nov 03 02:08:53.119710 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:08:53.119661 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fffc3fbb-097a-4a5d-82c8-b1e221d0e710-tls-certs\") pod \"distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5\" (UID: \"fffc3fbb-097a-4a5d-82c8-b1e221d0e710\") " pod="e2e-models-e2e-worker-w4-c7d758/distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5" Nov 03 02:08:53.119899 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:08:53.119776 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fffc3fbb-097a-4a5d-82c8-b1e221d0e710-tmp-dir\") pod \"distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5\" (UID: \"fffc3fbb-097a-4a5d-82c8-b1e221d0e710\") " pod="e2e-models-e2e-worker-w4-c7d758/distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5" Nov 03 02:08:53.120143 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:08:53.120116 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fffc3fbb-097a-4a5d-82c8-b1e221d0e710-home\") pod \"distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5\" (UID: \"fffc3fbb-097a-4a5d-82c8-b1e221d0e710\") " pod="e2e-models-e2e-worker-w4-c7d758/distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5" Nov 03 02:08:53.120324 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:08:53.120162 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fffc3fbb-097a-4a5d-82c8-b1e221d0e710-model-cache\") pod \"distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5\" (UID: \"fffc3fbb-097a-4a5d-82c8-b1e221d0e710\") " pod="e2e-models-e2e-worker-w4-c7d758/distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5" Nov 03 02:08:53.122009 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:08:53.121983 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fffc3fbb-097a-4a5d-82c8-b1e221d0e710-dshm\") pod \"distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5\" (UID: \"fffc3fbb-097a-4a5d-82c8-b1e221d0e710\") " pod="e2e-models-e2e-worker-w4-c7d758/distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5" Nov 03 02:08:53.122519 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:08:53.122496 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fffc3fbb-097a-4a5d-82c8-b1e221d0e710-tls-certs\") pod \"distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5\" (UID: \"fffc3fbb-097a-4a5d-82c8-b1e221d0e710\") " pod="e2e-models-e2e-worker-w4-c7d758/distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5" Nov 03 02:08:53.173173 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:08:53.173130 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srb5t\" (UniqueName: \"kubernetes.io/projected/fffc3fbb-097a-4a5d-82c8-b1e221d0e710-kube-api-access-srb5t\") pod \"distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5\" (UID: \"fffc3fbb-097a-4a5d-82c8-b1e221d0e710\") " pod="e2e-models-e2e-worker-w4-c7d758/distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5" Nov 03 02:08:53.451975 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:08:53.451873 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c7d758/distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5" Nov 03 02:08:53.823979 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:08:53.823938 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c7d758/distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5"] Nov 03 02:08:53.830211 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:08:53.830181 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfffc3fbb_097a_4a5d_82c8_b1e221d0e710.slice/crio-2b818c481602eb05f3a43c56b5b6249b70f08800abaca27f3c6a31530756249d WatchSource:0}: Error finding container 2b818c481602eb05f3a43c56b5b6249b70f08800abaca27f3c6a31530756249d: Status 404 returned error can't find the container with id 2b818c481602eb05f3a43c56b5b6249b70f08800abaca27f3c6a31530756249d Nov 03 02:08:53.917161 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:08:53.917120 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:08:54.554036 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:08:54.553991 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c7d758/distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5" event={"ID":"fffc3fbb-097a-4a5d-82c8-b1e221d0e710","Type":"ContainerStarted","Data":"ede419b3651cd05f584bfbde949fdae0a9bf930d245e968f6a29a25f8e9e0bb9"} Nov 03 02:08:54.554036 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:08:54.554043 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c7d758/distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5" event={"ID":"fffc3fbb-097a-4a5d-82c8-b1e221d0e710","Type":"ContainerStarted","Data":"2b818c481602eb05f3a43c56b5b6249b70f08800abaca27f3c6a31530756249d"} Nov 03 02:08:54.661679 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:08:54.661622 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-c7d758/distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5" podStartSLOduration=2.661608395 podStartE2EDuration="2.661608395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:08:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:08:54.660138607 +0000 UTC m=+1011.919336762" watchObservedRunningTime="2026-11-03 02:08:54.661608395 +0000 UTC m=+1011.920806600" Nov 03 02:08:55.452322 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:08:55.452276 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c7d758/distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5" Nov 03 02:09:02.969341 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:02.969301 5605 scope.go:122] "RemoveContainer" containerID="995fc17348a6f886bf5947bd7d66741767bf40aa7de54355ae6521d6b437c8cd" Nov 03 02:09:02.978089 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:02.978062 5605 scope.go:122] "RemoveContainer" containerID="f58061b775986f9ae6ff279a95a1f849fcc81e40c5682ae17d7c0059b6c3e12e" Nov 03 02:09:03.452375 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:03.452316 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-c7d758/distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5" Nov 03 02:09:03.465021 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:03.464988 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-c7d758/distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5" Nov 03 02:09:03.613491 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:03.613459 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c7d758/distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5" Nov 03 02:09:04.658453 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:04.658418 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:09:10.156031 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:10.155993 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-1bccbd/distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg"] Nov 03 02:09:10.161370 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:10.161330 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1bccbd/distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg" Nov 03 02:09:10.223169 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:10.223129 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdcmr\" (UniqueName: \"kubernetes.io/projected/5a71cdee-425e-4787-9217-5fb6b97e232b-kube-api-access-gdcmr\") pod \"distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg\" (UID: \"5a71cdee-425e-4787-9217-5fb6b97e232b\") " pod="e2e-models-e2e-worker-w1-1bccbd/distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg" Nov 03 02:09:10.223394 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:10.223182 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a71cdee-425e-4787-9217-5fb6b97e232b-home\") pod \"distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg\" (UID: \"5a71cdee-425e-4787-9217-5fb6b97e232b\") " pod="e2e-models-e2e-worker-w1-1bccbd/distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg" Nov 03 02:09:10.223394 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:10.223273 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a71cdee-425e-4787-9217-5fb6b97e232b-tls-certs\") pod \"distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg\" (UID: \"5a71cdee-425e-4787-9217-5fb6b97e232b\") " pod="e2e-models-e2e-worker-w1-1bccbd/distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg" Nov 03 02:09:10.223502 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:10.223466 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a71cdee-425e-4787-9217-5fb6b97e232b-model-cache\") pod \"distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg\" (UID: \"5a71cdee-425e-4787-9217-5fb6b97e232b\") " pod="e2e-models-e2e-worker-w1-1bccbd/distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg" Nov 03 02:09:10.223550 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:10.223509 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a71cdee-425e-4787-9217-5fb6b97e232b-tmp-dir\") pod \"distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg\" (UID: \"5a71cdee-425e-4787-9217-5fb6b97e232b\") " pod="e2e-models-e2e-worker-w1-1bccbd/distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg" Nov 03 02:09:10.223652 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:10.223634 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a71cdee-425e-4787-9217-5fb6b97e232b-dshm\") pod \"distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg\" (UID: \"5a71cdee-425e-4787-9217-5fb6b97e232b\") " pod="e2e-models-e2e-worker-w1-1bccbd/distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg" Nov 03 02:09:10.258105 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:10.258061 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-1bccbd\"/\"distinct-2-w1-1bccbd-kserve-self-signed-certs\"" Nov 03 02:09:10.260004 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:10.259975 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-1bccbd/distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg"] Nov 03 02:09:10.334634 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:10.329423 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a71cdee-425e-4787-9217-5fb6b97e232b-dshm\") pod \"distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg\" (UID: \"5a71cdee-425e-4787-9217-5fb6b97e232b\") " pod="e2e-models-e2e-worker-w1-1bccbd/distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg" Nov 03 02:09:10.334634 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:10.333966 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdcmr\" (UniqueName: \"kubernetes.io/projected/5a71cdee-425e-4787-9217-5fb6b97e232b-kube-api-access-gdcmr\") pod \"distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg\" (UID: \"5a71cdee-425e-4787-9217-5fb6b97e232b\") " pod="e2e-models-e2e-worker-w1-1bccbd/distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg" Nov 03 02:09:10.334634 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:10.334049 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a71cdee-425e-4787-9217-5fb6b97e232b-home\") pod \"distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg\" (UID: \"5a71cdee-425e-4787-9217-5fb6b97e232b\") " pod="e2e-models-e2e-worker-w1-1bccbd/distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg" Nov 03 02:09:10.334634 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:10.334089 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a71cdee-425e-4787-9217-5fb6b97e232b-tls-certs\") pod \"distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg\" (UID: \"5a71cdee-425e-4787-9217-5fb6b97e232b\") " pod="e2e-models-e2e-worker-w1-1bccbd/distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg" Nov 03 02:09:10.334634 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:10.334214 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a71cdee-425e-4787-9217-5fb6b97e232b-model-cache\") pod \"distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg\" (UID: \"5a71cdee-425e-4787-9217-5fb6b97e232b\") " pod="e2e-models-e2e-worker-w1-1bccbd/distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg" Nov 03 02:09:10.334634 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:10.334246 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a71cdee-425e-4787-9217-5fb6b97e232b-tmp-dir\") pod \"distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg\" (UID: \"5a71cdee-425e-4787-9217-5fb6b97e232b\") " pod="e2e-models-e2e-worker-w1-1bccbd/distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg" Nov 03 02:09:10.335271 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:10.334652 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a71cdee-425e-4787-9217-5fb6b97e232b-tmp-dir\") pod \"distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg\" (UID: \"5a71cdee-425e-4787-9217-5fb6b97e232b\") " pod="e2e-models-e2e-worker-w1-1bccbd/distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg" Nov 03 02:09:10.335271 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:10.334990 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a71cdee-425e-4787-9217-5fb6b97e232b-home\") pod \"distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg\" (UID: \"5a71cdee-425e-4787-9217-5fb6b97e232b\") " pod="e2e-models-e2e-worker-w1-1bccbd/distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg" Nov 03 02:09:10.335428 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:10.335291 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a71cdee-425e-4787-9217-5fb6b97e232b-model-cache\") pod \"distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg\" (UID: \"5a71cdee-425e-4787-9217-5fb6b97e232b\") " pod="e2e-models-e2e-worker-w1-1bccbd/distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg" Nov 03 02:09:10.336685 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:10.336659 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a71cdee-425e-4787-9217-5fb6b97e232b-dshm\") pod \"distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg\" (UID: \"5a71cdee-425e-4787-9217-5fb6b97e232b\") " pod="e2e-models-e2e-worker-w1-1bccbd/distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg" Nov 03 02:09:10.344900 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:10.344860 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a71cdee-425e-4787-9217-5fb6b97e232b-tls-certs\") pod \"distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg\" (UID: \"5a71cdee-425e-4787-9217-5fb6b97e232b\") " pod="e2e-models-e2e-worker-w1-1bccbd/distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg" Nov 03 02:09:10.388731 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:10.388700 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdcmr\" (UniqueName: \"kubernetes.io/projected/5a71cdee-425e-4787-9217-5fb6b97e232b-kube-api-access-gdcmr\") pod \"distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg\" (UID: \"5a71cdee-425e-4787-9217-5fb6b97e232b\") " pod="e2e-models-e2e-worker-w1-1bccbd/distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg" Nov 03 02:09:10.474721 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:10.474613 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1bccbd/distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg" Nov 03 02:09:10.874243 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:10.874205 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-1bccbd/distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg"] Nov 03 02:09:10.878815 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:09:10.878778 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a71cdee_425e_4787_9217_5fb6b97e232b.slice/crio-8c7389012f8a44ee77bb2d25aa0f46430e56462c717d1e476f0697bbcdb2f2c0 WatchSource:0}: Error finding container 8c7389012f8a44ee77bb2d25aa0f46430e56462c717d1e476f0697bbcdb2f2c0: Status 404 returned error can't find the container with id 8c7389012f8a44ee77bb2d25aa0f46430e56462c717d1e476f0697bbcdb2f2c0 Nov 03 02:09:11.664582 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:11.664541 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1bccbd/distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg" event={"ID":"5a71cdee-425e-4787-9217-5fb6b97e232b","Type":"ContainerStarted","Data":"f0d955ba3038db02d8d86da70f8cc3f3c58807b67f406e3e41f5f6cde9f182fb"} Nov 03 02:09:11.664582 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:11.664590 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1bccbd/distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg" event={"ID":"5a71cdee-425e-4787-9217-5fb6b97e232b","Type":"ContainerStarted","Data":"8c7389012f8a44ee77bb2d25aa0f46430e56462c717d1e476f0697bbcdb2f2c0"} Nov 03 02:09:11.771186 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:11.771124 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-1bccbd/distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg" podStartSLOduration=1.771108431 podStartE2EDuration="1.771108431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:09:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:09:11.769900524 +0000 UTC m=+1029.029098678" watchObservedRunningTime="2026-11-03 02:09:11.771108431 +0000 UTC m=+1029.030306586" Nov 03 02:09:12.475427 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:12.475353 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1bccbd/distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg" Nov 03 02:09:20.475306 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:20.475266 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-1bccbd/distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg" Nov 03 02:09:20.487963 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:20.487937 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-1bccbd/distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg" Nov 03 02:09:20.720271 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:20.720243 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1bccbd/distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg" Nov 03 02:09:23.911322 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:09:23.911283 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:09:24.820418 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:24.820377 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:09:31.258814 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:31.258775 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s"] Nov 03 02:09:31.262662 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:31.262636 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s" Nov 03 02:09:31.287278 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:31.287246 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8c9203e-5102-452b-bbc7-af1b50ea3ec7-workload-certs\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s\" (UID: \"b8c9203e-5102-452b-bbc7-af1b50ea3ec7\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s" Nov 03 02:09:31.287425 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:31.287292 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8c9203e-5102-452b-bbc7-af1b50ea3ec7-istio-token\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s\" (UID: \"b8c9203e-5102-452b-bbc7-af1b50ea3ec7\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s" Nov 03 02:09:31.287425 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:31.287410 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxhh7\" (UniqueName: \"kubernetes.io/projected/b8c9203e-5102-452b-bbc7-af1b50ea3ec7-kube-api-access-kxhh7\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s\" (UID: \"b8c9203e-5102-452b-bbc7-af1b50ea3ec7\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s" Nov 03 02:09:31.287522 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:31.287454 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8c9203e-5102-452b-bbc7-af1b50ea3ec7-istiod-ca-cert\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s\" (UID: \"b8c9203e-5102-452b-bbc7-af1b50ea3ec7\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s" Nov 03 02:09:31.287522 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:31.287503 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8c9203e-5102-452b-bbc7-af1b50ea3ec7-istio-envoy\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s\" (UID: \"b8c9203e-5102-452b-bbc7-af1b50ea3ec7\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s" Nov 03 02:09:31.287615 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:31.287596 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8c9203e-5102-452b-bbc7-af1b50ea3ec7-workload-socket\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s\" (UID: \"b8c9203e-5102-452b-bbc7-af1b50ea3ec7\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s" Nov 03 02:09:31.287659 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:31.287644 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8c9203e-5102-452b-bbc7-af1b50ea3ec7-istio-podinfo\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s\" (UID: \"b8c9203e-5102-452b-bbc7-af1b50ea3ec7\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s" Nov 03 02:09:31.287722 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:31.287708 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8c9203e-5102-452b-bbc7-af1b50ea3ec7-istio-data\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s\" (UID: \"b8c9203e-5102-452b-bbc7-af1b50ea3ec7\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s" Nov 03 02:09:31.287761 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:31.287740 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8c9203e-5102-452b-bbc7-af1b50ea3ec7-credential-socket\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s\" (UID: \"b8c9203e-5102-452b-bbc7-af1b50ea3ec7\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s" Nov 03 02:09:31.309587 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:31.306584 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28"] Nov 03 02:09:31.312489 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:31.312461 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28" Nov 03 02:09:31.356662 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:31.356625 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s"] Nov 03 02:09:31.389188 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:31.389157 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c586c92-59da-421e-8224-45b9a833c2d4-workload-certs\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28\" (UID: \"5c586c92-59da-421e-8224-45b9a833c2d4\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28" Nov 03 02:09:31.389389 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:31.389197 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxhh7\" (UniqueName: \"kubernetes.io/projected/b8c9203e-5102-452b-bbc7-af1b50ea3ec7-kube-api-access-kxhh7\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s\" (UID: \"b8c9203e-5102-452b-bbc7-af1b50ea3ec7\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s" Nov 03 02:09:31.389389 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:31.389221 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8c9203e-5102-452b-bbc7-af1b50ea3ec7-istiod-ca-cert\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s\" (UID: \"b8c9203e-5102-452b-bbc7-af1b50ea3ec7\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s" Nov 03 02:09:31.389389 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:31.389272 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c586c92-59da-421e-8224-45b9a833c2d4-istiod-ca-cert\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28\" (UID: \"5c586c92-59da-421e-8224-45b9a833c2d4\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28" Nov 03 02:09:31.389578 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:31.389417 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c586c92-59da-421e-8224-45b9a833c2d4-istio-data\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28\" (UID: \"5c586c92-59da-421e-8224-45b9a833c2d4\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28" Nov 03 02:09:31.389578 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:31.389471 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8c9203e-5102-452b-bbc7-af1b50ea3ec7-istio-envoy\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s\" (UID: \"b8c9203e-5102-452b-bbc7-af1b50ea3ec7\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s" Nov 03 02:09:31.389578 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:31.389531 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8c9203e-5102-452b-bbc7-af1b50ea3ec7-workload-socket\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s\" (UID: \"b8c9203e-5102-452b-bbc7-af1b50ea3ec7\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s" Nov 03 02:09:31.389578 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:31.389561 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8c9203e-5102-452b-bbc7-af1b50ea3ec7-istio-podinfo\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s\" (UID: \"b8c9203e-5102-452b-bbc7-af1b50ea3ec7\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s" Nov 03 02:09:31.389789 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:31.389615 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8c9203e-5102-452b-bbc7-af1b50ea3ec7-istio-data\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s\" (UID: \"b8c9203e-5102-452b-bbc7-af1b50ea3ec7\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s" Nov 03 02:09:31.389789 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:31.389642 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4w5n\" (UniqueName: \"kubernetes.io/projected/5c586c92-59da-421e-8224-45b9a833c2d4-kube-api-access-r4w5n\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28\" (UID: \"5c586c92-59da-421e-8224-45b9a833c2d4\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28" Nov 03 02:09:31.389789 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:31.389684 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8c9203e-5102-452b-bbc7-af1b50ea3ec7-credential-socket\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s\" (UID: \"b8c9203e-5102-452b-bbc7-af1b50ea3ec7\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s" Nov 03 02:09:31.389789 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:31.389709 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c586c92-59da-421e-8224-45b9a833c2d4-istio-podinfo\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28\" (UID: \"5c586c92-59da-421e-8224-45b9a833c2d4\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28" Nov 03 02:09:31.389991 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:31.389907 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8c9203e-5102-452b-bbc7-af1b50ea3ec7-workload-socket\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s\" (UID: \"b8c9203e-5102-452b-bbc7-af1b50ea3ec7\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s" Nov 03 02:09:31.390050 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:31.389992 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8c9203e-5102-452b-bbc7-af1b50ea3ec7-credential-socket\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s\" (UID: \"b8c9203e-5102-452b-bbc7-af1b50ea3ec7\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s" Nov 03 02:09:31.390050 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:31.390001 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c586c92-59da-421e-8224-45b9a833c2d4-istio-envoy\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28\" (UID: \"5c586c92-59da-421e-8224-45b9a833c2d4\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28" Nov 03 02:09:31.390133 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:31.390077 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8c9203e-5102-452b-bbc7-af1b50ea3ec7-istio-data\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s\" (UID: \"b8c9203e-5102-452b-bbc7-af1b50ea3ec7\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s" Nov 03 02:09:31.390133 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:31.390087 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8c9203e-5102-452b-bbc7-af1b50ea3ec7-istiod-ca-cert\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s\" (UID: \"b8c9203e-5102-452b-bbc7-af1b50ea3ec7\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s" Nov 03 02:09:31.390133 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:31.390090 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c586c92-59da-421e-8224-45b9a833c2d4-workload-socket\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28\" (UID: \"5c586c92-59da-421e-8224-45b9a833c2d4\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28" Nov 03 02:09:31.390278 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:31.390181 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8c9203e-5102-452b-bbc7-af1b50ea3ec7-workload-certs\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s\" (UID: \"b8c9203e-5102-452b-bbc7-af1b50ea3ec7\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s" Nov 03 02:09:31.390278 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:31.390231 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c586c92-59da-421e-8224-45b9a833c2d4-istio-token\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28\" (UID: \"5c586c92-59da-421e-8224-45b9a833c2d4\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28" Nov 03 02:09:31.390401 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:31.390298 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8c9203e-5102-452b-bbc7-af1b50ea3ec7-istio-token\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s\" (UID: \"b8c9203e-5102-452b-bbc7-af1b50ea3ec7\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s" Nov 03 02:09:31.390401 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:31.390331 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c586c92-59da-421e-8224-45b9a833c2d4-credential-socket\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28\" (UID: \"5c586c92-59da-421e-8224-45b9a833c2d4\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28" Nov 03 02:09:31.390550 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:31.390519 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8c9203e-5102-452b-bbc7-af1b50ea3ec7-workload-certs\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s\" (UID: \"b8c9203e-5102-452b-bbc7-af1b50ea3ec7\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s" Nov 03 02:09:31.392011 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:31.391982 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8c9203e-5102-452b-bbc7-af1b50ea3ec7-istio-envoy\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s\" (UID: \"b8c9203e-5102-452b-bbc7-af1b50ea3ec7\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s" Nov 03 02:09:31.392153 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:31.392131 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8c9203e-5102-452b-bbc7-af1b50ea3ec7-istio-podinfo\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s\" (UID: \"b8c9203e-5102-452b-bbc7-af1b50ea3ec7\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s" Nov 03 02:09:31.421452 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:31.421422 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28"] Nov 03 02:09:31.459310 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:31.459271 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8c9203e-5102-452b-bbc7-af1b50ea3ec7-istio-token\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s\" (UID: \"b8c9203e-5102-452b-bbc7-af1b50ea3ec7\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s" Nov 03 02:09:31.459482 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:31.459426 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxhh7\" (UniqueName: \"kubernetes.io/projected/b8c9203e-5102-452b-bbc7-af1b50ea3ec7-kube-api-access-kxhh7\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s\" (UID: \"b8c9203e-5102-452b-bbc7-af1b50ea3ec7\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s" Nov 03 02:09:31.491708 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:31.491674 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c586c92-59da-421e-8224-45b9a833c2d4-workload-certs\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28\" (UID: \"5c586c92-59da-421e-8224-45b9a833c2d4\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28" Nov 03 02:09:31.491894 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:31.491727 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c586c92-59da-421e-8224-45b9a833c2d4-istiod-ca-cert\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28\" (UID: \"5c586c92-59da-421e-8224-45b9a833c2d4\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28" Nov 03 02:09:31.491894 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:31.491768 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c586c92-59da-421e-8224-45b9a833c2d4-istio-data\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28\" (UID: \"5c586c92-59da-421e-8224-45b9a833c2d4\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28" Nov 03 02:09:31.491894 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:31.491819 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4w5n\" (UniqueName: \"kubernetes.io/projected/5c586c92-59da-421e-8224-45b9a833c2d4-kube-api-access-r4w5n\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28\" (UID: \"5c586c92-59da-421e-8224-45b9a833c2d4\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28" Nov 03 02:09:31.491894 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:31.491862 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c586c92-59da-421e-8224-45b9a833c2d4-istio-podinfo\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28\" (UID: \"5c586c92-59da-421e-8224-45b9a833c2d4\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28" Nov 03 02:09:31.492117 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:31.491898 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c586c92-59da-421e-8224-45b9a833c2d4-istio-envoy\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28\" (UID: \"5c586c92-59da-421e-8224-45b9a833c2d4\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28" Nov 03 02:09:31.492117 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:31.491944 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c586c92-59da-421e-8224-45b9a833c2d4-workload-socket\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28\" (UID: \"5c586c92-59da-421e-8224-45b9a833c2d4\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28" Nov 03 02:09:31.492117 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:31.491999 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c586c92-59da-421e-8224-45b9a833c2d4-istio-token\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28\" (UID: \"5c586c92-59da-421e-8224-45b9a833c2d4\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28" Nov 03 02:09:31.492117 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:31.492049 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c586c92-59da-421e-8224-45b9a833c2d4-credential-socket\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28\" (UID: \"5c586c92-59da-421e-8224-45b9a833c2d4\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28" Nov 03 02:09:31.492338 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:31.492157 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c586c92-59da-421e-8224-45b9a833c2d4-workload-certs\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28\" (UID: \"5c586c92-59da-421e-8224-45b9a833c2d4\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28" Nov 03 02:09:31.492338 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:31.492228 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c586c92-59da-421e-8224-45b9a833c2d4-istio-data\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28\" (UID: \"5c586c92-59da-421e-8224-45b9a833c2d4\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28" Nov 03 02:09:31.492496 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:31.492455 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c586c92-59da-421e-8224-45b9a833c2d4-workload-socket\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28\" (UID: \"5c586c92-59da-421e-8224-45b9a833c2d4\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28" Nov 03 02:09:31.492554 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:31.492526 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c586c92-59da-421e-8224-45b9a833c2d4-credential-socket\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28\" (UID: \"5c586c92-59da-421e-8224-45b9a833c2d4\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28" Nov 03 02:09:31.492604 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:31.492559 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c586c92-59da-421e-8224-45b9a833c2d4-istiod-ca-cert\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28\" (UID: \"5c586c92-59da-421e-8224-45b9a833c2d4\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28" Nov 03 02:09:31.494830 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:31.494802 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c586c92-59da-421e-8224-45b9a833c2d4-istio-envoy\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28\" (UID: \"5c586c92-59da-421e-8224-45b9a833c2d4\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28" Nov 03 02:09:31.495044 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:31.495017 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c586c92-59da-421e-8224-45b9a833c2d4-istio-podinfo\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28\" (UID: \"5c586c92-59da-421e-8224-45b9a833c2d4\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28" Nov 03 02:09:31.546166 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:31.546092 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4w5n\" (UniqueName: \"kubernetes.io/projected/5c586c92-59da-421e-8224-45b9a833c2d4-kube-api-access-r4w5n\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28\" (UID: \"5c586c92-59da-421e-8224-45b9a833c2d4\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28" Nov 03 02:09:31.546166 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:31.546146 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c586c92-59da-421e-8224-45b9a833c2d4-istio-token\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28\" (UID: \"5c586c92-59da-421e-8224-45b9a833c2d4\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28" Nov 03 02:09:31.575501 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:31.575470 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s" Nov 03 02:09:31.625711 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:31.625667 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28" Nov 03 02:09:31.995867 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:31.995834 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s"] Nov 03 02:09:32.043382 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:09:32.043339 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8c9203e_5102_452b_bbc7_af1b50ea3ec7.slice/crio-30aa9a73a05b703100900422a7eb19e61e8cc69ed499514918555711edd547fe WatchSource:0}: Error finding container 30aa9a73a05b703100900422a7eb19e61e8cc69ed499514918555711edd547fe: Status 404 returned error can't find the container with id 30aa9a73a05b703100900422a7eb19e61e8cc69ed499514918555711edd547fe Nov 03 02:09:32.045950 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:32.045908 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:09:32.046044 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:32.045991 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:09:32.046044 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:32.046022 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:09:32.047810 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:32.047783 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28"] Nov 03 02:09:32.055227 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:09:32.055200 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c586c92_59da_421e_8224_45b9a833c2d4.slice/crio-87d7f9ea389ead9c2d5dd6a6443037905478bf7ec2ac0cbcf2a730e387282e00 WatchSource:0}: Error finding container 87d7f9ea389ead9c2d5dd6a6443037905478bf7ec2ac0cbcf2a730e387282e00: Status 404 returned error can't find the container with id 87d7f9ea389ead9c2d5dd6a6443037905478bf7ec2ac0cbcf2a730e387282e00 Nov 03 02:09:32.057614 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:32.057579 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:09:32.057712 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:32.057659 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:09:32.057712 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:32.057699 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:09:32.774593 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:32.774553 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28" event={"ID":"5c586c92-59da-421e-8224-45b9a833c2d4","Type":"ContainerStarted","Data":"788c0b9b519150225b16e0b4159f319099b242ae05216b472cf01f8bef2be731"} Nov 03 02:09:32.775105 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:32.774622 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28" event={"ID":"5c586c92-59da-421e-8224-45b9a833c2d4","Type":"ContainerStarted","Data":"87d7f9ea389ead9c2d5dd6a6443037905478bf7ec2ac0cbcf2a730e387282e00"} Nov 03 02:09:32.776527 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:32.776500 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s" event={"ID":"b8c9203e-5102-452b-bbc7-af1b50ea3ec7","Type":"ContainerStarted","Data":"f6a7b2b01926b8282b17862cf0fcc588a6a8ca0086adfb62f8928d2a06255216"} Nov 03 02:09:32.776670 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:32.776544 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s" event={"ID":"b8c9203e-5102-452b-bbc7-af1b50ea3ec7","Type":"ContainerStarted","Data":"30aa9a73a05b703100900422a7eb19e61e8cc69ed499514918555711edd547fe"} Nov 03 02:09:32.920525 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:32.920471 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28" podStartSLOduration=1.920454446 podStartE2EDuration="1.920454446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:09:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:09:32.899845925 +0000 UTC m=+1050.159044079" watchObservedRunningTime="2026-11-03 02:09:32.920454446 +0000 UTC m=+1050.179652657" Nov 03 02:09:33.578789 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:33.578739 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s" Nov 03 02:09:33.585053 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:33.584640 5605 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 03 02:09:33.585053 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:33.584704 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s" podUID="b8c9203e-5102-452b-bbc7-af1b50ea3ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 03 02:09:33.626586 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:33.626552 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28" Nov 03 02:09:33.627845 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:33.627815 5605 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 03 02:09:33.627966 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:33.627875 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28" podUID="5c586c92-59da-421e-8224-45b9a833c2d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 03 02:09:34.582810 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:34.582762 5605 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 03 02:09:34.583329 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:34.582833 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s" podUID="b8c9203e-5102-452b-bbc7-af1b50ea3ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 03 02:09:34.626240 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:34.626202 5605 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 03 02:09:34.626446 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:34.626261 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28" podUID="5c586c92-59da-421e-8224-45b9a833c2d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 03 02:09:35.579382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:35.578918 5605 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 03 02:09:35.579382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:35.579002 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s" podUID="b8c9203e-5102-452b-bbc7-af1b50ea3ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 03 02:09:35.626388 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:35.626322 5605 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 03 02:09:35.626810 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:35.626405 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28" podUID="5c586c92-59da-421e-8224-45b9a833c2d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 03 02:09:36.580383 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:36.576580 5605 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 03 02:09:36.580383 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:36.576644 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s" podUID="b8c9203e-5102-452b-bbc7-af1b50ea3ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 03 02:09:36.626419 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:36.626352 5605 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 03 02:09:36.626805 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:36.626436 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28" podUID="5c586c92-59da-421e-8224-45b9a833c2d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 03 02:09:37.577385 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:37.576774 5605 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 03 02:09:37.577385 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:37.576837 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s" podUID="b8c9203e-5102-452b-bbc7-af1b50ea3ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 03 02:09:37.626083 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:37.626044 5605 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 03 02:09:37.626323 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:37.626295 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28" podUID="5c586c92-59da-421e-8224-45b9a833c2d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 03 02:09:38.576074 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:38.576039 5605 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 03 02:09:38.576492 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:38.576096 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s" podUID="b8c9203e-5102-452b-bbc7-af1b50ea3ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 03 02:09:38.627016 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:38.626976 5605 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 03 02:09:38.627191 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:38.627028 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28" podUID="5c586c92-59da-421e-8224-45b9a833c2d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 03 02:09:39.650382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:39.649548 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s" podStartSLOduration=8.64951182 podStartE2EDuration="8.64951182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:09:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:09:33.040311319 +0000 UTC m=+1050.299509473" watchObservedRunningTime="2026-11-03 02:09:39.64951182 +0000 UTC m=+1056.908709977" Nov 03 02:09:39.650853 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:39.650610 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-b-7e89b72d-kserve-7c88c79ddc-p9ttd"] Nov 03 02:09:39.921279 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:39.921193 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28" Nov 03 02:09:39.923736 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:39.923713 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28" Nov 03 02:09:39.926082 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:39.924893 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28" Nov 03 02:09:39.989922 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:39.989879 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s" Nov 03 02:09:39.991448 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:39.991422 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s" Nov 03 02:09:39.992436 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:39.992412 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s" Nov 03 02:09:47.515378 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:47.514389 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c"] Nov 03 02:09:47.523765 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:47.523733 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c" Nov 03 02:09:47.602152 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:47.602116 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c"] Nov 03 02:09:47.623726 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:47.623690 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69d57c7a-8495-4d67-9547-dfe082939941-workload-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c\" (UID: \"69d57c7a-8495-4d67-9547-dfe082939941\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c" Nov 03 02:09:47.623907 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:47.623757 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69d57c7a-8495-4d67-9547-dfe082939941-workload-certs\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c\" (UID: \"69d57c7a-8495-4d67-9547-dfe082939941\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c" Nov 03 02:09:47.623907 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:47.623855 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7tcj\" (UniqueName: \"kubernetes.io/projected/69d57c7a-8495-4d67-9547-dfe082939941-kube-api-access-q7tcj\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c\" (UID: \"69d57c7a-8495-4d67-9547-dfe082939941\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c" Nov 03 02:09:47.624025 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:47.623917 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69d57c7a-8495-4d67-9547-dfe082939941-istio-data\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c\" (UID: \"69d57c7a-8495-4d67-9547-dfe082939941\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c" Nov 03 02:09:47.624025 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:47.623955 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69d57c7a-8495-4d67-9547-dfe082939941-istiod-ca-cert\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c\" (UID: \"69d57c7a-8495-4d67-9547-dfe082939941\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c" Nov 03 02:09:47.624131 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:47.624057 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69d57c7a-8495-4d67-9547-dfe082939941-istio-envoy\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c\" (UID: \"69d57c7a-8495-4d67-9547-dfe082939941\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c" Nov 03 02:09:47.624131 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:47.624089 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69d57c7a-8495-4d67-9547-dfe082939941-credential-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c\" (UID: \"69d57c7a-8495-4d67-9547-dfe082939941\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c" Nov 03 02:09:47.624237 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:47.624206 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69d57c7a-8495-4d67-9547-dfe082939941-istio-token\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c\" (UID: \"69d57c7a-8495-4d67-9547-dfe082939941\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c" Nov 03 02:09:47.624288 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:47.624250 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69d57c7a-8495-4d67-9547-dfe082939941-istio-podinfo\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c\" (UID: \"69d57c7a-8495-4d67-9547-dfe082939941\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c" Nov 03 02:09:47.725014 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:47.724977 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69d57c7a-8495-4d67-9547-dfe082939941-workload-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c\" (UID: \"69d57c7a-8495-4d67-9547-dfe082939941\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c" Nov 03 02:09:47.725206 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:47.725120 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69d57c7a-8495-4d67-9547-dfe082939941-workload-certs\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c\" (UID: \"69d57c7a-8495-4d67-9547-dfe082939941\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c" Nov 03 02:09:47.725206 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:47.725154 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7tcj\" (UniqueName: \"kubernetes.io/projected/69d57c7a-8495-4d67-9547-dfe082939941-kube-api-access-q7tcj\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c\" (UID: \"69d57c7a-8495-4d67-9547-dfe082939941\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c" Nov 03 02:09:47.725206 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:47.725197 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69d57c7a-8495-4d67-9547-dfe082939941-istio-data\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c\" (UID: \"69d57c7a-8495-4d67-9547-dfe082939941\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c" Nov 03 02:09:47.725400 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:47.725221 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69d57c7a-8495-4d67-9547-dfe082939941-istiod-ca-cert\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c\" (UID: \"69d57c7a-8495-4d67-9547-dfe082939941\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c" Nov 03 02:09:47.725400 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:47.725258 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69d57c7a-8495-4d67-9547-dfe082939941-istio-envoy\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c\" (UID: \"69d57c7a-8495-4d67-9547-dfe082939941\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c" Nov 03 02:09:47.725400 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:47.725284 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69d57c7a-8495-4d67-9547-dfe082939941-credential-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c\" (UID: \"69d57c7a-8495-4d67-9547-dfe082939941\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c" Nov 03 02:09:47.725400 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:47.725370 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69d57c7a-8495-4d67-9547-dfe082939941-istio-token\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c\" (UID: \"69d57c7a-8495-4d67-9547-dfe082939941\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c" Nov 03 02:09:47.725603 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:47.725415 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69d57c7a-8495-4d67-9547-dfe082939941-istio-podinfo\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c\" (UID: \"69d57c7a-8495-4d67-9547-dfe082939941\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c" Nov 03 02:09:47.725603 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:47.725504 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69d57c7a-8495-4d67-9547-dfe082939941-workload-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c\" (UID: \"69d57c7a-8495-4d67-9547-dfe082939941\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c" Nov 03 02:09:47.725820 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:47.725797 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69d57c7a-8495-4d67-9547-dfe082939941-workload-certs\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c\" (UID: \"69d57c7a-8495-4d67-9547-dfe082939941\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c" Nov 03 02:09:47.726173 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:47.726148 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69d57c7a-8495-4d67-9547-dfe082939941-credential-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c\" (UID: \"69d57c7a-8495-4d67-9547-dfe082939941\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c" Nov 03 02:09:47.726796 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:47.726773 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69d57c7a-8495-4d67-9547-dfe082939941-istiod-ca-cert\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c\" (UID: \"69d57c7a-8495-4d67-9547-dfe082939941\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c" Nov 03 02:09:47.726978 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:47.726961 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69d57c7a-8495-4d67-9547-dfe082939941-istio-data\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c\" (UID: \"69d57c7a-8495-4d67-9547-dfe082939941\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c" Nov 03 02:09:47.728604 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:47.728580 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69d57c7a-8495-4d67-9547-dfe082939941-istio-envoy\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c\" (UID: \"69d57c7a-8495-4d67-9547-dfe082939941\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c" Nov 03 02:09:47.731852 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:47.731827 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69d57c7a-8495-4d67-9547-dfe082939941-istio-podinfo\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c\" (UID: \"69d57c7a-8495-4d67-9547-dfe082939941\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c" Nov 03 02:09:47.778622 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:47.778552 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69d57c7a-8495-4d67-9547-dfe082939941-istio-token\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c\" (UID: \"69d57c7a-8495-4d67-9547-dfe082939941\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c" Nov 03 02:09:47.778756 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:47.778729 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7tcj\" (UniqueName: \"kubernetes.io/projected/69d57c7a-8495-4d67-9547-dfe082939941-kube-api-access-q7tcj\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c\" (UID: \"69d57c7a-8495-4d67-9547-dfe082939941\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c" Nov 03 02:09:47.837371 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:47.837331 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c" Nov 03 02:09:48.258667 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:48.258637 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c"] Nov 03 02:09:48.264793 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:09:48.264763 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69d57c7a_8495_4d67_9547_dfe082939941.slice/crio-60663abcbc3c844e4d2b9238f153c68783706a6152bce58a0c40b834f95995fd WatchSource:0}: Error finding container 60663abcbc3c844e4d2b9238f153c68783706a6152bce58a0c40b834f95995fd: Status 404 returned error can't find the container with id 60663abcbc3c844e4d2b9238f153c68783706a6152bce58a0c40b834f95995fd Nov 03 02:09:48.267327 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:48.267282 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:09:48.267403 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:48.267389 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:09:48.267449 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:48.267423 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:09:48.939410 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:48.939339 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c" event={"ID":"69d57c7a-8495-4d67-9547-dfe082939941","Type":"ContainerStarted","Data":"25d644dfb50b5d35b22b71078dfa66be4afcc07b728aa0db3ca3de6bc774b4e3"} Nov 03 02:09:48.939911 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:48.939431 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c" event={"ID":"69d57c7a-8495-4d67-9547-dfe082939941","Type":"ContainerStarted","Data":"60663abcbc3c844e4d2b9238f153c68783706a6152bce58a0c40b834f95995fd"} Nov 03 02:09:49.048266 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:49.048200 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c" podStartSLOduration=2.048180536 podStartE2EDuration="2.048180536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:09:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:09:49.0468903 +0000 UTC m=+1066.306088453" watchObservedRunningTime="2026-11-03 02:09:49.048180536 +0000 UTC m=+1066.307378694" Nov 03 02:09:49.650901 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:49.650865 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:09:49.837748 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:49.837713 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c" Nov 03 02:09:49.838916 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:49.838892 5605 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 03 02:09:49.838983 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:49.838941 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c" podUID="69d57c7a-8495-4d67-9547-dfe082939941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 03 02:09:50.838880 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:50.838844 5605 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 03 02:09:50.839382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:50.838896 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c" podUID="69d57c7a-8495-4d67-9547-dfe082939941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 03 02:09:51.866381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:51.862505 5605 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 03 02:09:51.866381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:51.862569 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c" podUID="69d57c7a-8495-4d67-9547-dfe082939941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 03 02:09:52.838296 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:52.838263 5605 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 03 02:09:52.838500 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:52.838315 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c" podUID="69d57c7a-8495-4d67-9547-dfe082939941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 03 02:09:53.838276 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:53.838240 5605 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 03 02:09:53.838688 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:53.838301 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c" podUID="69d57c7a-8495-4d67-9547-dfe082939941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 03 02:09:53.912867 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:09:53.912830 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:09:54.683040 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:54.682989 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-b-7e89b72d-kserve-7c88c79ddc-p9ttd" podUID="ee3127db-b893-45d9-abc3-582b458aa4c3" containerName="main" containerID="cri-o://530fc5f1030541c4f19a8e734b0c3e465c233f474363e5cd9abfba31402705d3" gracePeriod=45 Nov 03 02:09:54.837802 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:54.837766 5605 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 03 02:09:54.838103 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:54.837833 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c" podUID="69d57c7a-8495-4d67-9547-dfe082939941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 03 02:09:54.966093 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:54.966065 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-b-7e89b72d-kserve-7c88c79ddc-p9ttd" Nov 03 02:09:55.005628 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:55.005579 5605 generic.go:356] "Generic (PLEG): container finished" podID="ee3127db-b893-45d9-abc3-582b458aa4c3" containerID="530fc5f1030541c4f19a8e734b0c3e465c233f474363e5cd9abfba31402705d3" exitCode=0 Nov 03 02:09:55.005801 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:55.005634 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-b-7e89b72d-kserve-7c88c79ddc-p9ttd" Nov 03 02:09:55.005801 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:55.005649 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-b-7e89b72d-kserve-7c88c79ddc-p9ttd" event={"ID":"ee3127db-b893-45d9-abc3-582b458aa4c3","Type":"ContainerDied","Data":"530fc5f1030541c4f19a8e734b0c3e465c233f474363e5cd9abfba31402705d3"} Nov 03 02:09:55.005801 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:55.005690 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-b-7e89b72d-kserve-7c88c79ddc-p9ttd" event={"ID":"ee3127db-b893-45d9-abc3-582b458aa4c3","Type":"ContainerDied","Data":"d6369a1093e25d8e5a2e13c3a1537b683baf8104579ad8a56e0c90a0249673f8"} Nov 03 02:09:55.005801 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:55.005730 5605 scope.go:122] "RemoveContainer" containerID="530fc5f1030541c4f19a8e734b0c3e465c233f474363e5cd9abfba31402705d3" Nov 03 02:09:55.016905 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:55.016879 5605 scope.go:122] "RemoveContainer" containerID="530fc5f1030541c4f19a8e734b0c3e465c233f474363e5cd9abfba31402705d3" Nov 03 02:09:55.017271 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:09:55.017244 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"530fc5f1030541c4f19a8e734b0c3e465c233f474363e5cd9abfba31402705d3\": container with ID starting with 530fc5f1030541c4f19a8e734b0c3e465c233f474363e5cd9abfba31402705d3 not found: ID does not exist" containerID="530fc5f1030541c4f19a8e734b0c3e465c233f474363e5cd9abfba31402705d3" Nov 03 02:09:55.017391 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:55.017274 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"530fc5f1030541c4f19a8e734b0c3e465c233f474363e5cd9abfba31402705d3"} err="failed to get container status \"530fc5f1030541c4f19a8e734b0c3e465c233f474363e5cd9abfba31402705d3\": rpc error: code = NotFound desc = could not find container \"530fc5f1030541c4f19a8e734b0c3e465c233f474363e5cd9abfba31402705d3\": container with ID starting with 530fc5f1030541c4f19a8e734b0c3e465c233f474363e5cd9abfba31402705d3 not found: ID does not exist" Nov 03 02:09:55.030692 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:55.030661 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee3127db-b893-45d9-abc3-582b458aa4c3-tls-certs\" (UniqueName: \"kubernetes.io/secret/ee3127db-b893-45d9-abc3-582b458aa4c3-tls-certs\") pod \"ee3127db-b893-45d9-abc3-582b458aa4c3\" (UID: \"ee3127db-b893-45d9-abc3-582b458aa4c3\") " Nov 03 02:09:55.030831 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:55.030770 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee3127db-b893-45d9-abc3-582b458aa4c3-home\" (UniqueName: \"kubernetes.io/empty-dir/ee3127db-b893-45d9-abc3-582b458aa4c3-home\") pod \"ee3127db-b893-45d9-abc3-582b458aa4c3\" (UID: \"ee3127db-b893-45d9-abc3-582b458aa4c3\") " Nov 03 02:09:55.030831 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:55.030793 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee3127db-b893-45d9-abc3-582b458aa4c3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee3127db-b893-45d9-abc3-582b458aa4c3-dshm\") pod \"ee3127db-b893-45d9-abc3-582b458aa4c3\" (UID: \"ee3127db-b893-45d9-abc3-582b458aa4c3\") " Nov 03 02:09:55.030917 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:55.030892 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee3127db-b893-45d9-abc3-582b458aa4c3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee3127db-b893-45d9-abc3-582b458aa4c3-model-cache\") pod \"ee3127db-b893-45d9-abc3-582b458aa4c3\" (UID: \"ee3127db-b893-45d9-abc3-582b458aa4c3\") " Nov 03 02:09:55.030972 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:55.030935 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee3127db-b893-45d9-abc3-582b458aa4c3-kube-api-access-7xmk9\" (UniqueName: \"kubernetes.io/projected/ee3127db-b893-45d9-abc3-582b458aa4c3-kube-api-access-7xmk9\") pod \"ee3127db-b893-45d9-abc3-582b458aa4c3\" (UID: \"ee3127db-b893-45d9-abc3-582b458aa4c3\") " Nov 03 02:09:55.031029 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:55.030980 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee3127db-b893-45d9-abc3-582b458aa4c3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee3127db-b893-45d9-abc3-582b458aa4c3-tmp-dir\") pod \"ee3127db-b893-45d9-abc3-582b458aa4c3\" (UID: \"ee3127db-b893-45d9-abc3-582b458aa4c3\") " Nov 03 02:09:55.031203 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:55.031173 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee3127db-b893-45d9-abc3-582b458aa4c3-model-cache" pod "ee3127db-b893-45d9-abc3-582b458aa4c3" (UID: "ee3127db-b893-45d9-abc3-582b458aa4c3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:09:55.031617 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:55.031393 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee3127db-b893-45d9-abc3-582b458aa4c3-tmp-dir" pod "ee3127db-b893-45d9-abc3-582b458aa4c3" (UID: "ee3127db-b893-45d9-abc3-582b458aa4c3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:09:55.031617 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:55.031418 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee3127db-b893-45d9-abc3-582b458aa4c3-home" pod "ee3127db-b893-45d9-abc3-582b458aa4c3" (UID: "ee3127db-b893-45d9-abc3-582b458aa4c3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:09:55.031617 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:55.031552 5605 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee3127db-b893-45d9-abc3-582b458aa4c3-model-cache\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:09:55.031617 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:55.031577 5605 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee3127db-b893-45d9-abc3-582b458aa4c3-tmp-dir\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:09:55.031617 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:55.031594 5605 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee3127db-b893-45d9-abc3-582b458aa4c3-home\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:09:55.033340 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:55.033309 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee3127db-b893-45d9-abc3-582b458aa4c3-kube-api-access-7xmk9" pod "ee3127db-b893-45d9-abc3-582b458aa4c3" (UID: "ee3127db-b893-45d9-abc3-582b458aa4c3"). InnerVolumeSpecName "kube-api-access-7xmk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:09:55.033340 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:55.033327 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee3127db-b893-45d9-abc3-582b458aa4c3-dshm" pod "ee3127db-b893-45d9-abc3-582b458aa4c3" (UID: "ee3127db-b893-45d9-abc3-582b458aa4c3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:09:55.033664 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:55.033643 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee3127db-b893-45d9-abc3-582b458aa4c3-tls-certs" pod "ee3127db-b893-45d9-abc3-582b458aa4c3" (UID: "ee3127db-b893-45d9-abc3-582b458aa4c3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:09:55.132989 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:55.132945 5605 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee3127db-b893-45d9-abc3-582b458aa4c3-dshm\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:09:55.132989 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:55.132987 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xmk9\" (UniqueName: \"kubernetes.io/projected/ee3127db-b893-45d9-abc3-582b458aa4c3-kube-api-access-7xmk9\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:09:55.133203 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:55.133003 5605 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee3127db-b893-45d9-abc3-582b458aa4c3-tls-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:09:55.422995 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:55.422958 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-b-7e89b72d-kserve-7c88c79ddc-p9ttd"] Nov 03 02:09:55.426935 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:55.426902 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-b-7e89b72d-kserve-7c88c79ddc-p9ttd"] Nov 03 02:09:55.844316 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:55.844281 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c" Nov 03 02:09:55.844694 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:55.844669 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c" Nov 03 02:09:55.845327 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:55.845304 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c" Nov 03 02:09:56.880947 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:09:56.870289 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee3127db-b893-45d9-abc3-582b458aa4c3" path="/var/lib/kubelet/pods/ee3127db-b893-45d9-abc3-582b458aa4c3/volumes" Nov 03 02:10:10.150556 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:10.150524 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2df4dda6-mds7w"] Nov 03 02:10:10.150990 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:10.150976 5605 cpu_manager.go:414] "containerMap: removing container" podUID="ee3127db-b893-45d9-abc3-582b458aa4c3" containerName="main" Nov 03 02:10:10.151033 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:10.150991 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee3127db-b893-45d9-abc3-582b458aa4c3" containerName="main" Nov 03 02:10:10.151088 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:10.151078 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee3127db-b893-45d9-abc3-582b458aa4c3" containerName="main" Nov 03 02:10:10.159981 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:10.159954 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2df4dda6-mds7w" Nov 03 02:10:10.206569 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:10.206514 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abd0f382-d3ad-4fbc-93da-69b7c9c98715-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2df4dda6-mds7w\" (UID: \"abd0f382-d3ad-4fbc-93da-69b7c9c98715\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2df4dda6-mds7w" Nov 03 02:10:10.206760 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:10.206630 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abd0f382-d3ad-4fbc-93da-69b7c9c98715-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2df4dda6-mds7w\" (UID: \"abd0f382-d3ad-4fbc-93da-69b7c9c98715\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2df4dda6-mds7w" Nov 03 02:10:10.206760 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:10.206660 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4z8k\" (UniqueName: \"kubernetes.io/projected/abd0f382-d3ad-4fbc-93da-69b7c9c98715-kube-api-access-v4z8k\") pod \"maas-api-revoke-sub-2df4dda6-mds7w\" (UID: \"abd0f382-d3ad-4fbc-93da-69b7c9c98715\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2df4dda6-mds7w" Nov 03 02:10:10.207337 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:10.207314 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2df4dda6-mds7w"] Nov 03 02:10:10.251277 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:10.251253 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:10:10.255310 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:10.255288 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:10:10.255400 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:10.255325 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:10:10.255612 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:10.255596 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:10:10.307303 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:10.307272 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abd0f382-d3ad-4fbc-93da-69b7c9c98715-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2df4dda6-mds7w\" (UID: \"abd0f382-d3ad-4fbc-93da-69b7c9c98715\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2df4dda6-mds7w" Nov 03 02:10:10.307472 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:10.307383 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abd0f382-d3ad-4fbc-93da-69b7c9c98715-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2df4dda6-mds7w\" (UID: \"abd0f382-d3ad-4fbc-93da-69b7c9c98715\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2df4dda6-mds7w" Nov 03 02:10:10.307472 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:10.307414 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4z8k\" (UniqueName: \"kubernetes.io/projected/abd0f382-d3ad-4fbc-93da-69b7c9c98715-kube-api-access-v4z8k\") pod \"maas-api-revoke-sub-2df4dda6-mds7w\" (UID: \"abd0f382-d3ad-4fbc-93da-69b7c9c98715\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2df4dda6-mds7w" Nov 03 02:10:10.308013 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:10.307989 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abd0f382-d3ad-4fbc-93da-69b7c9c98715-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2df4dda6-mds7w\" (UID: \"abd0f382-d3ad-4fbc-93da-69b7c9c98715\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2df4dda6-mds7w" Nov 03 02:10:10.309800 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:10.309779 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abd0f382-d3ad-4fbc-93da-69b7c9c98715-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2df4dda6-mds7w\" (UID: \"abd0f382-d3ad-4fbc-93da-69b7c9c98715\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2df4dda6-mds7w" Nov 03 02:10:10.360972 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:10.360938 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4z8k\" (UniqueName: \"kubernetes.io/projected/abd0f382-d3ad-4fbc-93da-69b7c9c98715-kube-api-access-v4z8k\") pod \"maas-api-revoke-sub-2df4dda6-mds7w\" (UID: \"abd0f382-d3ad-4fbc-93da-69b7c9c98715\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2df4dda6-mds7w" Nov 03 02:10:10.473845 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:10.473760 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2df4dda6-mds7w" Nov 03 02:10:10.841742 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:10.841658 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2df4dda6-mds7w"] Nov 03 02:10:10.847701 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:10:10.847669 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabd0f382_d3ad_4fbc_93da_69b7c9c98715.slice/crio-e87a62d44bba4f991d24fa8c1ff70ab91a52b3b300a5dac65d8f0016032bb9a0 WatchSource:0}: Error finding container e87a62d44bba4f991d24fa8c1ff70ab91a52b3b300a5dac65d8f0016032bb9a0: Status 404 returned error can't find the container with id e87a62d44bba4f991d24fa8c1ff70ab91a52b3b300a5dac65d8f0016032bb9a0 Nov 03 02:10:11.154250 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:11.154159 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2df4dda6-mds7w" event={"ID":"abd0f382-d3ad-4fbc-93da-69b7c9c98715","Type":"ContainerStarted","Data":"e87a62d44bba4f991d24fa8c1ff70ab91a52b3b300a5dac65d8f0016032bb9a0"} Nov 03 02:10:12.160617 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:12.160581 5605 generic.go:356] "Generic (PLEG): container finished" podID="abd0f382-d3ad-4fbc-93da-69b7c9c98715" containerID="1a2420b64178a15abce2299dc1f7c9bfafb7fd0e4f3e100890acdb0729157bd6" exitCode=0 Nov 03 02:10:12.161018 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:12.160669 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2df4dda6-mds7w" event={"ID":"abd0f382-d3ad-4fbc-93da-69b7c9c98715","Type":"ContainerDied","Data":"1a2420b64178a15abce2299dc1f7c9bfafb7fd0e4f3e100890acdb0729157bd6"} Nov 03 02:10:13.314277 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:13.314250 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2df4dda6-mds7w" Nov 03 02:10:13.349559 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:13.349526 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abd0f382-d3ad-4fbc-93da-69b7c9c98715-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abd0f382-d3ad-4fbc-93da-69b7c9c98715-maas-api-openshift-ca\") pod \"abd0f382-d3ad-4fbc-93da-69b7c9c98715\" (UID: \"abd0f382-d3ad-4fbc-93da-69b7c9c98715\") " Nov 03 02:10:13.349724 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:13.349565 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abd0f382-d3ad-4fbc-93da-69b7c9c98715-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abd0f382-d3ad-4fbc-93da-69b7c9c98715-maas-api-xks-ca\") pod \"abd0f382-d3ad-4fbc-93da-69b7c9c98715\" (UID: \"abd0f382-d3ad-4fbc-93da-69b7c9c98715\") " Nov 03 02:10:13.349724 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:13.349632 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abd0f382-d3ad-4fbc-93da-69b7c9c98715-kube-api-access-v4z8k\" (UniqueName: \"kubernetes.io/projected/abd0f382-d3ad-4fbc-93da-69b7c9c98715-kube-api-access-v4z8k\") pod \"abd0f382-d3ad-4fbc-93da-69b7c9c98715\" (UID: \"abd0f382-d3ad-4fbc-93da-69b7c9c98715\") " Nov 03 02:10:13.349904 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:13.349880 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abd0f382-d3ad-4fbc-93da-69b7c9c98715-maas-api-openshift-ca" pod "abd0f382-d3ad-4fbc-93da-69b7c9c98715" (UID: "abd0f382-d3ad-4fbc-93da-69b7c9c98715"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:10:13.350125 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:13.350105 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abd0f382-d3ad-4fbc-93da-69b7c9c98715-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:13.351718 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:13.351698 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abd0f382-d3ad-4fbc-93da-69b7c9c98715-maas-api-xks-ca" pod "abd0f382-d3ad-4fbc-93da-69b7c9c98715" (UID: "abd0f382-d3ad-4fbc-93da-69b7c9c98715"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:10:13.351789 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:13.351760 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abd0f382-d3ad-4fbc-93da-69b7c9c98715-kube-api-access-v4z8k" pod "abd0f382-d3ad-4fbc-93da-69b7c9c98715" (UID: "abd0f382-d3ad-4fbc-93da-69b7c9c98715"). InnerVolumeSpecName "kube-api-access-v4z8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:10:13.451740 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:13.451639 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abd0f382-d3ad-4fbc-93da-69b7c9c98715-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:13.451740 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:13.451687 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4z8k\" (UniqueName: \"kubernetes.io/projected/abd0f382-d3ad-4fbc-93da-69b7c9c98715-kube-api-access-v4z8k\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:14.171410 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:14.171379 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2df4dda6-mds7w" Nov 03 02:10:14.171410 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:14.171386 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2df4dda6-mds7w" event={"ID":"abd0f382-d3ad-4fbc-93da-69b7c9c98715","Type":"ContainerDied","Data":"e87a62d44bba4f991d24fa8c1ff70ab91a52b3b300a5dac65d8f0016032bb9a0"} Nov 03 02:10:14.171629 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:14.171444 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e87a62d44bba4f991d24fa8c1ff70ab91a52b3b300a5dac65d8f0016032bb9a0" Nov 03 02:10:14.839700 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:14.839667 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:10:17.609503 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:17.609453 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5" podUID="fffc3fbb-097a-4a5d-82c8-b1e221d0e710" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:17.744390 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:17.713419 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1bccbd/distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg" podUID="5a71cdee-425e-4787-9217-5fb6b97e232b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:17.744390 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:17.743095 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-521f1f/simulator-w2-521f1f-kserve-5f47b86f6-mgksf" podUID="6e1b7fda-0ade-4042-ad63-5115bc66b454" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:17.834980 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:17.834938 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a0e72e/simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq" podUID="1901f57b-279b-4cbf-bf0c-054abc5786d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Nov 03 02:10:23.912200 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:10:23.912152 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:10:26.810214 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:26.810177 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29c33734-vqktr"] Nov 03 02:10:26.810716 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:26.810613 5605 cpu_manager.go:414] "containerMap: removing container" podUID="abd0f382-d3ad-4fbc-93da-69b7c9c98715" containerName="revoke-keys" Nov 03 02:10:26.810716 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:26.810625 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abd0f382-d3ad-4fbc-93da-69b7c9c98715" containerName="revoke-keys" Nov 03 02:10:26.810804 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:26.810737 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="abd0f382-d3ad-4fbc-93da-69b7c9c98715" containerName="revoke-keys" Nov 03 02:10:26.814424 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:26.814400 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29c33734-vqktr" Nov 03 02:10:26.864216 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:26.864186 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:10:26.867296 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:26.867270 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29c33734-vqktr"] Nov 03 02:10:26.904955 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:26.904687 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6fkj\" (UniqueName: \"kubernetes.io/projected/555f7460-fcf6-4517-8181-157e5bd65ddf-kube-api-access-d6fkj\") pod \"maas-api-revoke-sub-29c33734-vqktr\" (UID: \"555f7460-fcf6-4517-8181-157e5bd65ddf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29c33734-vqktr" Nov 03 02:10:26.904955 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:26.904736 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/555f7460-fcf6-4517-8181-157e5bd65ddf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29c33734-vqktr\" (UID: \"555f7460-fcf6-4517-8181-157e5bd65ddf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29c33734-vqktr" Nov 03 02:10:26.904955 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:26.904944 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/555f7460-fcf6-4517-8181-157e5bd65ddf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29c33734-vqktr\" (UID: \"555f7460-fcf6-4517-8181-157e5bd65ddf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29c33734-vqktr" Nov 03 02:10:26.910922 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:26.910895 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:10:26.911070 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:26.911019 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:10:26.911070 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:26.911031 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:10:27.006088 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:27.006055 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6fkj\" (UniqueName: \"kubernetes.io/projected/555f7460-fcf6-4517-8181-157e5bd65ddf-kube-api-access-d6fkj\") pod \"maas-api-revoke-sub-29c33734-vqktr\" (UID: \"555f7460-fcf6-4517-8181-157e5bd65ddf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29c33734-vqktr" Nov 03 02:10:27.006088 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:27.006091 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/555f7460-fcf6-4517-8181-157e5bd65ddf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29c33734-vqktr\" (UID: \"555f7460-fcf6-4517-8181-157e5bd65ddf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29c33734-vqktr" Nov 03 02:10:27.006276 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:27.006143 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/555f7460-fcf6-4517-8181-157e5bd65ddf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29c33734-vqktr\" (UID: \"555f7460-fcf6-4517-8181-157e5bd65ddf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29c33734-vqktr" Nov 03 02:10:27.006818 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:27.006787 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/555f7460-fcf6-4517-8181-157e5bd65ddf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29c33734-vqktr\" (UID: \"555f7460-fcf6-4517-8181-157e5bd65ddf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29c33734-vqktr" Nov 03 02:10:27.008539 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:27.008518 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/555f7460-fcf6-4517-8181-157e5bd65ddf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29c33734-vqktr\" (UID: \"555f7460-fcf6-4517-8181-157e5bd65ddf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29c33734-vqktr" Nov 03 02:10:27.059061 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:27.059029 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6fkj\" (UniqueName: \"kubernetes.io/projected/555f7460-fcf6-4517-8181-157e5bd65ddf-kube-api-access-d6fkj\") pod \"maas-api-revoke-sub-29c33734-vqktr\" (UID: \"555f7460-fcf6-4517-8181-157e5bd65ddf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29c33734-vqktr" Nov 03 02:10:27.128084 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:27.128000 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29c33734-vqktr" Nov 03 02:10:27.496110 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:27.496029 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29c33734-vqktr"] Nov 03 02:10:27.501751 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:10:27.501723 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod555f7460_fcf6_4517_8181_157e5bd65ddf.slice/crio-20cfce35a4a0dc0c5b310084e494eda2f3e5e55d06fd99c2cbba50fb85eac4f9 WatchSource:0}: Error finding container 20cfce35a4a0dc0c5b310084e494eda2f3e5e55d06fd99c2cbba50fb85eac4f9: Status 404 returned error can't find the container with id 20cfce35a4a0dc0c5b310084e494eda2f3e5e55d06fd99c2cbba50fb85eac4f9 Nov 03 02:10:28.270252 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:28.270218 5605 generic.go:356] "Generic (PLEG): container finished" podID="555f7460-fcf6-4517-8181-157e5bd65ddf" containerID="7739f3acfd1d29155b570fe5cfab0aa8f0d68ba8741c381f020ef8cea05afa88" exitCode=0 Nov 03 02:10:28.270615 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:28.270295 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29c33734-vqktr" event={"ID":"555f7460-fcf6-4517-8181-157e5bd65ddf","Type":"ContainerDied","Data":"7739f3acfd1d29155b570fe5cfab0aa8f0d68ba8741c381f020ef8cea05afa88"} Nov 03 02:10:28.270615 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:28.270346 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29c33734-vqktr" event={"ID":"555f7460-fcf6-4517-8181-157e5bd65ddf","Type":"ContainerStarted","Data":"20cfce35a4a0dc0c5b310084e494eda2f3e5e55d06fd99c2cbba50fb85eac4f9"} Nov 03 02:10:29.409513 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:29.409485 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29c33734-vqktr" Nov 03 02:10:29.432794 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:29.432766 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/555f7460-fcf6-4517-8181-157e5bd65ddf-kube-api-access-d6fkj\" (UniqueName: \"kubernetes.io/projected/555f7460-fcf6-4517-8181-157e5bd65ddf-kube-api-access-d6fkj\") pod \"555f7460-fcf6-4517-8181-157e5bd65ddf\" (UID: \"555f7460-fcf6-4517-8181-157e5bd65ddf\") " Nov 03 02:10:29.432927 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:29.432815 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/555f7460-fcf6-4517-8181-157e5bd65ddf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/555f7460-fcf6-4517-8181-157e5bd65ddf-maas-api-xks-ca\") pod \"555f7460-fcf6-4517-8181-157e5bd65ddf\" (UID: \"555f7460-fcf6-4517-8181-157e5bd65ddf\") " Nov 03 02:10:29.432927 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:29.432866 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/555f7460-fcf6-4517-8181-157e5bd65ddf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/555f7460-fcf6-4517-8181-157e5bd65ddf-maas-api-openshift-ca\") pod \"555f7460-fcf6-4517-8181-157e5bd65ddf\" (UID: \"555f7460-fcf6-4517-8181-157e5bd65ddf\") " Nov 03 02:10:29.433249 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:29.433227 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/555f7460-fcf6-4517-8181-157e5bd65ddf-maas-api-openshift-ca" pod "555f7460-fcf6-4517-8181-157e5bd65ddf" (UID: "555f7460-fcf6-4517-8181-157e5bd65ddf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:10:29.434810 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:29.434786 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/555f7460-fcf6-4517-8181-157e5bd65ddf-maas-api-xks-ca" pod "555f7460-fcf6-4517-8181-157e5bd65ddf" (UID: "555f7460-fcf6-4517-8181-157e5bd65ddf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:10:29.434871 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:29.434805 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/555f7460-fcf6-4517-8181-157e5bd65ddf-kube-api-access-d6fkj" pod "555f7460-fcf6-4517-8181-157e5bd65ddf" (UID: "555f7460-fcf6-4517-8181-157e5bd65ddf"). InnerVolumeSpecName "kube-api-access-d6fkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:10:29.534294 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:29.534216 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6fkj\" (UniqueName: \"kubernetes.io/projected/555f7460-fcf6-4517-8181-157e5bd65ddf-kube-api-access-d6fkj\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:29.534294 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:29.534244 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/555f7460-fcf6-4517-8181-157e5bd65ddf-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:29.534294 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:29.534256 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/555f7460-fcf6-4517-8181-157e5bd65ddf-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:30.283496 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:30.283460 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29c33734-vqktr" Nov 03 02:10:30.283668 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:30.283486 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29c33734-vqktr" event={"ID":"555f7460-fcf6-4517-8181-157e5bd65ddf","Type":"ContainerDied","Data":"20cfce35a4a0dc0c5b310084e494eda2f3e5e55d06fd99c2cbba50fb85eac4f9"} Nov 03 02:10:30.283668 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:30.283538 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20cfce35a4a0dc0c5b310084e494eda2f3e5e55d06fd99c2cbba50fb85eac4f9" Nov 03 02:10:32.834262 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:32.834230 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22"] Nov 03 02:10:32.834789 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:32.834763 5605 cpu_manager.go:414] "containerMap: removing container" podUID="555f7460-fcf6-4517-8181-157e5bd65ddf" containerName="revoke-keys" Nov 03 02:10:32.834847 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:32.834791 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="555f7460-fcf6-4517-8181-157e5bd65ddf" containerName="revoke-keys" Nov 03 02:10:32.834935 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:32.834921 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="555f7460-fcf6-4517-8181-157e5bd65ddf" containerName="revoke-keys" Nov 03 02:10:32.838668 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:32.838647 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22" Nov 03 02:10:32.884545 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:32.884513 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdbfc7ca-1928-4b51-8967-cb5576ca80cb-workload-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22\" (UID: \"fdbfc7ca-1928-4b51-8967-cb5576ca80cb\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22" Nov 03 02:10:32.884665 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:32.884556 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdbfc7ca-1928-4b51-8967-cb5576ca80cb-istio-data\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22\" (UID: \"fdbfc7ca-1928-4b51-8967-cb5576ca80cb\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22" Nov 03 02:10:32.884665 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:32.884610 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdbfc7ca-1928-4b51-8967-cb5576ca80cb-istio-envoy\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22\" (UID: \"fdbfc7ca-1928-4b51-8967-cb5576ca80cb\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22" Nov 03 02:10:32.884752 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:32.884699 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdbfc7ca-1928-4b51-8967-cb5576ca80cb-istiod-ca-cert\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22\" (UID: \"fdbfc7ca-1928-4b51-8967-cb5576ca80cb\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22" Nov 03 02:10:32.884787 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:32.884764 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdbfc7ca-1928-4b51-8967-cb5576ca80cb-istio-podinfo\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22\" (UID: \"fdbfc7ca-1928-4b51-8967-cb5576ca80cb\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22" Nov 03 02:10:32.884865 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:32.884849 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdbfc7ca-1928-4b51-8967-cb5576ca80cb-credential-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22\" (UID: \"fdbfc7ca-1928-4b51-8967-cb5576ca80cb\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22" Nov 03 02:10:32.884899 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:32.884877 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdbfc7ca-1928-4b51-8967-cb5576ca80cb-workload-certs\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22\" (UID: \"fdbfc7ca-1928-4b51-8967-cb5576ca80cb\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22" Nov 03 02:10:32.884939 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:32.884899 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j86gx\" (UniqueName: \"kubernetes.io/projected/fdbfc7ca-1928-4b51-8967-cb5576ca80cb-kube-api-access-j86gx\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22\" (UID: \"fdbfc7ca-1928-4b51-8967-cb5576ca80cb\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22" Nov 03 02:10:32.884980 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:32.884966 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdbfc7ca-1928-4b51-8967-cb5576ca80cb-istio-token\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22\" (UID: \"fdbfc7ca-1928-4b51-8967-cb5576ca80cb\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22" Nov 03 02:10:32.891735 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:32.891709 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22"] Nov 03 02:10:32.986584 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:32.986551 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdbfc7ca-1928-4b51-8967-cb5576ca80cb-credential-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22\" (UID: \"fdbfc7ca-1928-4b51-8967-cb5576ca80cb\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22" Nov 03 02:10:32.986584 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:32.986589 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdbfc7ca-1928-4b51-8967-cb5576ca80cb-workload-certs\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22\" (UID: \"fdbfc7ca-1928-4b51-8967-cb5576ca80cb\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22" Nov 03 02:10:32.986796 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:32.986610 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j86gx\" (UniqueName: \"kubernetes.io/projected/fdbfc7ca-1928-4b51-8967-cb5576ca80cb-kube-api-access-j86gx\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22\" (UID: \"fdbfc7ca-1928-4b51-8967-cb5576ca80cb\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22" Nov 03 02:10:32.986796 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:32.986736 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdbfc7ca-1928-4b51-8967-cb5576ca80cb-istio-token\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22\" (UID: \"fdbfc7ca-1928-4b51-8967-cb5576ca80cb\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22" Nov 03 02:10:32.986914 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:32.986859 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdbfc7ca-1928-4b51-8967-cb5576ca80cb-workload-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22\" (UID: \"fdbfc7ca-1928-4b51-8967-cb5576ca80cb\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22" Nov 03 02:10:32.986914 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:32.986881 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdbfc7ca-1928-4b51-8967-cb5576ca80cb-istio-data\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22\" (UID: \"fdbfc7ca-1928-4b51-8967-cb5576ca80cb\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22" Nov 03 02:10:32.986914 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:32.986902 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdbfc7ca-1928-4b51-8967-cb5576ca80cb-istio-envoy\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22\" (UID: \"fdbfc7ca-1928-4b51-8967-cb5576ca80cb\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22" Nov 03 02:10:32.987057 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:32.986942 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdbfc7ca-1928-4b51-8967-cb5576ca80cb-istiod-ca-cert\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22\" (UID: \"fdbfc7ca-1928-4b51-8967-cb5576ca80cb\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22" Nov 03 02:10:32.987057 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:32.987001 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdbfc7ca-1928-4b51-8967-cb5576ca80cb-workload-certs\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22\" (UID: \"fdbfc7ca-1928-4b51-8967-cb5576ca80cb\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22" Nov 03 02:10:32.987057 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:32.987014 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdbfc7ca-1928-4b51-8967-cb5576ca80cb-credential-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22\" (UID: \"fdbfc7ca-1928-4b51-8967-cb5576ca80cb\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22" Nov 03 02:10:32.987057 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:32.987006 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdbfc7ca-1928-4b51-8967-cb5576ca80cb-istio-podinfo\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22\" (UID: \"fdbfc7ca-1928-4b51-8967-cb5576ca80cb\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22" Nov 03 02:10:32.987245 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:32.987217 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdbfc7ca-1928-4b51-8967-cb5576ca80cb-workload-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22\" (UID: \"fdbfc7ca-1928-4b51-8967-cb5576ca80cb\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22" Nov 03 02:10:32.987307 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:32.987245 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdbfc7ca-1928-4b51-8967-cb5576ca80cb-istio-data\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22\" (UID: \"fdbfc7ca-1928-4b51-8967-cb5576ca80cb\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22" Nov 03 02:10:32.987698 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:32.987679 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdbfc7ca-1928-4b51-8967-cb5576ca80cb-istiod-ca-cert\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22\" (UID: \"fdbfc7ca-1928-4b51-8967-cb5576ca80cb\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22" Nov 03 02:10:32.989197 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:32.989176 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdbfc7ca-1928-4b51-8967-cb5576ca80cb-istio-envoy\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22\" (UID: \"fdbfc7ca-1928-4b51-8967-cb5576ca80cb\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22" Nov 03 02:10:32.989323 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:32.989304 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdbfc7ca-1928-4b51-8967-cb5576ca80cb-istio-podinfo\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22\" (UID: \"fdbfc7ca-1928-4b51-8967-cb5576ca80cb\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22" Nov 03 02:10:33.039262 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:33.039235 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdbfc7ca-1928-4b51-8967-cb5576ca80cb-istio-token\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22\" (UID: \"fdbfc7ca-1928-4b51-8967-cb5576ca80cb\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22" Nov 03 02:10:33.039437 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:33.039390 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j86gx\" (UniqueName: \"kubernetes.io/projected/fdbfc7ca-1928-4b51-8967-cb5576ca80cb-kube-api-access-j86gx\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22\" (UID: \"fdbfc7ca-1928-4b51-8967-cb5576ca80cb\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22" Nov 03 02:10:33.151762 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:33.151681 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22" Nov 03 02:10:33.572854 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:33.572824 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22"] Nov 03 02:10:33.578452 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:10:33.578420 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdbfc7ca_1928_4b51_8967_cb5576ca80cb.slice/crio-2468a114429c5f57ed35b0f11cdfb99aac9555b29c93d383d880d82430cde7d3 WatchSource:0}: Error finding container 2468a114429c5f57ed35b0f11cdfb99aac9555b29c93d383d880d82430cde7d3: Status 404 returned error can't find the container with id 2468a114429c5f57ed35b0f11cdfb99aac9555b29c93d383d880d82430cde7d3 Nov 03 02:10:33.580778 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:33.580744 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:10:33.580871 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:33.580811 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:10:33.580871 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:33.580840 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:10:34.305281 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:34.305245 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22" event={"ID":"fdbfc7ca-1928-4b51-8967-cb5576ca80cb","Type":"ContainerStarted","Data":"4817d715e5eedfff169b9ee48028f6efbce0d42dc0a03116f95f99fe8448e052"} Nov 03 02:10:34.305712 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:34.305296 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22" event={"ID":"fdbfc7ca-1928-4b51-8967-cb5576ca80cb","Type":"ContainerStarted","Data":"2468a114429c5f57ed35b0f11cdfb99aac9555b29c93d383d880d82430cde7d3"} Nov 03 02:10:34.478383 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:34.477339 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22" podStartSLOduration=2.477321116 podStartE2EDuration="2.477321116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:10:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:10:34.476697526 +0000 UTC m=+1111.735895702" watchObservedRunningTime="2026-11-03 02:10:34.477321116 +0000 UTC m=+1111.736519270" Nov 03 02:10:35.152702 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:35.152666 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22" Nov 03 02:10:35.154334 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:35.154302 5605 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 03 02:10:35.154490 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:35.154369 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22" podUID="fdbfc7ca-1928-4b51-8967-cb5576ca80cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 03 02:10:36.152799 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:36.152756 5605 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 03 02:10:36.153317 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:36.152805 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22" podUID="fdbfc7ca-1928-4b51-8967-cb5576ca80cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 03 02:10:36.223620 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:36.223588 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f7d9bb/distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr"] Nov 03 02:10:36.227461 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:36.227440 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr" Nov 03 02:10:36.280022 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:36.279988 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f7d9bb\"/\"distinct-w0-f7d9bb-kserve-self-signed-certs\"" Nov 03 02:10:36.325212 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:36.325173 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f7d9bb/distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr"] Nov 03 02:10:36.328098 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:36.328068 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8282371b-c42d-43ab-a5c9-e07f2464dcdc-model-cache\") pod \"distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr\" (UID: \"8282371b-c42d-43ab-a5c9-e07f2464dcdc\") " pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr" Nov 03 02:10:36.328233 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:36.328161 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8282371b-c42d-43ab-a5c9-e07f2464dcdc-tls-certs\") pod \"distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr\" (UID: \"8282371b-c42d-43ab-a5c9-e07f2464dcdc\") " pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr" Nov 03 02:10:36.328233 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:36.328201 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8282371b-c42d-43ab-a5c9-e07f2464dcdc-dshm\") pod \"distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr\" (UID: \"8282371b-c42d-43ab-a5c9-e07f2464dcdc\") " pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr" Nov 03 02:10:36.328311 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:36.328292 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8282371b-c42d-43ab-a5c9-e07f2464dcdc-tmp-dir\") pod \"distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr\" (UID: \"8282371b-c42d-43ab-a5c9-e07f2464dcdc\") " pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr" Nov 03 02:10:36.328377 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:36.328337 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8282371b-c42d-43ab-a5c9-e07f2464dcdc-home\") pod \"distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr\" (UID: \"8282371b-c42d-43ab-a5c9-e07f2464dcdc\") " pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr" Nov 03 02:10:36.328522 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:36.328503 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx29r\" (UniqueName: \"kubernetes.io/projected/8282371b-c42d-43ab-a5c9-e07f2464dcdc-kube-api-access-bx29r\") pod \"distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr\" (UID: \"8282371b-c42d-43ab-a5c9-e07f2464dcdc\") " pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr" Nov 03 02:10:36.437432 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:36.435087 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8282371b-c42d-43ab-a5c9-e07f2464dcdc-model-cache\") pod \"distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr\" (UID: \"8282371b-c42d-43ab-a5c9-e07f2464dcdc\") " pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr" Nov 03 02:10:36.437432 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:36.435163 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8282371b-c42d-43ab-a5c9-e07f2464dcdc-tls-certs\") pod \"distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr\" (UID: \"8282371b-c42d-43ab-a5c9-e07f2464dcdc\") " pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr" Nov 03 02:10:36.437432 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:36.435236 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8282371b-c42d-43ab-a5c9-e07f2464dcdc-dshm\") pod \"distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr\" (UID: \"8282371b-c42d-43ab-a5c9-e07f2464dcdc\") " pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr" Nov 03 02:10:36.437432 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:36.435286 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8282371b-c42d-43ab-a5c9-e07f2464dcdc-tmp-dir\") pod \"distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr\" (UID: \"8282371b-c42d-43ab-a5c9-e07f2464dcdc\") " pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr" Nov 03 02:10:36.437432 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:36.435323 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8282371b-c42d-43ab-a5c9-e07f2464dcdc-home\") pod \"distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr\" (UID: \"8282371b-c42d-43ab-a5c9-e07f2464dcdc\") " pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr" Nov 03 02:10:36.437432 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:36.435434 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx29r\" (UniqueName: \"kubernetes.io/projected/8282371b-c42d-43ab-a5c9-e07f2464dcdc-kube-api-access-bx29r\") pod \"distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr\" (UID: \"8282371b-c42d-43ab-a5c9-e07f2464dcdc\") " pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr" Nov 03 02:10:36.437432 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:36.436168 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8282371b-c42d-43ab-a5c9-e07f2464dcdc-model-cache\") pod \"distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr\" (UID: \"8282371b-c42d-43ab-a5c9-e07f2464dcdc\") " pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr" Nov 03 02:10:36.437432 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:36.436836 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8282371b-c42d-43ab-a5c9-e07f2464dcdc-home\") pod \"distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr\" (UID: \"8282371b-c42d-43ab-a5c9-e07f2464dcdc\") " pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr" Nov 03 02:10:36.437970 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:36.437512 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8282371b-c42d-43ab-a5c9-e07f2464dcdc-tmp-dir\") pod \"distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr\" (UID: \"8282371b-c42d-43ab-a5c9-e07f2464dcdc\") " pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr" Nov 03 02:10:36.455765 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:36.451050 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8282371b-c42d-43ab-a5c9-e07f2464dcdc-dshm\") pod \"distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr\" (UID: \"8282371b-c42d-43ab-a5c9-e07f2464dcdc\") " pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr" Nov 03 02:10:36.459914 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:36.459881 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8282371b-c42d-43ab-a5c9-e07f2464dcdc-tls-certs\") pod \"distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr\" (UID: \"8282371b-c42d-43ab-a5c9-e07f2464dcdc\") " pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr" Nov 03 02:10:36.503833 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:36.503609 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx29r\" (UniqueName: \"kubernetes.io/projected/8282371b-c42d-43ab-a5c9-e07f2464dcdc-kube-api-access-bx29r\") pod \"distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr\" (UID: \"8282371b-c42d-43ab-a5c9-e07f2464dcdc\") " pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr" Nov 03 02:10:36.541269 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:36.541207 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr" Nov 03 02:10:37.138120 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:37.138029 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f7d9bb/distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr"] Nov 03 02:10:37.145579 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:10:37.145549 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8282371b_c42d_43ab_a5c9_e07f2464dcdc.slice/crio-e219fd27167f74f99b4c15bb989a0c138ece678ec7484e623e208af985d622fc WatchSource:0}: Error finding container e219fd27167f74f99b4c15bb989a0c138ece678ec7484e623e208af985d622fc: Status 404 returned error can't find the container with id e219fd27167f74f99b4c15bb989a0c138ece678ec7484e623e208af985d622fc Nov 03 02:10:37.152592 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:37.152547 5605 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 03 02:10:37.152727 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:37.152620 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22" podUID="fdbfc7ca-1928-4b51-8967-cb5576ca80cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 03 02:10:37.364182 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:37.364141 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr" event={"ID":"8282371b-c42d-43ab-a5c9-e07f2464dcdc","Type":"ContainerStarted","Data":"480a71ce8070590f25882499d2be0a4c3fb366ef13abf4514f5982aa74a059fb"} Nov 03 02:10:37.364786 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:37.364758 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr" event={"ID":"8282371b-c42d-43ab-a5c9-e07f2464dcdc","Type":"ContainerStarted","Data":"e219fd27167f74f99b4c15bb989a0c138ece678ec7484e623e208af985d622fc"} Nov 03 02:10:37.491432 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:37.477974 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr" podStartSLOduration=1.477954065 podStartE2EDuration="1.477954065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:10:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:10:37.474727524 +0000 UTC m=+1114.733925678" watchObservedRunningTime="2026-11-03 02:10:37.477954065 +0000 UTC m=+1114.737152217" Nov 03 02:10:37.541743 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:37.541711 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr" Nov 03 02:10:38.164119 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:38.164081 5605 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 03 02:10:38.164457 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:38.164421 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22" podUID="fdbfc7ca-1928-4b51-8967-cb5576ca80cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 03 02:10:39.153550 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:39.153512 5605 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 03 02:10:39.154156 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:39.154126 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22" podUID="fdbfc7ca-1928-4b51-8967-cb5576ca80cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 03 02:10:39.651579 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:39.651516 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5" podUID="fffc3fbb-097a-4a5d-82c8-b1e221d0e710" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:39.763520 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:39.763474 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1bccbd/distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg" podUID="5a71cdee-425e-4787-9217-5fb6b97e232b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:40.104446 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:40.044310 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fb9aaea-f46v2"] Nov 03 02:10:40.113379 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:40.111712 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fb9aaea-f46v2"] Nov 03 02:10:40.152144 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:40.152107 5605 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 03 02:10:40.152465 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:40.152433 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22" podUID="fdbfc7ca-1928-4b51-8967-cb5576ca80cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 03 02:10:40.153156 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:40.152693 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f7d9bb/simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls" podUID="9173364c-14ba-4287-8587-45f25118e3e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:40.232071 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:40.214502 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1bccbd/distinct-w1-1bccbd-kserve-5dfc85555f-tkvg6" podUID="dd4a9d44-c63e-4f67-bcac-a4f98ecf1796" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:40.687523 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:40.687471 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5" podUID="fffc3fbb-097a-4a5d-82c8-b1e221d0e710" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:40.762588 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:40.748419 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c7d758/distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5" Nov 03 02:10:40.762588 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:40.724192 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-521f1f/simulator-w2-521f1f-kserve-5f47b86f6-mgksf" podUID="6e1b7fda-0ade-4042-ad63-5115bc66b454" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:40.800941 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:40.800904 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c7d758/distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5" Nov 03 02:10:40.836539 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:40.836486 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a0e72e/simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq" podUID="1901f57b-279b-4cbf-bf0c-054abc5786d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:40.852943 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:40.852906 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8c56fa6-4d99-4622-b342-dd39dd949145" path="/var/lib/kubelet/pods/d8c56fa6-4d99-4622-b342-dd39dd949145/volumes" Nov 03 02:10:41.153210 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:41.153182 5605 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 03 02:10:41.153383 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:41.153251 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22" podUID="fdbfc7ca-1928-4b51-8967-cb5576ca80cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 03 02:10:43.194553 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:43.194507 5605 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:10:43.195039 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:43.194580 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22" podUID="fdbfc7ca-1928-4b51-8967-cb5576ca80cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:43.405555 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:43.405518 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22" Nov 03 02:10:43.406187 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:43.406158 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22" Nov 03 02:10:43.408823 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:43.408800 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22" Nov 03 02:10:44.060330 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:44.060300 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fca8eb6-sj67k"] Nov 03 02:10:44.065147 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:44.065119 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fca8eb6-sj67k" Nov 03 02:10:44.085100 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:44.085068 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff93bdda-9ad2-4fbb-b36b-107aebce350d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fca8eb6-sj67k\" (UID: \"ff93bdda-9ad2-4fbb-b36b-107aebce350d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fca8eb6-sj67k" Nov 03 02:10:44.085271 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:44.085243 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2vmd\" (UniqueName: \"kubernetes.io/projected/ff93bdda-9ad2-4fbb-b36b-107aebce350d-kube-api-access-t2vmd\") pod \"maas-api-revoke-sub-3fca8eb6-sj67k\" (UID: \"ff93bdda-9ad2-4fbb-b36b-107aebce350d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fca8eb6-sj67k" Nov 03 02:10:44.085343 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:44.085305 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff93bdda-9ad2-4fbb-b36b-107aebce350d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fca8eb6-sj67k\" (UID: \"ff93bdda-9ad2-4fbb-b36b-107aebce350d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fca8eb6-sj67k" Nov 03 02:10:44.114103 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:44.114073 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:10:44.114286 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:44.114167 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:10:44.114515 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:44.114488 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:10:44.114662 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:44.114554 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:10:44.152394 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:44.149911 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fca8eb6-sj67k"] Nov 03 02:10:44.186937 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:44.186905 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2vmd\" (UniqueName: \"kubernetes.io/projected/ff93bdda-9ad2-4fbb-b36b-107aebce350d-kube-api-access-t2vmd\") pod \"maas-api-revoke-sub-3fca8eb6-sj67k\" (UID: \"ff93bdda-9ad2-4fbb-b36b-107aebce350d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fca8eb6-sj67k" Nov 03 02:10:44.187125 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:44.186959 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff93bdda-9ad2-4fbb-b36b-107aebce350d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fca8eb6-sj67k\" (UID: \"ff93bdda-9ad2-4fbb-b36b-107aebce350d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fca8eb6-sj67k" Nov 03 02:10:44.187125 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:44.187016 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff93bdda-9ad2-4fbb-b36b-107aebce350d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fca8eb6-sj67k\" (UID: \"ff93bdda-9ad2-4fbb-b36b-107aebce350d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fca8eb6-sj67k" Nov 03 02:10:44.187808 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:44.187783 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff93bdda-9ad2-4fbb-b36b-107aebce350d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fca8eb6-sj67k\" (UID: \"ff93bdda-9ad2-4fbb-b36b-107aebce350d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fca8eb6-sj67k" Nov 03 02:10:44.189543 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:44.189516 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff93bdda-9ad2-4fbb-b36b-107aebce350d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fca8eb6-sj67k\" (UID: \"ff93bdda-9ad2-4fbb-b36b-107aebce350d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fca8eb6-sj67k" Nov 03 02:10:44.240648 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:44.240618 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2vmd\" (UniqueName: \"kubernetes.io/projected/ff93bdda-9ad2-4fbb-b36b-107aebce350d-kube-api-access-t2vmd\") pod \"maas-api-revoke-sub-3fca8eb6-sj67k\" (UID: \"ff93bdda-9ad2-4fbb-b36b-107aebce350d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fca8eb6-sj67k" Nov 03 02:10:44.380259 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:44.380169 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fca8eb6-sj67k" Nov 03 02:10:44.758654 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:44.758474 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fca8eb6-sj67k"] Nov 03 02:10:44.763679 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:10:44.763648 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff93bdda_9ad2_4fbb_b36b_107aebce350d.slice/crio-636dc15323e0519ab2672396ffcde7f0f0775ea09fb3b327aee4d554556e2167 WatchSource:0}: Error finding container 636dc15323e0519ab2672396ffcde7f0f0775ea09fb3b327aee4d554556e2167: Status 404 returned error can't find the container with id 636dc15323e0519ab2672396ffcde7f0f0775ea09fb3b327aee4d554556e2167 Nov 03 02:10:44.783545 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:44.783507 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fca8eb6-sj67k" event={"ID":"ff93bdda-9ad2-4fbb-b36b-107aebce350d","Type":"ContainerStarted","Data":"636dc15323e0519ab2672396ffcde7f0f0775ea09fb3b327aee4d554556e2167"} Nov 03 02:10:45.805724 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:45.805687 5605 generic.go:356] "Generic (PLEG): container finished" podID="ff93bdda-9ad2-4fbb-b36b-107aebce350d" containerID="75bbffff08316585c934d7f4467f64d709e2fa387d6343000ed29f280529d4d9" exitCode=0 Nov 03 02:10:45.807048 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:45.806531 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fca8eb6-sj67k" event={"ID":"ff93bdda-9ad2-4fbb-b36b-107aebce350d","Type":"ContainerDied","Data":"75bbffff08316585c934d7f4467f64d709e2fa387d6343000ed29f280529d4d9"} Nov 03 02:10:46.543472 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:46.543436 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr" Nov 03 02:10:46.563329 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:46.563290 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr" Nov 03 02:10:46.827056 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:46.827025 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr" Nov 03 02:10:47.010821 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:47.010792 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fca8eb6-sj67k" Nov 03 02:10:47.051165 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:47.051129 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff93bdda-9ad2-4fbb-b36b-107aebce350d-kube-api-access-t2vmd\" (UniqueName: \"kubernetes.io/projected/ff93bdda-9ad2-4fbb-b36b-107aebce350d-kube-api-access-t2vmd\") pod \"ff93bdda-9ad2-4fbb-b36b-107aebce350d\" (UID: \"ff93bdda-9ad2-4fbb-b36b-107aebce350d\") " Nov 03 02:10:47.051395 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:47.051206 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff93bdda-9ad2-4fbb-b36b-107aebce350d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff93bdda-9ad2-4fbb-b36b-107aebce350d-maas-api-openshift-ca\") pod \"ff93bdda-9ad2-4fbb-b36b-107aebce350d\" (UID: \"ff93bdda-9ad2-4fbb-b36b-107aebce350d\") " Nov 03 02:10:47.051395 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:47.051282 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff93bdda-9ad2-4fbb-b36b-107aebce350d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff93bdda-9ad2-4fbb-b36b-107aebce350d-maas-api-xks-ca\") pod \"ff93bdda-9ad2-4fbb-b36b-107aebce350d\" (UID: \"ff93bdda-9ad2-4fbb-b36b-107aebce350d\") " Nov 03 02:10:47.051825 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:47.051792 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff93bdda-9ad2-4fbb-b36b-107aebce350d-maas-api-openshift-ca" pod "ff93bdda-9ad2-4fbb-b36b-107aebce350d" (UID: "ff93bdda-9ad2-4fbb-b36b-107aebce350d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:10:47.053686 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:47.053662 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff93bdda-9ad2-4fbb-b36b-107aebce350d-kube-api-access-t2vmd" pod "ff93bdda-9ad2-4fbb-b36b-107aebce350d" (UID: "ff93bdda-9ad2-4fbb-b36b-107aebce350d"). InnerVolumeSpecName "kube-api-access-t2vmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:10:47.053804 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:47.053717 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff93bdda-9ad2-4fbb-b36b-107aebce350d-maas-api-xks-ca" pod "ff93bdda-9ad2-4fbb-b36b-107aebce350d" (UID: "ff93bdda-9ad2-4fbb-b36b-107aebce350d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:10:47.152584 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:47.152454 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2vmd\" (UniqueName: \"kubernetes.io/projected/ff93bdda-9ad2-4fbb-b36b-107aebce350d-kube-api-access-t2vmd\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:47.152584 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:47.152489 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff93bdda-9ad2-4fbb-b36b-107aebce350d-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:47.152584 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:47.152499 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff93bdda-9ad2-4fbb-b36b-107aebce350d-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:47.821843 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:47.821816 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fca8eb6-sj67k" Nov 03 02:10:47.822014 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:47.821839 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fca8eb6-sj67k" event={"ID":"ff93bdda-9ad2-4fbb-b36b-107aebce350d","Type":"ContainerDied","Data":"636dc15323e0519ab2672396ffcde7f0f0775ea09fb3b327aee4d554556e2167"} Nov 03 02:10:47.822014 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:47.821916 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="636dc15323e0519ab2672396ffcde7f0f0775ea09fb3b327aee4d554556e2167" Nov 03 02:10:53.912146 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:10:53.912108 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:10:54.101986 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:54.101949 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:10:55.097393 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:55.055242 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-641c7353-fwvxm"] Nov 03 02:10:55.122976 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:55.122912 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-641c7353-fwvxm"] Nov 03 02:10:55.830522 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:55.829451 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a0e72e/simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq" podUID="1901f57b-279b-4cbf-bf0c-054abc5786d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:55.982535 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:55.981462 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-521f1f/premium-w2-521f1f-kserve-7578fb7957-89wfb" podUID="3d988573-d8b1-43ba-a423-51a63ca12337" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Nov 03 02:10:56.154561 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:56.154454 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f7d9bb/simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls" podUID="9173364c-14ba-4287-8587-45f25118e3e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Nov 03 02:10:56.526544 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:56.526437 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0e38c2b-jdbqx"] Nov 03 02:10:56.527526 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:56.527496 5605 cpu_manager.go:414] "containerMap: removing container" podUID="ff93bdda-9ad2-4fbb-b36b-107aebce350d" containerName="revoke-keys" Nov 03 02:10:56.527702 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:56.527686 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff93bdda-9ad2-4fbb-b36b-107aebce350d" containerName="revoke-keys" Nov 03 02:10:56.527956 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:56.527939 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff93bdda-9ad2-4fbb-b36b-107aebce350d" containerName="revoke-keys" Nov 03 02:10:56.550384 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:56.549979 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0e38c2b-jdbqx" Nov 03 02:10:56.590457 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:56.573881 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/171615bb-573d-4b45-8566-dd2a44262a38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0e38c2b-jdbqx\" (UID: \"171615bb-573d-4b45-8566-dd2a44262a38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0e38c2b-jdbqx" Nov 03 02:10:56.590457 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:56.573953 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvxdz\" (UniqueName: \"kubernetes.io/projected/171615bb-573d-4b45-8566-dd2a44262a38-kube-api-access-zvxdz\") pod \"maas-api-revoke-sub-a0e38c2b-jdbqx\" (UID: \"171615bb-573d-4b45-8566-dd2a44262a38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0e38c2b-jdbqx" Nov 03 02:10:56.590457 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:56.574053 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/171615bb-573d-4b45-8566-dd2a44262a38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0e38c2b-jdbqx\" (UID: \"171615bb-573d-4b45-8566-dd2a44262a38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0e38c2b-jdbqx" Nov 03 02:10:56.590457 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:56.576437 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0e38c2b-jdbqx"] Nov 03 02:10:56.719761 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:56.719723 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:10:56.720116 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:56.720097 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:10:56.720492 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:56.720458 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:10:56.721523 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:56.721496 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:10:56.752492 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:56.722180 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1bccbd/distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg" podUID="5a71cdee-425e-4787-9217-5fb6b97e232b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:56.753061 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:56.724294 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/171615bb-573d-4b45-8566-dd2a44262a38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0e38c2b-jdbqx\" (UID: \"171615bb-573d-4b45-8566-dd2a44262a38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0e38c2b-jdbqx" Nov 03 02:10:56.753450 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:56.753422 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/171615bb-573d-4b45-8566-dd2a44262a38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0e38c2b-jdbqx\" (UID: \"171615bb-573d-4b45-8566-dd2a44262a38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0e38c2b-jdbqx" Nov 03 02:10:56.753664 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:56.753643 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvxdz\" (UniqueName: \"kubernetes.io/projected/171615bb-573d-4b45-8566-dd2a44262a38-kube-api-access-zvxdz\") pod \"maas-api-revoke-sub-a0e38c2b-jdbqx\" (UID: \"171615bb-573d-4b45-8566-dd2a44262a38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0e38c2b-jdbqx" Nov 03 02:10:56.769062 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:56.769029 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/171615bb-573d-4b45-8566-dd2a44262a38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0e38c2b-jdbqx\" (UID: \"171615bb-573d-4b45-8566-dd2a44262a38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0e38c2b-jdbqx" Nov 03 02:10:56.831440 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:56.801733 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5" podUID="fffc3fbb-097a-4a5d-82c8-b1e221d0e710" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:56.831440 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:56.826257 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr" podUID="8282371b-c42d-43ab-a5c9-e07f2464dcdc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:56.831671 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:56.831428 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a0e72e/simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq" podUID="1901f57b-279b-4cbf-bf0c-054abc5786d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:56.831801 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:56.831780 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a0e72e/simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq" Nov 03 02:10:56.832151 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:56.832121 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/171615bb-573d-4b45-8566-dd2a44262a38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0e38c2b-jdbqx\" (UID: \"171615bb-573d-4b45-8566-dd2a44262a38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0e38c2b-jdbqx" Nov 03 02:10:56.908379 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:56.908298 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvxdz\" (UniqueName: \"kubernetes.io/projected/171615bb-573d-4b45-8566-dd2a44262a38-kube-api-access-zvxdz\") pod \"maas-api-revoke-sub-a0e38c2b-jdbqx\" (UID: \"171615bb-573d-4b45-8566-dd2a44262a38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0e38c2b-jdbqx" Nov 03 02:10:56.932383 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:56.915833 5605 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:10:56.932383 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:56.915896 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c" podUID="69d57c7a-8495-4d67-9547-dfe082939941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:56.933442 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:56.933393 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0e38c2b-jdbqx" Nov 03 02:10:56.998156 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:56.984889 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-521f1f/premium-w2-521f1f-kserve-7578fb7957-89wfb" podUID="3d988573-d8b1-43ba-a423-51a63ca12337" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:57.013719 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:57.013667 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f01f063e-88ed-4dcd-850a-4ac3723f7472" path="/var/lib/kubelet/pods/f01f063e-88ed-4dcd-850a-4ac3723f7472/volumes" Nov 03 02:10:57.014522 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:57.014502 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-521f1f/premium-w2-521f1f-kserve-7578fb7957-89wfb" Nov 03 02:10:57.114268 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:57.109497 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a0e72e/premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv" podUID="a9b9af44-ab6d-4caf-89af-c986ea25b4c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:57.174417 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:57.160034 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f7d9bb/simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls" podUID="9173364c-14ba-4287-8587-45f25118e3e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:57.174417 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:57.160428 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f7d9bb/simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls" Nov 03 02:10:57.199890 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:57.199849 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1bccbd/distinct-w1-1bccbd-kserve-5dfc85555f-tkvg6" podUID="dd4a9d44-c63e-4f67-bcac-a4f98ecf1796" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 03 02:10:57.784755 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:57.776667 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1bccbd/distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg" podUID="5a71cdee-425e-4787-9217-5fb6b97e232b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:57.784755 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:57.777312 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1bccbd/distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg" Nov 03 02:10:57.808624 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:57.808110 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a0e72e/simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq" Nov 03 02:10:57.814324 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:57.813869 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-521f1f/premium-w2-521f1f-kserve-7578fb7957-89wfb" Nov 03 02:10:57.872585 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:57.872537 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr" podUID="8282371b-c42d-43ab-a5c9-e07f2464dcdc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:57.896024 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:57.895994 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr" Nov 03 02:10:57.916378 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:57.912664 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1bccbd/distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg" Nov 03 02:10:57.965443 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:57.956666 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f7d9bb/simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls" Nov 03 02:10:57.993195 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:57.992521 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr" Nov 03 02:10:58.218641 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:58.218553 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0e38c2b-jdbqx"] Nov 03 02:10:58.223676 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:10:58.223647 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod171615bb_573d_4b45_8566_dd2a44262a38.slice/crio-128d47523d4adac70226285b25abcef272885726df4667a702232ac4a4a52baa WatchSource:0}: Error finding container 128d47523d4adac70226285b25abcef272885726df4667a702232ac4a4a52baa: Status 404 returned error can't find the container with id 128d47523d4adac70226285b25abcef272885726df4667a702232ac4a4a52baa Nov 03 02:10:59.118650 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:59.118613 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0e38c2b-jdbqx" event={"ID":"171615bb-573d-4b45-8566-dd2a44262a38","Type":"ContainerStarted","Data":"3166cd7d006757d5590cf65dbcbf161b28fa36e9bb60b3c6bb68da5f7531f81f"} Nov 03 02:10:59.118789 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:10:59.118678 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0e38c2b-jdbqx" event={"ID":"171615bb-573d-4b45-8566-dd2a44262a38","Type":"ContainerStarted","Data":"128d47523d4adac70226285b25abcef272885726df4667a702232ac4a4a52baa"} Nov 03 02:11:00.132260 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.132210 5605 generic.go:356] "Generic (PLEG): container finished" podID="171615bb-573d-4b45-8566-dd2a44262a38" containerID="3166cd7d006757d5590cf65dbcbf161b28fa36e9bb60b3c6bb68da5f7531f81f" exitCode=0 Nov 03 02:11:00.132793 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.132304 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0e38c2b-jdbqx" event={"ID":"171615bb-573d-4b45-8566-dd2a44262a38","Type":"ContainerDied","Data":"3166cd7d006757d5590cf65dbcbf161b28fa36e9bb60b3c6bb68da5f7531f81f"} Nov 03 02:11:00.275394 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.275347 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0e38c2b-jdbqx" Nov 03 02:11:00.327942 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.327909 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/171615bb-573d-4b45-8566-dd2a44262a38-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/171615bb-573d-4b45-8566-dd2a44262a38-maas-api-xks-ca\") pod \"171615bb-573d-4b45-8566-dd2a44262a38\" (UID: \"171615bb-573d-4b45-8566-dd2a44262a38\") " Nov 03 02:11:00.328088 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.327952 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/171615bb-573d-4b45-8566-dd2a44262a38-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/171615bb-573d-4b45-8566-dd2a44262a38-maas-api-openshift-ca\") pod \"171615bb-573d-4b45-8566-dd2a44262a38\" (UID: \"171615bb-573d-4b45-8566-dd2a44262a38\") " Nov 03 02:11:00.328088 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.328016 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/171615bb-573d-4b45-8566-dd2a44262a38-kube-api-access-zvxdz\" (UniqueName: \"kubernetes.io/projected/171615bb-573d-4b45-8566-dd2a44262a38-kube-api-access-zvxdz\") pod \"171615bb-573d-4b45-8566-dd2a44262a38\" (UID: \"171615bb-573d-4b45-8566-dd2a44262a38\") " Nov 03 02:11:00.328412 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.328336 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/171615bb-573d-4b45-8566-dd2a44262a38-maas-api-openshift-ca" pod "171615bb-573d-4b45-8566-dd2a44262a38" (UID: "171615bb-573d-4b45-8566-dd2a44262a38"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:11:00.328530 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.328510 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/171615bb-573d-4b45-8566-dd2a44262a38-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:11:00.329989 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.329963 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/171615bb-573d-4b45-8566-dd2a44262a38-maas-api-xks-ca" pod "171615bb-573d-4b45-8566-dd2a44262a38" (UID: "171615bb-573d-4b45-8566-dd2a44262a38"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:11:00.330099 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.330080 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/171615bb-573d-4b45-8566-dd2a44262a38-kube-api-access-zvxdz" pod "171615bb-573d-4b45-8566-dd2a44262a38" (UID: "171615bb-573d-4b45-8566-dd2a44262a38"). InnerVolumeSpecName "kube-api-access-zvxdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:11:00.429486 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.429409 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvxdz\" (UniqueName: \"kubernetes.io/projected/171615bb-573d-4b45-8566-dd2a44262a38-kube-api-access-zvxdz\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:11:00.429486 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.429438 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/171615bb-573d-4b45-8566-dd2a44262a38-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:11:00.522556 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.522523 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2"] Nov 03 02:11:00.522983 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.522970 5605 cpu_manager.go:414] "containerMap: removing container" podUID="171615bb-573d-4b45-8566-dd2a44262a38" containerName="revoke-keys" Nov 03 02:11:00.523025 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.522984 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="171615bb-573d-4b45-8566-dd2a44262a38" containerName="revoke-keys" Nov 03 02:11:00.523079 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.523070 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="171615bb-573d-4b45-8566-dd2a44262a38" containerName="revoke-keys" Nov 03 02:11:00.526324 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.526303 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2" Nov 03 02:11:00.549064 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.549031 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr"] Nov 03 02:11:00.552529 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.552511 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr" Nov 03 02:11:00.580501 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.580476 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2"] Nov 03 02:11:00.631206 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.631170 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae6b294b-6d5b-4bd7-820b-7e179011a2f8-istio-podinfo\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2\" (UID: \"ae6b294b-6d5b-4bd7-820b-7e179011a2f8\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2" Nov 03 02:11:00.631403 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.631215 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae6b294b-6d5b-4bd7-820b-7e179011a2f8-workload-certs\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2\" (UID: \"ae6b294b-6d5b-4bd7-820b-7e179011a2f8\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2" Nov 03 02:11:00.631403 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.631235 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3b0cfb1-29ab-4570-81b6-4b55627e8067-istio-podinfo\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr\" (UID: \"e3b0cfb1-29ab-4570-81b6-4b55627e8067\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr" Nov 03 02:11:00.631403 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.631297 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6b294b-6d5b-4bd7-820b-7e179011a2f8-credential-socket\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2\" (UID: \"ae6b294b-6d5b-4bd7-820b-7e179011a2f8\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2" Nov 03 02:11:00.631403 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.631323 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b0cfb1-29ab-4570-81b6-4b55627e8067-workload-socket\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr\" (UID: \"e3b0cfb1-29ab-4570-81b6-4b55627e8067\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr" Nov 03 02:11:00.631403 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.631383 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae6b294b-6d5b-4bd7-820b-7e179011a2f8-istio-data\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2\" (UID: \"ae6b294b-6d5b-4bd7-820b-7e179011a2f8\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2" Nov 03 02:11:00.631693 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.631413 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae6b294b-6d5b-4bd7-820b-7e179011a2f8-istio-token\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2\" (UID: \"ae6b294b-6d5b-4bd7-820b-7e179011a2f8\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2" Nov 03 02:11:00.631693 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.631446 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae6b294b-6d5b-4bd7-820b-7e179011a2f8-istio-envoy\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2\" (UID: \"ae6b294b-6d5b-4bd7-820b-7e179011a2f8\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2" Nov 03 02:11:00.631693 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.631482 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b0cfb1-29ab-4570-81b6-4b55627e8067-credential-socket\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr\" (UID: \"e3b0cfb1-29ab-4570-81b6-4b55627e8067\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr" Nov 03 02:11:00.631693 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.631510 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3b0cfb1-29ab-4570-81b6-4b55627e8067-istiod-ca-cert\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr\" (UID: \"e3b0cfb1-29ab-4570-81b6-4b55627e8067\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr" Nov 03 02:11:00.631693 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.631568 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llmt4\" (UniqueName: \"kubernetes.io/projected/ae6b294b-6d5b-4bd7-820b-7e179011a2f8-kube-api-access-llmt4\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2\" (UID: \"ae6b294b-6d5b-4bd7-820b-7e179011a2f8\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2" Nov 03 02:11:00.631693 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.631591 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3b0cfb1-29ab-4570-81b6-4b55627e8067-istio-data\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr\" (UID: \"e3b0cfb1-29ab-4570-81b6-4b55627e8067\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr" Nov 03 02:11:00.631693 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.631633 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3b0cfb1-29ab-4570-81b6-4b55627e8067-workload-certs\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr\" (UID: \"e3b0cfb1-29ab-4570-81b6-4b55627e8067\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr" Nov 03 02:11:00.631693 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.631678 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6b294b-6d5b-4bd7-820b-7e179011a2f8-workload-socket\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2\" (UID: \"ae6b294b-6d5b-4bd7-820b-7e179011a2f8\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2" Nov 03 02:11:00.632096 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.631729 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3b0cfb1-29ab-4570-81b6-4b55627e8067-istio-envoy\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr\" (UID: \"e3b0cfb1-29ab-4570-81b6-4b55627e8067\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr" Nov 03 02:11:00.632096 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.631753 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3b0cfb1-29ab-4570-81b6-4b55627e8067-istio-token\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr\" (UID: \"e3b0cfb1-29ab-4570-81b6-4b55627e8067\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr" Nov 03 02:11:00.632096 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.631781 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae6b294b-6d5b-4bd7-820b-7e179011a2f8-istiod-ca-cert\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2\" (UID: \"ae6b294b-6d5b-4bd7-820b-7e179011a2f8\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2" Nov 03 02:11:00.632096 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.631805 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7ctj\" (UniqueName: \"kubernetes.io/projected/e3b0cfb1-29ab-4570-81b6-4b55627e8067-kube-api-access-h7ctj\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr\" (UID: \"e3b0cfb1-29ab-4570-81b6-4b55627e8067\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr" Nov 03 02:11:00.645326 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.645296 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr"] Nov 03 02:11:00.741428 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.734131 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6b294b-6d5b-4bd7-820b-7e179011a2f8-credential-socket\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2\" (UID: \"ae6b294b-6d5b-4bd7-820b-7e179011a2f8\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2" Nov 03 02:11:00.741428 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.734217 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6b294b-6d5b-4bd7-820b-7e179011a2f8-credential-socket\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2\" (UID: \"ae6b294b-6d5b-4bd7-820b-7e179011a2f8\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2" Nov 03 02:11:00.741428 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.734291 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b0cfb1-29ab-4570-81b6-4b55627e8067-workload-socket\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr\" (UID: \"e3b0cfb1-29ab-4570-81b6-4b55627e8067\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr" Nov 03 02:11:00.741428 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.739340 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b0cfb1-29ab-4570-81b6-4b55627e8067-workload-socket\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr\" (UID: \"e3b0cfb1-29ab-4570-81b6-4b55627e8067\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr" Nov 03 02:11:00.741428 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.739572 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae6b294b-6d5b-4bd7-820b-7e179011a2f8-istio-data\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2\" (UID: \"ae6b294b-6d5b-4bd7-820b-7e179011a2f8\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2" Nov 03 02:11:00.741428 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.739635 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae6b294b-6d5b-4bd7-820b-7e179011a2f8-istio-data\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2\" (UID: \"ae6b294b-6d5b-4bd7-820b-7e179011a2f8\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2" Nov 03 02:11:00.741428 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.739738 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae6b294b-6d5b-4bd7-820b-7e179011a2f8-istio-token\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2\" (UID: \"ae6b294b-6d5b-4bd7-820b-7e179011a2f8\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2" Nov 03 02:11:00.741428 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.739777 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae6b294b-6d5b-4bd7-820b-7e179011a2f8-istio-envoy\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2\" (UID: \"ae6b294b-6d5b-4bd7-820b-7e179011a2f8\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2" Nov 03 02:11:00.741428 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.739819 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b0cfb1-29ab-4570-81b6-4b55627e8067-credential-socket\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr\" (UID: \"e3b0cfb1-29ab-4570-81b6-4b55627e8067\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr" Nov 03 02:11:00.741428 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.739842 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3b0cfb1-29ab-4570-81b6-4b55627e8067-istiod-ca-cert\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr\" (UID: \"e3b0cfb1-29ab-4570-81b6-4b55627e8067\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr" Nov 03 02:11:00.755145 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.755108 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b0cfb1-29ab-4570-81b6-4b55627e8067-credential-socket\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr\" (UID: \"e3b0cfb1-29ab-4570-81b6-4b55627e8067\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr" Nov 03 02:11:00.755340 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.755235 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llmt4\" (UniqueName: \"kubernetes.io/projected/ae6b294b-6d5b-4bd7-820b-7e179011a2f8-kube-api-access-llmt4\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2\" (UID: \"ae6b294b-6d5b-4bd7-820b-7e179011a2f8\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2" Nov 03 02:11:00.755340 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.755267 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3b0cfb1-29ab-4570-81b6-4b55627e8067-istio-data\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr\" (UID: \"e3b0cfb1-29ab-4570-81b6-4b55627e8067\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr" Nov 03 02:11:00.755340 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.755312 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3b0cfb1-29ab-4570-81b6-4b55627e8067-workload-certs\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr\" (UID: \"e3b0cfb1-29ab-4570-81b6-4b55627e8067\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr" Nov 03 02:11:00.755540 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.755372 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6b294b-6d5b-4bd7-820b-7e179011a2f8-workload-socket\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2\" (UID: \"ae6b294b-6d5b-4bd7-820b-7e179011a2f8\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2" Nov 03 02:11:00.755540 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.755477 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3b0cfb1-29ab-4570-81b6-4b55627e8067-istio-envoy\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr\" (UID: \"e3b0cfb1-29ab-4570-81b6-4b55627e8067\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr" Nov 03 02:11:00.755651 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.755537 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3b0cfb1-29ab-4570-81b6-4b55627e8067-istio-token\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr\" (UID: \"e3b0cfb1-29ab-4570-81b6-4b55627e8067\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr" Nov 03 02:11:00.755651 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.755596 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae6b294b-6d5b-4bd7-820b-7e179011a2f8-istiod-ca-cert\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2\" (UID: \"ae6b294b-6d5b-4bd7-820b-7e179011a2f8\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2" Nov 03 02:11:00.755651 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.755624 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7ctj\" (UniqueName: \"kubernetes.io/projected/e3b0cfb1-29ab-4570-81b6-4b55627e8067-kube-api-access-h7ctj\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr\" (UID: \"e3b0cfb1-29ab-4570-81b6-4b55627e8067\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr" Nov 03 02:11:00.757656 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.757502 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae6b294b-6d5b-4bd7-820b-7e179011a2f8-istio-podinfo\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2\" (UID: \"ae6b294b-6d5b-4bd7-820b-7e179011a2f8\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2" Nov 03 02:11:00.757656 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.757583 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae6b294b-6d5b-4bd7-820b-7e179011a2f8-workload-certs\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2\" (UID: \"ae6b294b-6d5b-4bd7-820b-7e179011a2f8\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2" Nov 03 02:11:00.757656 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.757613 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3b0cfb1-29ab-4570-81b6-4b55627e8067-istio-podinfo\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr\" (UID: \"e3b0cfb1-29ab-4570-81b6-4b55627e8067\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr" Nov 03 02:11:00.758235 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.758216 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae6b294b-6d5b-4bd7-820b-7e179011a2f8-istio-envoy\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2\" (UID: \"ae6b294b-6d5b-4bd7-820b-7e179011a2f8\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2" Nov 03 02:11:00.764555 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.761874 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3b0cfb1-29ab-4570-81b6-4b55627e8067-istio-data\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr\" (UID: \"e3b0cfb1-29ab-4570-81b6-4b55627e8067\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr" Nov 03 02:11:00.764555 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.762417 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3b0cfb1-29ab-4570-81b6-4b55627e8067-istio-envoy\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr\" (UID: \"e3b0cfb1-29ab-4570-81b6-4b55627e8067\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr" Nov 03 02:11:00.764555 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.762667 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3b0cfb1-29ab-4570-81b6-4b55627e8067-workload-certs\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr\" (UID: \"e3b0cfb1-29ab-4570-81b6-4b55627e8067\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr" Nov 03 02:11:00.764555 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.762979 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6b294b-6d5b-4bd7-820b-7e179011a2f8-workload-socket\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2\" (UID: \"ae6b294b-6d5b-4bd7-820b-7e179011a2f8\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2" Nov 03 02:11:00.764555 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.763421 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3b0cfb1-29ab-4570-81b6-4b55627e8067-istio-podinfo\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr\" (UID: \"e3b0cfb1-29ab-4570-81b6-4b55627e8067\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr" Nov 03 02:11:00.764555 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.764334 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae6b294b-6d5b-4bd7-820b-7e179011a2f8-istiod-ca-cert\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2\" (UID: \"ae6b294b-6d5b-4bd7-820b-7e179011a2f8\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2" Nov 03 02:11:00.764555 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.764494 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3b0cfb1-29ab-4570-81b6-4b55627e8067-istiod-ca-cert\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr\" (UID: \"e3b0cfb1-29ab-4570-81b6-4b55627e8067\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr" Nov 03 02:11:00.764940 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.764627 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae6b294b-6d5b-4bd7-820b-7e179011a2f8-workload-certs\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2\" (UID: \"ae6b294b-6d5b-4bd7-820b-7e179011a2f8\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2" Nov 03 02:11:00.766040 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.766021 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae6b294b-6d5b-4bd7-820b-7e179011a2f8-istio-podinfo\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2\" (UID: \"ae6b294b-6d5b-4bd7-820b-7e179011a2f8\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2" Nov 03 02:11:00.815201 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.815175 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7ctj\" (UniqueName: \"kubernetes.io/projected/e3b0cfb1-29ab-4570-81b6-4b55627e8067-kube-api-access-h7ctj\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr\" (UID: \"e3b0cfb1-29ab-4570-81b6-4b55627e8067\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr" Nov 03 02:11:00.815788 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.815765 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llmt4\" (UniqueName: \"kubernetes.io/projected/ae6b294b-6d5b-4bd7-820b-7e179011a2f8-kube-api-access-llmt4\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2\" (UID: \"ae6b294b-6d5b-4bd7-820b-7e179011a2f8\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2" Nov 03 02:11:00.816894 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.816871 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae6b294b-6d5b-4bd7-820b-7e179011a2f8-istio-token\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2\" (UID: \"ae6b294b-6d5b-4bd7-820b-7e179011a2f8\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2" Nov 03 02:11:00.817309 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.817291 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3b0cfb1-29ab-4570-81b6-4b55627e8067-istio-token\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr\" (UID: \"e3b0cfb1-29ab-4570-81b6-4b55627e8067\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr" Nov 03 02:11:00.839317 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.839292 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2" Nov 03 02:11:00.865628 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:00.865604 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr" Nov 03 02:11:01.145531 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:01.145499 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0e38c2b-jdbqx" Nov 03 02:11:01.145944 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:01.145497 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0e38c2b-jdbqx" event={"ID":"171615bb-573d-4b45-8566-dd2a44262a38","Type":"ContainerDied","Data":"128d47523d4adac70226285b25abcef272885726df4667a702232ac4a4a52baa"} Nov 03 02:11:01.145944 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:01.145671 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="128d47523d4adac70226285b25abcef272885726df4667a702232ac4a4a52baa" Nov 03 02:11:01.263816 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:01.263755 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr"] Nov 03 02:11:01.267405 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:01.267374 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2"] Nov 03 02:11:01.270416 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:11:01.270386 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3b0cfb1_29ab_4570_81b6_4b55627e8067.slice/crio-fa9236f574cfa889f563f7e4711012c26c896614ae89498852573a9f75699318 WatchSource:0}: Error finding container fa9236f574cfa889f563f7e4711012c26c896614ae89498852573a9f75699318: Status 404 returned error can't find the container with id fa9236f574cfa889f563f7e4711012c26c896614ae89498852573a9f75699318 Nov 03 02:11:01.272780 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:01.272748 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:11:01.272863 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:01.272813 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:11:01.272863 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:01.272842 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:11:01.311452 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:11:01.311423 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae6b294b_6d5b_4bd7_820b_7e179011a2f8.slice/crio-60a9404eec397b9e99c96cad05d507599a2520fa18f87246596015ee12df2ac2 WatchSource:0}: Error finding container 60a9404eec397b9e99c96cad05d507599a2520fa18f87246596015ee12df2ac2: Status 404 returned error can't find the container with id 60a9404eec397b9e99c96cad05d507599a2520fa18f87246596015ee12df2ac2 Nov 03 02:11:01.313840 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:01.313810 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:11:01.313934 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:01.313877 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:11:01.313934 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:01.313907 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:11:02.152301 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:02.152267 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr" event={"ID":"e3b0cfb1-29ab-4570-81b6-4b55627e8067","Type":"ContainerStarted","Data":"3257f498235885415411dd42ed8752bdbf810d415bb432c5d9d4c59dba99dfb4"} Nov 03 02:11:02.152803 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:02.152327 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr" event={"ID":"e3b0cfb1-29ab-4570-81b6-4b55627e8067","Type":"ContainerStarted","Data":"fa9236f574cfa889f563f7e4711012c26c896614ae89498852573a9f75699318"} Nov 03 02:11:02.153972 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:02.153943 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2" event={"ID":"ae6b294b-6d5b-4bd7-820b-7e179011a2f8","Type":"ContainerStarted","Data":"3c69dd3953d802332e6cac7a63d9989e286cd77454f05713df39eec6d02f426d"} Nov 03 02:11:02.154085 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:02.153999 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2" event={"ID":"ae6b294b-6d5b-4bd7-820b-7e179011a2f8","Type":"ContainerStarted","Data":"60a9404eec397b9e99c96cad05d507599a2520fa18f87246596015ee12df2ac2"} Nov 03 02:11:02.369385 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:02.369304 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr" podStartSLOduration=2.369284299 podStartE2EDuration="2.369284299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:11:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:11:02.319392672 +0000 UTC m=+1139.578590823" watchObservedRunningTime="2026-11-03 02:11:02.369284299 +0000 UTC m=+1139.628482454" Nov 03 02:11:02.431150 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:02.431047 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2" podStartSLOduration=2.431035019 podStartE2EDuration="2.431035019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:11:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:11:02.430067589 +0000 UTC m=+1139.689265743" watchObservedRunningTime="2026-11-03 02:11:02.431035019 +0000 UTC m=+1139.690233170" Nov 03 02:11:02.840421 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:02.840386 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2" Nov 03 02:11:02.841831 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:02.841809 5605 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 03 02:11:02.841899 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:02.841864 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2" podUID="ae6b294b-6d5b-4bd7-820b-7e179011a2f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 03 02:11:02.871755 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:02.871720 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr" Nov 03 02:11:02.876514 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:02.876478 5605 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 03 02:11:02.876767 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:02.876732 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr" podUID="e3b0cfb1-29ab-4570-81b6-4b55627e8067" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 03 02:11:03.840303 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:03.840264 5605 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 03 02:11:03.840724 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:03.840316 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2" podUID="ae6b294b-6d5b-4bd7-820b-7e179011a2f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 03 02:11:03.870380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:03.866655 5605 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 03 02:11:03.870380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:03.866710 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr" podUID="e3b0cfb1-29ab-4570-81b6-4b55627e8067" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 03 02:11:04.839918 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:04.839880 5605 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 03 02:11:04.840097 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:04.839936 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2" podUID="ae6b294b-6d5b-4bd7-820b-7e179011a2f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 03 02:11:04.876382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:04.872808 5605 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 03 02:11:04.876382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:04.872896 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr" podUID="e3b0cfb1-29ab-4570-81b6-4b55627e8067" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 03 02:11:05.840495 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:05.840459 5605 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 03 02:11:05.840668 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:05.840511 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2" podUID="ae6b294b-6d5b-4bd7-820b-7e179011a2f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 03 02:11:05.875715 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:05.875673 5605 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 03 02:11:05.875985 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:05.875957 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr" podUID="e3b0cfb1-29ab-4570-81b6-4b55627e8067" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 03 02:11:06.840267 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:06.840233 5605 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 03 02:11:06.840715 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:06.840292 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2" podUID="ae6b294b-6d5b-4bd7-820b-7e179011a2f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 03 02:11:06.879376 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:06.878896 5605 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 03 02:11:06.879376 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:06.878971 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr" podUID="e3b0cfb1-29ab-4570-81b6-4b55627e8067" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 03 02:11:07.839726 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:07.839691 5605 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 03 02:11:07.839922 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:07.839743 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2" podUID="ae6b294b-6d5b-4bd7-820b-7e179011a2f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 03 02:11:07.866941 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:07.866911 5605 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 03 02:11:07.867290 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:07.866978 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr" podUID="e3b0cfb1-29ab-4570-81b6-4b55627e8067" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 03 02:11:08.850015 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:08.849982 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2" Nov 03 02:11:08.850335 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:08.850314 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2" Nov 03 02:11:08.850950 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:08.850929 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2" Nov 03 02:11:08.870266 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:08.870245 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr" Nov 03 02:11:08.870611 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:08.870578 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr" Nov 03 02:11:08.871287 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:08.871269 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr" Nov 03 02:11:12.872692 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:12.872659 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n"] Nov 03 02:11:12.877563 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:12.877541 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n" Nov 03 02:11:12.969298 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:12.969263 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58ecd8f7-099d-401b-a63a-6c4e79e56077-istio-podinfo\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n\" (UID: \"58ecd8f7-099d-401b-a63a-6c4e79e56077\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n" Nov 03 02:11:12.969482 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:12.969316 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58ecd8f7-099d-401b-a63a-6c4e79e56077-workload-certs\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n\" (UID: \"58ecd8f7-099d-401b-a63a-6c4e79e56077\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n" Nov 03 02:11:12.969546 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:12.969473 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjzg5\" (UniqueName: \"kubernetes.io/projected/58ecd8f7-099d-401b-a63a-6c4e79e56077-kube-api-access-kjzg5\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n\" (UID: \"58ecd8f7-099d-401b-a63a-6c4e79e56077\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n" Nov 03 02:11:12.969546 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:12.969507 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58ecd8f7-099d-401b-a63a-6c4e79e56077-istiod-ca-cert\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n\" (UID: \"58ecd8f7-099d-401b-a63a-6c4e79e56077\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n" Nov 03 02:11:12.969628 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:12.969574 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58ecd8f7-099d-401b-a63a-6c4e79e56077-istio-token\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n\" (UID: \"58ecd8f7-099d-401b-a63a-6c4e79e56077\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n" Nov 03 02:11:12.969695 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:12.969669 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58ecd8f7-099d-401b-a63a-6c4e79e56077-istio-data\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n\" (UID: \"58ecd8f7-099d-401b-a63a-6c4e79e56077\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n" Nov 03 02:11:12.969761 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:12.969744 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58ecd8f7-099d-401b-a63a-6c4e79e56077-istio-envoy\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n\" (UID: \"58ecd8f7-099d-401b-a63a-6c4e79e56077\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n" Nov 03 02:11:12.969806 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:12.969774 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ecd8f7-099d-401b-a63a-6c4e79e56077-credential-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n\" (UID: \"58ecd8f7-099d-401b-a63a-6c4e79e56077\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n" Nov 03 02:11:12.969806 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:12.969797 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ecd8f7-099d-401b-a63a-6c4e79e56077-workload-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n\" (UID: \"58ecd8f7-099d-401b-a63a-6c4e79e56077\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n" Nov 03 02:11:13.066195 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:13.066163 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n"] Nov 03 02:11:13.070645 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:13.070613 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58ecd8f7-099d-401b-a63a-6c4e79e56077-istio-envoy\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n\" (UID: \"58ecd8f7-099d-401b-a63a-6c4e79e56077\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n" Nov 03 02:11:13.070759 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:13.070653 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ecd8f7-099d-401b-a63a-6c4e79e56077-credential-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n\" (UID: \"58ecd8f7-099d-401b-a63a-6c4e79e56077\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n" Nov 03 02:11:13.070759 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:13.070674 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ecd8f7-099d-401b-a63a-6c4e79e56077-workload-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n\" (UID: \"58ecd8f7-099d-401b-a63a-6c4e79e56077\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n" Nov 03 02:11:13.070759 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:13.070715 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58ecd8f7-099d-401b-a63a-6c4e79e56077-istio-podinfo\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n\" (UID: \"58ecd8f7-099d-401b-a63a-6c4e79e56077\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n" Nov 03 02:11:13.070759 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:13.070738 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58ecd8f7-099d-401b-a63a-6c4e79e56077-workload-certs\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n\" (UID: \"58ecd8f7-099d-401b-a63a-6c4e79e56077\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n" Nov 03 02:11:13.070976 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:13.070807 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjzg5\" (UniqueName: \"kubernetes.io/projected/58ecd8f7-099d-401b-a63a-6c4e79e56077-kube-api-access-kjzg5\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n\" (UID: \"58ecd8f7-099d-401b-a63a-6c4e79e56077\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n" Nov 03 02:11:13.070976 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:13.070838 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58ecd8f7-099d-401b-a63a-6c4e79e56077-istiod-ca-cert\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n\" (UID: \"58ecd8f7-099d-401b-a63a-6c4e79e56077\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n" Nov 03 02:11:13.070976 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:13.070896 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58ecd8f7-099d-401b-a63a-6c4e79e56077-istio-token\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n\" (UID: \"58ecd8f7-099d-401b-a63a-6c4e79e56077\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n" Nov 03 02:11:13.071130 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:13.070993 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58ecd8f7-099d-401b-a63a-6c4e79e56077-istio-data\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n\" (UID: \"58ecd8f7-099d-401b-a63a-6c4e79e56077\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n" Nov 03 02:11:13.071130 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:13.071094 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ecd8f7-099d-401b-a63a-6c4e79e56077-credential-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n\" (UID: \"58ecd8f7-099d-401b-a63a-6c4e79e56077\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n" Nov 03 02:11:13.071592 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:13.071564 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58ecd8f7-099d-401b-a63a-6c4e79e56077-istiod-ca-cert\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n\" (UID: \"58ecd8f7-099d-401b-a63a-6c4e79e56077\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n" Nov 03 02:11:13.072624 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:13.072594 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58ecd8f7-099d-401b-a63a-6c4e79e56077-workload-certs\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n\" (UID: \"58ecd8f7-099d-401b-a63a-6c4e79e56077\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n" Nov 03 02:11:13.072824 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:13.072644 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ecd8f7-099d-401b-a63a-6c4e79e56077-workload-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n\" (UID: \"58ecd8f7-099d-401b-a63a-6c4e79e56077\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n" Nov 03 02:11:13.072824 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:13.072731 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58ecd8f7-099d-401b-a63a-6c4e79e56077-istio-data\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n\" (UID: \"58ecd8f7-099d-401b-a63a-6c4e79e56077\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n" Nov 03 02:11:13.073781 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:13.073754 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58ecd8f7-099d-401b-a63a-6c4e79e56077-istio-podinfo\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n\" (UID: \"58ecd8f7-099d-401b-a63a-6c4e79e56077\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n" Nov 03 02:11:13.076761 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:13.076738 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58ecd8f7-099d-401b-a63a-6c4e79e56077-istio-envoy\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n\" (UID: \"58ecd8f7-099d-401b-a63a-6c4e79e56077\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n" Nov 03 02:11:13.123971 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:13.123893 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58ecd8f7-099d-401b-a63a-6c4e79e56077-istio-token\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n\" (UID: \"58ecd8f7-099d-401b-a63a-6c4e79e56077\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n" Nov 03 02:11:13.123971 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:13.123933 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjzg5\" (UniqueName: \"kubernetes.io/projected/58ecd8f7-099d-401b-a63a-6c4e79e56077-kube-api-access-kjzg5\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n\" (UID: \"58ecd8f7-099d-401b-a63a-6c4e79e56077\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n" Nov 03 02:11:13.192212 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:13.192179 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n" Nov 03 02:11:13.615316 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:13.615278 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n"] Nov 03 02:11:13.658088 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:11:13.658056 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58ecd8f7_099d_401b_a63a_6c4e79e56077.slice/crio-13c28f50395a97493176a5f1d431b2331336a73b74c262cabad0f841218c3bf8 WatchSource:0}: Error finding container 13c28f50395a97493176a5f1d431b2331336a73b74c262cabad0f841218c3bf8: Status 404 returned error can't find the container with id 13c28f50395a97493176a5f1d431b2331336a73b74c262cabad0f841218c3bf8 Nov 03 02:11:13.660569 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:13.660532 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:11:13.660649 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:13.660608 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:11:13.660649 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:13.660638 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:11:14.284266 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:14.284224 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n" event={"ID":"58ecd8f7-099d-401b-a63a-6c4e79e56077","Type":"ContainerStarted","Data":"d37cf73f92722c2c7fc104b8c1c6285b5c68634bc03028bfbe49f62143b4694a"} Nov 03 02:11:14.284804 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:14.284290 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n" event={"ID":"58ecd8f7-099d-401b-a63a-6c4e79e56077","Type":"ContainerStarted","Data":"13c28f50395a97493176a5f1d431b2331336a73b74c262cabad0f841218c3bf8"} Nov 03 02:11:14.298867 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:14.298832 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e0624c5-fcdxk"] Nov 03 02:11:14.302806 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:14.302784 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0624c5-fcdxk" Nov 03 02:11:14.385402 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:14.385285 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e0624c5-fcdxk"] Nov 03 02:11:14.385974 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:14.385945 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:11:14.386126 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:14.386021 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:11:14.386211 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:14.386131 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:11:14.386333 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:14.386213 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:11:14.427801 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:14.427742 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n" podStartSLOduration=2.427724773 podStartE2EDuration="2.427724773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:11:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:11:14.401553037 +0000 UTC m=+1151.660751212" watchObservedRunningTime="2026-11-03 02:11:14.427724773 +0000 UTC m=+1151.686922927" Nov 03 02:11:14.487084 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:14.487039 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8772ad29-8e2b-4236-a07a-99878ef696cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e0624c5-fcdxk\" (UID: \"8772ad29-8e2b-4236-a07a-99878ef696cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0624c5-fcdxk" Nov 03 02:11:14.487264 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:14.487245 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff496\" (UniqueName: \"kubernetes.io/projected/8772ad29-8e2b-4236-a07a-99878ef696cc-kube-api-access-ff496\") pod \"maas-api-revoke-sub-8e0624c5-fcdxk\" (UID: \"8772ad29-8e2b-4236-a07a-99878ef696cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0624c5-fcdxk" Nov 03 02:11:14.487404 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:14.487323 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8772ad29-8e2b-4236-a07a-99878ef696cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e0624c5-fcdxk\" (UID: \"8772ad29-8e2b-4236-a07a-99878ef696cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0624c5-fcdxk" Nov 03 02:11:14.588113 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:14.588062 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff496\" (UniqueName: \"kubernetes.io/projected/8772ad29-8e2b-4236-a07a-99878ef696cc-kube-api-access-ff496\") pod \"maas-api-revoke-sub-8e0624c5-fcdxk\" (UID: \"8772ad29-8e2b-4236-a07a-99878ef696cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0624c5-fcdxk" Nov 03 02:11:14.588288 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:14.588170 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8772ad29-8e2b-4236-a07a-99878ef696cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e0624c5-fcdxk\" (UID: \"8772ad29-8e2b-4236-a07a-99878ef696cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0624c5-fcdxk" Nov 03 02:11:14.588288 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:14.588242 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8772ad29-8e2b-4236-a07a-99878ef696cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e0624c5-fcdxk\" (UID: \"8772ad29-8e2b-4236-a07a-99878ef696cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0624c5-fcdxk" Nov 03 02:11:14.589090 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:14.589059 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8772ad29-8e2b-4236-a07a-99878ef696cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e0624c5-fcdxk\" (UID: \"8772ad29-8e2b-4236-a07a-99878ef696cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0624c5-fcdxk" Nov 03 02:11:14.590800 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:14.590775 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8772ad29-8e2b-4236-a07a-99878ef696cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e0624c5-fcdxk\" (UID: \"8772ad29-8e2b-4236-a07a-99878ef696cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0624c5-fcdxk" Nov 03 02:11:14.641590 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:14.641558 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff496\" (UniqueName: \"kubernetes.io/projected/8772ad29-8e2b-4236-a07a-99878ef696cc-kube-api-access-ff496\") pod \"maas-api-revoke-sub-8e0624c5-fcdxk\" (UID: \"8772ad29-8e2b-4236-a07a-99878ef696cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0624c5-fcdxk" Nov 03 02:11:14.916633 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:14.916549 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0624c5-fcdxk" Nov 03 02:11:15.192757 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:15.192657 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n" Nov 03 02:11:15.194204 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:15.194169 5605 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 03 02:11:15.194336 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:15.194230 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n" podUID="58ecd8f7-099d-401b-a63a-6c4e79e56077" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 03 02:11:15.343525 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:15.343475 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e0624c5-fcdxk"] Nov 03 02:11:16.192888 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:16.192846 5605 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 03 02:11:16.193095 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:16.192913 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n" podUID="58ecd8f7-099d-401b-a63a-6c4e79e56077" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 03 02:11:16.315338 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:16.315299 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0624c5-fcdxk" event={"ID":"8772ad29-8e2b-4236-a07a-99878ef696cc","Type":"ContainerStarted","Data":"e7bde04ea44ea5f394d0662d3e1d5e4501e0672bb2c74557b5937265caa6642c"} Nov 03 02:11:16.315574 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:16.315392 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0624c5-fcdxk" event={"ID":"8772ad29-8e2b-4236-a07a-99878ef696cc","Type":"ContainerStarted","Data":"f03535182cdc3f08a4bf2ed8dfc41259f8ec51715ccb35882edd0cbed3999896"} Nov 03 02:11:16.434766 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:16.434705 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0624c5-fcdxk" podStartSLOduration=2.434690361 podStartE2EDuration="2.434690361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:11:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:11:16.431502815 +0000 UTC m=+1153.690700970" watchObservedRunningTime="2026-11-03 02:11:16.434690361 +0000 UTC m=+1153.693888514" Nov 03 02:11:17.192700 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:17.192661 5605 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 03 02:11:17.192870 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:17.192721 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n" podUID="58ecd8f7-099d-401b-a63a-6c4e79e56077" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 03 02:11:17.323499 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:17.323456 5605 generic.go:356] "Generic (PLEG): container finished" podID="8772ad29-8e2b-4236-a07a-99878ef696cc" containerID="e7bde04ea44ea5f394d0662d3e1d5e4501e0672bb2c74557b5937265caa6642c" exitCode=0 Nov 03 02:11:17.323725 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:17.323661 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0624c5-fcdxk" event={"ID":"8772ad29-8e2b-4236-a07a-99878ef696cc","Type":"ContainerDied","Data":"e7bde04ea44ea5f394d0662d3e1d5e4501e0672bb2c74557b5937265caa6642c"} Nov 03 02:11:18.192549 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:18.192512 5605 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 03 02:11:18.193093 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:18.192567 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n" podUID="58ecd8f7-099d-401b-a63a-6c4e79e56077" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 03 02:11:18.518285 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:18.518254 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0624c5-fcdxk" Nov 03 02:11:18.639162 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:18.639122 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8772ad29-8e2b-4236-a07a-99878ef696cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8772ad29-8e2b-4236-a07a-99878ef696cc-maas-api-openshift-ca\") pod \"8772ad29-8e2b-4236-a07a-99878ef696cc\" (UID: \"8772ad29-8e2b-4236-a07a-99878ef696cc\") " Nov 03 02:11:18.639348 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:18.639199 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8772ad29-8e2b-4236-a07a-99878ef696cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8772ad29-8e2b-4236-a07a-99878ef696cc-maas-api-xks-ca\") pod \"8772ad29-8e2b-4236-a07a-99878ef696cc\" (UID: \"8772ad29-8e2b-4236-a07a-99878ef696cc\") " Nov 03 02:11:18.639348 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:18.639230 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8772ad29-8e2b-4236-a07a-99878ef696cc-kube-api-access-ff496\" (UniqueName: \"kubernetes.io/projected/8772ad29-8e2b-4236-a07a-99878ef696cc-kube-api-access-ff496\") pod \"8772ad29-8e2b-4236-a07a-99878ef696cc\" (UID: \"8772ad29-8e2b-4236-a07a-99878ef696cc\") " Nov 03 02:11:18.639606 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:18.639578 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8772ad29-8e2b-4236-a07a-99878ef696cc-maas-api-openshift-ca" pod "8772ad29-8e2b-4236-a07a-99878ef696cc" (UID: "8772ad29-8e2b-4236-a07a-99878ef696cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:11:18.641612 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:18.641583 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8772ad29-8e2b-4236-a07a-99878ef696cc-maas-api-xks-ca" pod "8772ad29-8e2b-4236-a07a-99878ef696cc" (UID: "8772ad29-8e2b-4236-a07a-99878ef696cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:11:18.641876 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:18.641848 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8772ad29-8e2b-4236-a07a-99878ef696cc-kube-api-access-ff496" pod "8772ad29-8e2b-4236-a07a-99878ef696cc" (UID: "8772ad29-8e2b-4236-a07a-99878ef696cc"). InnerVolumeSpecName "kube-api-access-ff496". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:11:18.740836 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:18.740804 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8772ad29-8e2b-4236-a07a-99878ef696cc-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:11:18.740836 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:18.740834 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8772ad29-8e2b-4236-a07a-99878ef696cc-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:11:18.740836 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:18.740846 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ff496\" (UniqueName: \"kubernetes.io/projected/8772ad29-8e2b-4236-a07a-99878ef696cc-kube-api-access-ff496\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:11:19.193069 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:19.192989 5605 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 03 02:11:19.193069 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:19.193043 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n" podUID="58ecd8f7-099d-401b-a63a-6c4e79e56077" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 03 02:11:19.336372 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:19.336329 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0624c5-fcdxk" Nov 03 02:11:19.336372 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:19.336346 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0624c5-fcdxk" event={"ID":"8772ad29-8e2b-4236-a07a-99878ef696cc","Type":"ContainerDied","Data":"f03535182cdc3f08a4bf2ed8dfc41259f8ec51715ccb35882edd0cbed3999896"} Nov 03 02:11:19.336617 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:19.336420 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f03535182cdc3f08a4bf2ed8dfc41259f8ec51715ccb35882edd0cbed3999896" Nov 03 02:11:20.193047 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:20.193001 5605 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 03 02:11:20.193270 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:20.193079 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n" podUID="58ecd8f7-099d-401b-a63a-6c4e79e56077" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 03 02:11:21.745392 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:21.729472 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-521f1f/simulator-w2-521f1f-kserve-5f47b86f6-mgksf" podUID="6e1b7fda-0ade-4042-ad63-5115bc66b454" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:21.803561 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:21.802874 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5" podUID="fffc3fbb-097a-4a5d-82c8-b1e221d0e710" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 03 02:11:21.817525 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:21.817467 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-521f1f/premium-w2-521f1f-kserve-7578fb7957-89wfb" podUID="3d988573-d8b1-43ba-a423-51a63ca12337" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:21.818968 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:21.817836 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a0e72e/simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq" podUID="1901f57b-279b-4cbf-bf0c-054abc5786d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Nov 03 02:11:21.920227 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:21.920181 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1bccbd/distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg" podUID="5a71cdee-425e-4787-9217-5fb6b97e232b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:21.962818 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:21.962776 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f7d9bb/simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls" podUID="9173364c-14ba-4287-8587-45f25118e3e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:22.006643 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:22.006493 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr" podUID="8282371b-c42d-43ab-a5c9-e07f2464dcdc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:22.110843 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:22.110788 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a0e72e/premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv" podUID="a9b9af44-ab6d-4caf-89af-c986ea25b4c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Nov 03 02:11:22.200646 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:22.200594 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1bccbd/distinct-w1-1bccbd-kserve-5dfc85555f-tkvg6" podUID="dd4a9d44-c63e-4f67-bcac-a4f98ecf1796" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:22.217529 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:22.217485 5605 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:11:22.217817 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:22.217784 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n" podUID="58ecd8f7-099d-401b-a63a-6c4e79e56077" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:22.348455 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:22.288802 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n" Nov 03 02:11:22.348455 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:22.290161 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n" Nov 03 02:11:22.348455 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:22.308677 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n" Nov 03 02:11:22.748584 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:22.748472 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-521f1f/simulator-w2-521f1f-kserve-5f47b86f6-mgksf" podUID="6e1b7fda-0ade-4042-ad63-5115bc66b454" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:22.782414 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:22.757915 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-521f1f/simulator-w2-521f1f-kserve-5f47b86f6-mgksf" Nov 03 02:11:22.782414 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:22.757449 5605 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f7d9bb/simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls" podUID="9173364c-14ba-4287-8587-45f25118e3e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:22.941386 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:22.811459 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5" podUID="fffc3fbb-097a-4a5d-82c8-b1e221d0e710" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:22.941386 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:22.811951 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c7d758/distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5" Nov 03 02:11:22.941386 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:22.829476 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a0e72e/simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq" podUID="1901f57b-279b-4cbf-bf0c-054abc5786d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:22.941386 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:22.837484 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-521f1f/premium-w2-521f1f-kserve-7578fb7957-89wfb" podUID="3d988573-d8b1-43ba-a423-51a63ca12337" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Nov 03 02:11:22.941386 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:22.842457 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a0e72e/simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq" Nov 03 02:11:22.941386 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:22.842530 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-521f1f/premium-w2-521f1f-kserve-7578fb7957-89wfb" Nov 03 02:11:22.941386 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:22.929858 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1bccbd/distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg" podUID="5a71cdee-425e-4787-9217-5fb6b97e232b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 03 02:11:22.969384 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:22.963887 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f7d9bb/simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls" podUID="9173364c-14ba-4287-8587-45f25118e3e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:23.068475 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:23.042480 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr" podUID="8282371b-c42d-43ab-a5c9-e07f2464dcdc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 03 02:11:23.153323 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:23.111856 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a0e72e/premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv" podUID="a9b9af44-ab6d-4caf-89af-c986ea25b4c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:23.182384 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:23.153752 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1bccbd/distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg" Nov 03 02:11:23.270384 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:23.184187 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f7d9bb/simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls" Nov 03 02:11:23.271350 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:23.271322 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr" Nov 03 02:11:23.293532 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:23.293489 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a0e72e/premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv" Nov 03 02:11:23.318386 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:23.211706 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1bccbd/distinct-w1-1bccbd-kserve-5dfc85555f-tkvg6" podUID="dd4a9d44-c63e-4f67-bcac-a4f98ecf1796" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 03 02:11:23.348497 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:23.327785 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1bccbd/distinct-w1-1bccbd-kserve-5dfc85555f-tkvg6" Nov 03 02:11:23.415663 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:23.401168 5605 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-521f1f/simulator-w2-521f1f-kserve-5f47b86f6-mgksf" podUID="6e1b7fda-0ade-4042-ad63-5115bc66b454" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": context deadline exceeded" Nov 03 02:11:23.775570 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:23.775464 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-521f1f/simulator-w2-521f1f-kserve-5f47b86f6-mgksf" podUID="6e1b7fda-0ade-4042-ad63-5115bc66b454" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:23.898385 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:23.896398 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5" podUID="fffc3fbb-097a-4a5d-82c8-b1e221d0e710" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 03 02:11:23.909426 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:23.900954 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a0e72e/simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq" podUID="1901f57b-279b-4cbf-bf0c-054abc5786d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:23.909426 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:23.902566 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-521f1f/premium-w2-521f1f-kserve-7578fb7957-89wfb" podUID="3d988573-d8b1-43ba-a423-51a63ca12337" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Nov 03 02:11:23.936803 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:23.930501 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1bccbd/distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg" podUID="5a71cdee-425e-4787-9217-5fb6b97e232b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:23.941933 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:23.941793 5605 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:11:23.941933 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:23.941858 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g" podUID="f05a0c36-8c90-4813-b84c-0825be1947a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:23.982293 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:23.980625 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f7d9bb/simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls" podUID="9173364c-14ba-4287-8587-45f25118e3e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:24.208882 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:24.188819 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr" podUID="8282371b-c42d-43ab-a5c9-e07f2464dcdc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:24.322701 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:24.187842 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a0e72e/premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv" podUID="a9b9af44-ab6d-4caf-89af-c986ea25b4c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Nov 03 02:11:24.351705 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:24.351656 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1bccbd/distinct-w1-1bccbd-kserve-5dfc85555f-tkvg6" podUID="dd4a9d44-c63e-4f67-bcac-a4f98ecf1796" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:24.448758 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:24.448625 5605 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:11:24.449155 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:24.449007 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz" podUID="f014c618-b1ce-44b0-b042-f1234c247228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:24.465688 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:24.465533 5605 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c7d758/distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5" podUID="fffc3fbb-097a-4a5d-82c8-b1e221d0e710" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:24.802632 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:24.802589 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-521f1f/simulator-w2-521f1f-kserve-5f47b86f6-mgksf" podUID="6e1b7fda-0ade-4042-ad63-5115bc66b454" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:24.896830 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:24.896765 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5" podUID="fffc3fbb-097a-4a5d-82c8-b1e221d0e710" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:24.901612 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:24.901549 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a0e72e/simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq" podUID="1901f57b-279b-4cbf-bf0c-054abc5786d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:24.906102 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:24.906054 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-521f1f/premium-w2-521f1f-kserve-7578fb7957-89wfb" podUID="3d988573-d8b1-43ba-a423-51a63ca12337" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:24.984194 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:24.984139 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f7d9bb/simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls" podUID="9173364c-14ba-4287-8587-45f25118e3e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:25.046480 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:11:25.046410 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:11:25.060529 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:25.052848 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1bccbd/distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg" podUID="5a71cdee-425e-4787-9217-5fb6b97e232b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 03 02:11:25.281555 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:25.281505 5605 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:11:25.281924 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:25.281887 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr" podUID="e3b0cfb1-29ab-4570-81b6-4b55627e8067" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:25.328498 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:25.313579 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr" podUID="8282371b-c42d-43ab-a5c9-e07f2464dcdc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:25.337293 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:25.337239 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a0e72e/premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv" podUID="a9b9af44-ab6d-4caf-89af-c986ea25b4c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:25.627536 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:25.626308 5605 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a0e72e/simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq" podUID="1901f57b-279b-4cbf-bf0c-054abc5786d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:25.635400 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:25.634672 5605 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-521f1f/premium-w2-521f1f-kserve-7578fb7957-89wfb" podUID="3d988573-d8b1-43ba-a423-51a63ca12337" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:25.668149 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:25.665500 5605 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:11:25.668149 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:25.665585 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2" podUID="ae6b294b-6d5b-4bd7-820b-7e179011a2f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:25.678478 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:25.678411 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1bccbd/distinct-w1-1bccbd-kserve-5dfc85555f-tkvg6" podUID="dd4a9d44-c63e-4f67-bcac-a4f98ecf1796" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:25.730533 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:25.730486 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5911deb-p7d59"] Nov 03 02:11:25.737891 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:25.737857 5605 cpu_manager.go:414] "containerMap: removing container" podUID="8772ad29-8e2b-4236-a07a-99878ef696cc" containerName="revoke-keys" Nov 03 02:11:25.738144 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:25.738125 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8772ad29-8e2b-4236-a07a-99878ef696cc" containerName="revoke-keys" Nov 03 02:11:25.742612 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:25.742588 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="8772ad29-8e2b-4236-a07a-99878ef696cc" containerName="revoke-keys" Nov 03 02:11:25.760648 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:25.760618 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5911deb-p7d59"] Nov 03 02:11:25.760994 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:25.760974 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5911deb-p7d59" Nov 03 02:11:25.803840 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:25.803787 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-521f1f/simulator-w2-521f1f-kserve-5f47b86f6-mgksf" podUID="6e1b7fda-0ade-4042-ad63-5115bc66b454" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:25.806951 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:25.805103 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31c54e65-3b9f-4c2e-b8a8-a9a5166fdedf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5911deb-p7d59\" (UID: \"31c54e65-3b9f-4c2e-b8a8-a9a5166fdedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5911deb-p7d59" Nov 03 02:11:25.806951 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:25.806540 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gm9r\" (UniqueName: \"kubernetes.io/projected/31c54e65-3b9f-4c2e-b8a8-a9a5166fdedf-kube-api-access-6gm9r\") pod \"maas-api-revoke-sub-c5911deb-p7d59\" (UID: \"31c54e65-3b9f-4c2e-b8a8-a9a5166fdedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5911deb-p7d59" Nov 03 02:11:25.806951 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:25.806821 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31c54e65-3b9f-4c2e-b8a8-a9a5166fdedf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5911deb-p7d59\" (UID: \"31c54e65-3b9f-4c2e-b8a8-a9a5166fdedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5911deb-p7d59" Nov 03 02:11:25.816806 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:25.816659 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:11:25.816806 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:25.816752 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:11:25.862550 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:25.862445 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:11:25.862550 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:25.862520 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:11:25.903340 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:25.902476 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5" podUID="fffc3fbb-097a-4a5d-82c8-b1e221d0e710" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 03 02:11:25.915411 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:25.914545 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a0e72e/simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq" podUID="1901f57b-279b-4cbf-bf0c-054abc5786d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:25.917452 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:25.916503 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31c54e65-3b9f-4c2e-b8a8-a9a5166fdedf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5911deb-p7d59\" (UID: \"31c54e65-3b9f-4c2e-b8a8-a9a5166fdedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5911deb-p7d59" Nov 03 02:11:25.917452 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:25.916566 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31c54e65-3b9f-4c2e-b8a8-a9a5166fdedf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5911deb-p7d59\" (UID: \"31c54e65-3b9f-4c2e-b8a8-a9a5166fdedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5911deb-p7d59" Nov 03 02:11:25.917452 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:25.916594 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gm9r\" (UniqueName: \"kubernetes.io/projected/31c54e65-3b9f-4c2e-b8a8-a9a5166fdedf-kube-api-access-6gm9r\") pod \"maas-api-revoke-sub-c5911deb-p7d59\" (UID: \"31c54e65-3b9f-4c2e-b8a8-a9a5166fdedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5911deb-p7d59" Nov 03 02:11:25.933761 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:25.933710 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31c54e65-3b9f-4c2e-b8a8-a9a5166fdedf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5911deb-p7d59\" (UID: \"31c54e65-3b9f-4c2e-b8a8-a9a5166fdedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5911deb-p7d59" Nov 03 02:11:25.941079 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:25.940993 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31c54e65-3b9f-4c2e-b8a8-a9a5166fdedf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5911deb-p7d59\" (UID: \"31c54e65-3b9f-4c2e-b8a8-a9a5166fdedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5911deb-p7d59" Nov 03 02:11:25.958328 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:25.957532 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-521f1f/premium-w2-521f1f-kserve-7578fb7957-89wfb" podUID="3d988573-d8b1-43ba-a423-51a63ca12337" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:25.985271 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:25.984851 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f7d9bb/simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls" podUID="9173364c-14ba-4287-8587-45f25118e3e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:26.012681 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:26.012646 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gm9r\" (UniqueName: \"kubernetes.io/projected/31c54e65-3b9f-4c2e-b8a8-a9a5166fdedf-kube-api-access-6gm9r\") pod \"maas-api-revoke-sub-c5911deb-p7d59\" (UID: \"31c54e65-3b9f-4c2e-b8a8-a9a5166fdedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5911deb-p7d59" Nov 03 02:11:26.094387 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:26.075853 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1bccbd/distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg" podUID="5a71cdee-425e-4787-9217-5fb6b97e232b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:26.142516 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:26.142476 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5911deb-p7d59" Nov 03 02:11:26.178929 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:26.176696 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a0e72e/simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq" Nov 03 02:11:26.319384 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:26.314084 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr" podUID="8282371b-c42d-43ab-a5c9-e07f2464dcdc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:26.551379 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:26.549933 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c7d758/distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5" Nov 03 02:11:26.633924 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:26.633894 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1bccbd/distinct-w1-1bccbd-kserve-5dfc85555f-tkvg6" Nov 03 02:11:26.646420 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:26.642340 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a0e72e/premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv" podUID="a9b9af44-ab6d-4caf-89af-c986ea25b4c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Nov 03 02:11:26.744875 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:26.744787 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1bccbd/distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg" Nov 03 02:11:26.761763 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:26.760834 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a0e72e/premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv" Nov 03 02:11:26.850162 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:26.849490 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-521f1f/simulator-w2-521f1f-kserve-5f47b86f6-mgksf" podUID="6e1b7fda-0ade-4042-ad63-5115bc66b454" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:26.942421 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:26.915733 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f7d9bb/simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls" Nov 03 02:11:26.971390 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:26.958895 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-521f1f/premium-w2-521f1f-kserve-7578fb7957-89wfb" Nov 03 02:11:27.023166 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:27.021667 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-521f1f/simulator-w2-521f1f-kserve-5f47b86f6-mgksf" Nov 03 02:11:27.023166 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:27.022551 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr" Nov 03 02:11:27.084376 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:27.082867 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5911deb-p7d59"] Nov 03 02:11:27.773413 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:27.773286 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5911deb-p7d59" event={"ID":"31c54e65-3b9f-4c2e-b8a8-a9a5166fdedf","Type":"ContainerStarted","Data":"f0ecdde3cf90200cbdd8c92f35242e20f4c2ea8210e94fbf46ca024b5d2c91ff"} Nov 03 02:11:27.773413 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:27.773400 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5911deb-p7d59" event={"ID":"31c54e65-3b9f-4c2e-b8a8-a9a5166fdedf","Type":"ContainerStarted","Data":"21c23b527c11043e1ddc83a4689100434274b68f4aca74d4c714f4189ce6f975"} Nov 03 02:11:27.880516 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:27.880460 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5911deb-p7d59" podStartSLOduration=3.8804274149999998 podStartE2EDuration="3.880427415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:11:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:11:27.879048584 +0000 UTC m=+1165.138246738" watchObservedRunningTime="2026-11-03 02:11:27.880427415 +0000 UTC m=+1165.139625598" Nov 03 02:11:28.818181 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:28.818142 5605 generic.go:356] "Generic (PLEG): container finished" podID="31c54e65-3b9f-4c2e-b8a8-a9a5166fdedf" containerID="f0ecdde3cf90200cbdd8c92f35242e20f4c2ea8210e94fbf46ca024b5d2c91ff" exitCode=0 Nov 03 02:11:28.818494 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:28.818464 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5911deb-p7d59" event={"ID":"31c54e65-3b9f-4c2e-b8a8-a9a5166fdedf","Type":"ContainerDied","Data":"f0ecdde3cf90200cbdd8c92f35242e20f4c2ea8210e94fbf46ca024b5d2c91ff"} Nov 03 02:11:29.083824 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:29.083727 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf"] Nov 03 02:11:29.087419 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:29.087340 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf" Nov 03 02:11:29.143612 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:29.143574 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf"] Nov 03 02:11:29.166837 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:29.166797 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-701ba30f-openshift-default-dockercfg-dcsnq\"" Nov 03 02:11:29.196968 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:29.196933 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3aa07ac7-fa23-451b-a6a6-0bea3103c9d1-istio-token\") pod \"e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf\" (UID: \"3aa07ac7-fa23-451b-a6a6-0bea3103c9d1\") " pod="openshift-ingress/e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf" Nov 03 02:11:29.197160 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:29.196981 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aa07ac7-fa23-451b-a6a6-0bea3103c9d1-workload-certs\") pod \"e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf\" (UID: \"3aa07ac7-fa23-451b-a6a6-0bea3103c9d1\") " pod="openshift-ingress/e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf" Nov 03 02:11:29.197160 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:29.197107 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa07ac7-fa23-451b-a6a6-0bea3103c9d1-workload-socket\") pod \"e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf\" (UID: \"3aa07ac7-fa23-451b-a6a6-0bea3103c9d1\") " pod="openshift-ingress/e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf" Nov 03 02:11:29.197160 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:29.197136 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aa07ac7-fa23-451b-a6a6-0bea3103c9d1-istio-envoy\") pod \"e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf\" (UID: \"3aa07ac7-fa23-451b-a6a6-0bea3103c9d1\") " pod="openshift-ingress/e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf" Nov 03 02:11:29.197299 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:29.197233 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z74sx\" (UniqueName: \"kubernetes.io/projected/3aa07ac7-fa23-451b-a6a6-0bea3103c9d1-kube-api-access-z74sx\") pod \"e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf\" (UID: \"3aa07ac7-fa23-451b-a6a6-0bea3103c9d1\") " pod="openshift-ingress/e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf" Nov 03 02:11:29.197299 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:29.197274 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aa07ac7-fa23-451b-a6a6-0bea3103c9d1-istio-data\") pod \"e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf\" (UID: \"3aa07ac7-fa23-451b-a6a6-0bea3103c9d1\") " pod="openshift-ingress/e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf" Nov 03 02:11:29.197423 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:29.197413 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aa07ac7-fa23-451b-a6a6-0bea3103c9d1-istiod-ca-cert\") pod \"e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf\" (UID: \"3aa07ac7-fa23-451b-a6a6-0bea3103c9d1\") " pod="openshift-ingress/e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf" Nov 03 02:11:29.197469 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:29.197454 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa07ac7-fa23-451b-a6a6-0bea3103c9d1-credential-socket\") pod \"e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf\" (UID: \"3aa07ac7-fa23-451b-a6a6-0bea3103c9d1\") " pod="openshift-ingress/e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf" Nov 03 02:11:29.197576 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:29.197549 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aa07ac7-fa23-451b-a6a6-0bea3103c9d1-istio-podinfo\") pod \"e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf\" (UID: \"3aa07ac7-fa23-451b-a6a6-0bea3103c9d1\") " pod="openshift-ingress/e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf" Nov 03 02:11:29.301386 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:29.300102 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa07ac7-fa23-451b-a6a6-0bea3103c9d1-credential-socket\") pod \"e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf\" (UID: \"3aa07ac7-fa23-451b-a6a6-0bea3103c9d1\") " pod="openshift-ingress/e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf" Nov 03 02:11:29.301386 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:29.300200 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aa07ac7-fa23-451b-a6a6-0bea3103c9d1-istio-podinfo\") pod \"e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf\" (UID: \"3aa07ac7-fa23-451b-a6a6-0bea3103c9d1\") " pod="openshift-ingress/e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf" Nov 03 02:11:29.301386 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:29.300247 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3aa07ac7-fa23-451b-a6a6-0bea3103c9d1-istio-token\") pod \"e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf\" (UID: \"3aa07ac7-fa23-451b-a6a6-0bea3103c9d1\") " pod="openshift-ingress/e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf" Nov 03 02:11:29.301386 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:29.300283 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aa07ac7-fa23-451b-a6a6-0bea3103c9d1-workload-certs\") pod \"e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf\" (UID: \"3aa07ac7-fa23-451b-a6a6-0bea3103c9d1\") " pod="openshift-ingress/e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf" Nov 03 02:11:29.301386 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:29.300375 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa07ac7-fa23-451b-a6a6-0bea3103c9d1-workload-socket\") pod \"e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf\" (UID: \"3aa07ac7-fa23-451b-a6a6-0bea3103c9d1\") " pod="openshift-ingress/e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf" Nov 03 02:11:29.301386 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:29.300401 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aa07ac7-fa23-451b-a6a6-0bea3103c9d1-istio-envoy\") pod \"e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf\" (UID: \"3aa07ac7-fa23-451b-a6a6-0bea3103c9d1\") " pod="openshift-ingress/e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf" Nov 03 02:11:29.301386 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:29.300495 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z74sx\" (UniqueName: \"kubernetes.io/projected/3aa07ac7-fa23-451b-a6a6-0bea3103c9d1-kube-api-access-z74sx\") pod \"e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf\" (UID: \"3aa07ac7-fa23-451b-a6a6-0bea3103c9d1\") " pod="openshift-ingress/e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf" Nov 03 02:11:29.301386 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:29.300533 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aa07ac7-fa23-451b-a6a6-0bea3103c9d1-istio-data\") pod \"e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf\" (UID: \"3aa07ac7-fa23-451b-a6a6-0bea3103c9d1\") " pod="openshift-ingress/e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf" Nov 03 02:11:29.301386 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:29.300637 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aa07ac7-fa23-451b-a6a6-0bea3103c9d1-istiod-ca-cert\") pod \"e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf\" (UID: \"3aa07ac7-fa23-451b-a6a6-0bea3103c9d1\") " pod="openshift-ingress/e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf" Nov 03 02:11:29.301981 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:29.301701 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aa07ac7-fa23-451b-a6a6-0bea3103c9d1-istiod-ca-cert\") pod \"e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf\" (UID: \"3aa07ac7-fa23-451b-a6a6-0bea3103c9d1\") " pod="openshift-ingress/e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf" Nov 03 02:11:29.302030 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:29.301987 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa07ac7-fa23-451b-a6a6-0bea3103c9d1-credential-socket\") pod \"e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf\" (UID: \"3aa07ac7-fa23-451b-a6a6-0bea3103c9d1\") " pod="openshift-ingress/e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf" Nov 03 02:11:29.308392 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:29.306731 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa07ac7-fa23-451b-a6a6-0bea3103c9d1-workload-socket\") pod \"e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf\" (UID: \"3aa07ac7-fa23-451b-a6a6-0bea3103c9d1\") " pod="openshift-ingress/e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf" Nov 03 02:11:29.308392 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:29.307426 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aa07ac7-fa23-451b-a6a6-0bea3103c9d1-workload-certs\") pod \"e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf\" (UID: \"3aa07ac7-fa23-451b-a6a6-0bea3103c9d1\") " pod="openshift-ingress/e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf" Nov 03 02:11:29.316394 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:29.313501 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aa07ac7-fa23-451b-a6a6-0bea3103c9d1-istio-podinfo\") pod \"e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf\" (UID: \"3aa07ac7-fa23-451b-a6a6-0bea3103c9d1\") " pod="openshift-ingress/e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf" Nov 03 02:11:29.316394 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:29.313884 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aa07ac7-fa23-451b-a6a6-0bea3103c9d1-istio-data\") pod \"e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf\" (UID: \"3aa07ac7-fa23-451b-a6a6-0bea3103c9d1\") " pod="openshift-ingress/e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf" Nov 03 02:11:29.321516 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:29.321481 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aa07ac7-fa23-451b-a6a6-0bea3103c9d1-istio-envoy\") pod \"e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf\" (UID: \"3aa07ac7-fa23-451b-a6a6-0bea3103c9d1\") " pod="openshift-ingress/e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf" Nov 03 02:11:29.401557 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:29.393376 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z74sx\" (UniqueName: \"kubernetes.io/projected/3aa07ac7-fa23-451b-a6a6-0bea3103c9d1-kube-api-access-z74sx\") pod \"e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf\" (UID: \"3aa07ac7-fa23-451b-a6a6-0bea3103c9d1\") " pod="openshift-ingress/e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf" Nov 03 02:11:29.401557 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:29.400628 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3aa07ac7-fa23-451b-a6a6-0bea3103c9d1-istio-token\") pod \"e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf\" (UID: \"3aa07ac7-fa23-451b-a6a6-0bea3103c9d1\") " pod="openshift-ingress/e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf" Nov 03 02:11:29.422869 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:29.422821 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf" Nov 03 02:11:29.965493 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:29.965458 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5911deb-p7d59" Nov 03 02:11:30.039559 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:30.039493 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31c54e65-3b9f-4c2e-b8a8-a9a5166fdedf-kube-api-access-6gm9r\" (UniqueName: \"kubernetes.io/projected/31c54e65-3b9f-4c2e-b8a8-a9a5166fdedf-kube-api-access-6gm9r\") pod \"31c54e65-3b9f-4c2e-b8a8-a9a5166fdedf\" (UID: \"31c54e65-3b9f-4c2e-b8a8-a9a5166fdedf\") " Nov 03 02:11:30.039559 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:30.039561 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31c54e65-3b9f-4c2e-b8a8-a9a5166fdedf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31c54e65-3b9f-4c2e-b8a8-a9a5166fdedf-maas-api-openshift-ca\") pod \"31c54e65-3b9f-4c2e-b8a8-a9a5166fdedf\" (UID: \"31c54e65-3b9f-4c2e-b8a8-a9a5166fdedf\") " Nov 03 02:11:30.039829 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:30.039672 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31c54e65-3b9f-4c2e-b8a8-a9a5166fdedf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31c54e65-3b9f-4c2e-b8a8-a9a5166fdedf-maas-api-xks-ca\") pod \"31c54e65-3b9f-4c2e-b8a8-a9a5166fdedf\" (UID: \"31c54e65-3b9f-4c2e-b8a8-a9a5166fdedf\") " Nov 03 02:11:30.040286 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:30.040253 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31c54e65-3b9f-4c2e-b8a8-a9a5166fdedf-maas-api-openshift-ca" pod "31c54e65-3b9f-4c2e-b8a8-a9a5166fdedf" (UID: "31c54e65-3b9f-4c2e-b8a8-a9a5166fdedf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:11:30.040631 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:30.040609 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31c54e65-3b9f-4c2e-b8a8-a9a5166fdedf-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:11:30.042629 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:30.042594 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31c54e65-3b9f-4c2e-b8a8-a9a5166fdedf-maas-api-xks-ca" pod "31c54e65-3b9f-4c2e-b8a8-a9a5166fdedf" (UID: "31c54e65-3b9f-4c2e-b8a8-a9a5166fdedf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:11:30.042758 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:30.042748 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31c54e65-3b9f-4c2e-b8a8-a9a5166fdedf-kube-api-access-6gm9r" pod "31c54e65-3b9f-4c2e-b8a8-a9a5166fdedf" (UID: "31c54e65-3b9f-4c2e-b8a8-a9a5166fdedf"). InnerVolumeSpecName "kube-api-access-6gm9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:11:30.092758 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:11:30.092709 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aa07ac7_fa23_451b_a6a6_0bea3103c9d1.slice/crio-5e4e7fea4dea39540780919d9144df879441976cbaaf032257613421d4c7b4c5 WatchSource:0}: Error finding container 5e4e7fea4dea39540780919d9144df879441976cbaaf032257613421d4c7b4c5: Status 404 returned error can't find the container with id 5e4e7fea4dea39540780919d9144df879441976cbaaf032257613421d4c7b4c5 Nov 03 02:11:30.096726 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:30.096679 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:11:30.096833 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:30.096782 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:11:30.096879 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:30.096834 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:11:30.128618 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:30.122149 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf"] Nov 03 02:11:30.142012 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:30.141979 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gm9r\" (UniqueName: \"kubernetes.io/projected/31c54e65-3b9f-4c2e-b8a8-a9a5166fdedf-kube-api-access-6gm9r\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:11:30.142012 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:30.142012 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31c54e65-3b9f-4c2e-b8a8-a9a5166fdedf-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:11:30.833164 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:30.833121 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf" event={"ID":"3aa07ac7-fa23-451b-a6a6-0bea3103c9d1","Type":"ContainerStarted","Data":"15d7220b13fb52a80c5cf245b3f6ee02c489b270610086b5e918c017e3b411d6"} Nov 03 02:11:30.833399 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:30.833217 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf" event={"ID":"3aa07ac7-fa23-451b-a6a6-0bea3103c9d1","Type":"ContainerStarted","Data":"5e4e7fea4dea39540780919d9144df879441976cbaaf032257613421d4c7b4c5"} Nov 03 02:11:30.834682 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:30.834652 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5911deb-p7d59" event={"ID":"31c54e65-3b9f-4c2e-b8a8-a9a5166fdedf","Type":"ContainerDied","Data":"21c23b527c11043e1ddc83a4689100434274b68f4aca74d4c714f4189ce6f975"} Nov 03 02:11:30.834819 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:30.834687 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5911deb-p7d59" Nov 03 02:11:30.834819 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:30.834696 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21c23b527c11043e1ddc83a4689100434274b68f4aca74d4c714f4189ce6f975" Nov 03 02:11:30.991126 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:30.991067 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf" podStartSLOduration=1.991052951 podStartE2EDuration="1.991052951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:11:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:11:30.988044947 +0000 UTC m=+1168.247243101" watchObservedRunningTime="2026-11-03 02:11:30.991052951 +0000 UTC m=+1168.250251104" Nov 03 02:11:31.423767 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:31.423727 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf" Nov 03 02:11:31.429404 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:31.429372 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf" Nov 03 02:11:31.841309 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:31.841276 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf" Nov 03 02:11:31.842241 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:31.842218 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf" Nov 03 02:11:34.907454 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:34.907410 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2"] Nov 03 02:11:34.908005 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:34.907661 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2" podUID="868119ea-b49f-4566-8a46-bb3990a1e6f6" containerName="istio-proxy" containerID="cri-o://baaf9d99689962b25fee4f5f220b168b506a4210f39550c37999e78a23f86a78" gracePeriod=30 Nov 03 02:11:40.182503 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:40.182438 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2" Nov 03 02:11:40.262609 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:40.262574 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/868119ea-b49f-4566-8a46-bb3990a1e6f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/868119ea-b49f-4566-8a46-bb3990a1e6f6-istiod-ca-cert\") pod \"868119ea-b49f-4566-8a46-bb3990a1e6f6\" (UID: \"868119ea-b49f-4566-8a46-bb3990a1e6f6\") " Nov 03 02:11:40.262795 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:40.262636 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/868119ea-b49f-4566-8a46-bb3990a1e6f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/868119ea-b49f-4566-8a46-bb3990a1e6f6-istio-podinfo\") pod \"868119ea-b49f-4566-8a46-bb3990a1e6f6\" (UID: \"868119ea-b49f-4566-8a46-bb3990a1e6f6\") " Nov 03 02:11:40.262795 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:40.262687 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/868119ea-b49f-4566-8a46-bb3990a1e6f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/868119ea-b49f-4566-8a46-bb3990a1e6f6-istio-data\") pod \"868119ea-b49f-4566-8a46-bb3990a1e6f6\" (UID: \"868119ea-b49f-4566-8a46-bb3990a1e6f6\") " Nov 03 02:11:40.262795 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:40.262723 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/868119ea-b49f-4566-8a46-bb3990a1e6f6-kube-api-access-l4vdn\" (UniqueName: \"kubernetes.io/projected/868119ea-b49f-4566-8a46-bb3990a1e6f6-kube-api-access-l4vdn\") pod \"868119ea-b49f-4566-8a46-bb3990a1e6f6\" (UID: \"868119ea-b49f-4566-8a46-bb3990a1e6f6\") " Nov 03 02:11:40.262795 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:40.262768 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/868119ea-b49f-4566-8a46-bb3990a1e6f6-istio-token\" (UniqueName: \"kubernetes.io/projected/868119ea-b49f-4566-8a46-bb3990a1e6f6-istio-token\") pod \"868119ea-b49f-4566-8a46-bb3990a1e6f6\" (UID: \"868119ea-b49f-4566-8a46-bb3990a1e6f6\") " Nov 03 02:11:40.263002 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:40.262800 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/868119ea-b49f-4566-8a46-bb3990a1e6f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/868119ea-b49f-4566-8a46-bb3990a1e6f6-workload-socket\") pod \"868119ea-b49f-4566-8a46-bb3990a1e6f6\" (UID: \"868119ea-b49f-4566-8a46-bb3990a1e6f6\") " Nov 03 02:11:40.263002 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:40.262875 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/868119ea-b49f-4566-8a46-bb3990a1e6f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/868119ea-b49f-4566-8a46-bb3990a1e6f6-credential-socket\") pod \"868119ea-b49f-4566-8a46-bb3990a1e6f6\" (UID: \"868119ea-b49f-4566-8a46-bb3990a1e6f6\") " Nov 03 02:11:40.263002 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:40.262904 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/868119ea-b49f-4566-8a46-bb3990a1e6f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/868119ea-b49f-4566-8a46-bb3990a1e6f6-workload-certs\") pod \"868119ea-b49f-4566-8a46-bb3990a1e6f6\" (UID: \"868119ea-b49f-4566-8a46-bb3990a1e6f6\") " Nov 03 02:11:40.263002 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:40.262965 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/868119ea-b49f-4566-8a46-bb3990a1e6f6-istio-data" pod "868119ea-b49f-4566-8a46-bb3990a1e6f6" (UID: "868119ea-b49f-4566-8a46-bb3990a1e6f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:11:40.263002 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:40.262994 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/868119ea-b49f-4566-8a46-bb3990a1e6f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/868119ea-b49f-4566-8a46-bb3990a1e6f6-istio-envoy\") pod \"868119ea-b49f-4566-8a46-bb3990a1e6f6\" (UID: \"868119ea-b49f-4566-8a46-bb3990a1e6f6\") " Nov 03 02:11:40.263229 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:40.263141 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/868119ea-b49f-4566-8a46-bb3990a1e6f6-workload-socket" pod "868119ea-b49f-4566-8a46-bb3990a1e6f6" (UID: "868119ea-b49f-4566-8a46-bb3990a1e6f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:11:40.263229 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:40.263168 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/868119ea-b49f-4566-8a46-bb3990a1e6f6-credential-socket" pod "868119ea-b49f-4566-8a46-bb3990a1e6f6" (UID: "868119ea-b49f-4566-8a46-bb3990a1e6f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:11:40.263229 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:40.263047 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/868119ea-b49f-4566-8a46-bb3990a1e6f6-istiod-ca-cert" pod "868119ea-b49f-4566-8a46-bb3990a1e6f6" (UID: "868119ea-b49f-4566-8a46-bb3990a1e6f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:11:40.263473 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:40.263450 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/868119ea-b49f-4566-8a46-bb3990a1e6f6-workload-certs" pod "868119ea-b49f-4566-8a46-bb3990a1e6f6" (UID: "868119ea-b49f-4566-8a46-bb3990a1e6f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:11:40.263544 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:40.263529 5605 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/868119ea-b49f-4566-8a46-bb3990a1e6f6-credential-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:11:40.263598 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:40.263553 5605 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/868119ea-b49f-4566-8a46-bb3990a1e6f6-istiod-ca-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:11:40.263598 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:40.263568 5605 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/868119ea-b49f-4566-8a46-bb3990a1e6f6-istio-data\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:11:40.263598 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:40.263582 5605 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/868119ea-b49f-4566-8a46-bb3990a1e6f6-workload-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:11:40.264955 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:40.264926 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/868119ea-b49f-4566-8a46-bb3990a1e6f6-istio-podinfo" pod "868119ea-b49f-4566-8a46-bb3990a1e6f6" (UID: "868119ea-b49f-4566-8a46-bb3990a1e6f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:11:40.265142 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:40.265120 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/868119ea-b49f-4566-8a46-bb3990a1e6f6-istio-envoy" pod "868119ea-b49f-4566-8a46-bb3990a1e6f6" (UID: "868119ea-b49f-4566-8a46-bb3990a1e6f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:11:40.265206 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:40.265152 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/868119ea-b49f-4566-8a46-bb3990a1e6f6-istio-token" pod "868119ea-b49f-4566-8a46-bb3990a1e6f6" (UID: "868119ea-b49f-4566-8a46-bb3990a1e6f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:11:40.265456 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:40.265433 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/868119ea-b49f-4566-8a46-bb3990a1e6f6-kube-api-access-l4vdn" pod "868119ea-b49f-4566-8a46-bb3990a1e6f6" (UID: "868119ea-b49f-4566-8a46-bb3990a1e6f6"). InnerVolumeSpecName "kube-api-access-l4vdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:11:40.365120 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:40.365071 5605 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/868119ea-b49f-4566-8a46-bb3990a1e6f6-istio-envoy\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:11:40.365120 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:40.365098 5605 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/868119ea-b49f-4566-8a46-bb3990a1e6f6-istio-podinfo\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:11:40.365120 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:40.365107 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4vdn\" (UniqueName: \"kubernetes.io/projected/868119ea-b49f-4566-8a46-bb3990a1e6f6-kube-api-access-l4vdn\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:11:40.365120 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:40.365119 5605 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/868119ea-b49f-4566-8a46-bb3990a1e6f6-istio-token\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:11:40.365120 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:40.365128 5605 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/868119ea-b49f-4566-8a46-bb3990a1e6f6-workload-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:11:40.893644 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:40.893598 5605 generic.go:356] "Generic (PLEG): container finished" podID="868119ea-b49f-4566-8a46-bb3990a1e6f6" containerID="baaf9d99689962b25fee4f5f220b168b506a4210f39550c37999e78a23f86a78" exitCode=0 Nov 03 02:11:40.893820 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:40.893674 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2" Nov 03 02:11:40.893820 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:40.893673 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2" event={"ID":"868119ea-b49f-4566-8a46-bb3990a1e6f6","Type":"ContainerDied","Data":"baaf9d99689962b25fee4f5f220b168b506a4210f39550c37999e78a23f86a78"} Nov 03 02:11:40.893820 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:40.893809 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2" event={"ID":"868119ea-b49f-4566-8a46-bb3990a1e6f6","Type":"ContainerDied","Data":"f25ca3af36c6193b6373bbc6ad8a4b0da5abd44124d7935a870501a02c1f07b5"} Nov 03 02:11:40.893923 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:40.893840 5605 scope.go:122] "RemoveContainer" containerID="baaf9d99689962b25fee4f5f220b168b506a4210f39550c37999e78a23f86a78" Nov 03 02:11:40.904664 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:40.904639 5605 scope.go:122] "RemoveContainer" containerID="baaf9d99689962b25fee4f5f220b168b506a4210f39550c37999e78a23f86a78" Nov 03 02:11:40.904979 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:11:40.904955 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baaf9d99689962b25fee4f5f220b168b506a4210f39550c37999e78a23f86a78\": container with ID starting with baaf9d99689962b25fee4f5f220b168b506a4210f39550c37999e78a23f86a78 not found: ID does not exist" containerID="baaf9d99689962b25fee4f5f220b168b506a4210f39550c37999e78a23f86a78" Nov 03 02:11:40.905093 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:40.904985 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baaf9d99689962b25fee4f5f220b168b506a4210f39550c37999e78a23f86a78"} err="failed to get container status \"baaf9d99689962b25fee4f5f220b168b506a4210f39550c37999e78a23f86a78\": rpc error: code = NotFound desc = could not find container \"baaf9d99689962b25fee4f5f220b168b506a4210f39550c37999e78a23f86a78\": container with ID starting with baaf9d99689962b25fee4f5f220b168b506a4210f39550c37999e78a23f86a78 not found: ID does not exist" Nov 03 02:11:41.094847 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:41.094816 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2"] Nov 03 02:11:41.141603 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:41.141516 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-4nmh2"] Nov 03 02:11:42.849621 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:42.849585 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="868119ea-b49f-4566-8a46-bb3990a1e6f6" path="/var/lib/kubelet/pods/868119ea-b49f-4566-8a46-bb3990a1e6f6/volumes" Nov 03 02:11:53.916915 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:11:53.916858 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:11:55.540486 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:55.540449 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:11:58.364383 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:11:58.362698 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr" podUID="8282371b-c42d-43ab-a5c9-e07f2464dcdc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 02:12:01.109955 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:01.109912 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-jl8k2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:12:01.110728 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:01.110691 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jl8k2" podUID="92507c2a-dbae-47e0-96f7-d50a7ccbd312" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:12:01.242789 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:01.242748 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-4gq8z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:12:01.242953 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:01.242821 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4gq8z" podUID="1d83861f-abc5-452e-a1d9-3148c6cd3774" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:12:01.563306 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:01.563272 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6b5d8ddb76-qp2r8"] Nov 03 02:12:01.563749 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:01.563736 5605 cpu_manager.go:414] "containerMap: removing container" podUID="31c54e65-3b9f-4c2e-b8a8-a9a5166fdedf" containerName="revoke-keys" Nov 03 02:12:01.563797 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:01.563750 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31c54e65-3b9f-4c2e-b8a8-a9a5166fdedf" containerName="revoke-keys" Nov 03 02:12:01.563797 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:01.563775 5605 cpu_manager.go:414] "containerMap: removing container" podUID="868119ea-b49f-4566-8a46-bb3990a1e6f6" containerName="istio-proxy" Nov 03 02:12:01.563797 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:01.563781 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="868119ea-b49f-4566-8a46-bb3990a1e6f6" containerName="istio-proxy" Nov 03 02:12:01.563888 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:01.563862 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="31c54e65-3b9f-4c2e-b8a8-a9a5166fdedf" containerName="revoke-keys" Nov 03 02:12:01.563888 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:01.563874 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="868119ea-b49f-4566-8a46-bb3990a1e6f6" containerName="istio-proxy" Nov 03 02:12:01.567534 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:01.567511 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6b5d8ddb76-qp2r8" Nov 03 02:12:01.606581 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:01.606544 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/63bedac3-9254-4e1b-adf2-8f70b6425674-routing-overlay-volume\") pod \"payload-processing-external-model-6b5d8ddb76-qp2r8\" (UID: \"63bedac3-9254-4e1b-adf2-8f70b6425674\") " pod="models-as-a-service/payload-processing-external-model-6b5d8ddb76-qp2r8" Nov 03 02:12:01.606751 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:01.606622 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63bedac3-9254-4e1b-adf2-8f70b6425674-plugins-config-volume\") pod \"payload-processing-external-model-6b5d8ddb76-qp2r8\" (UID: \"63bedac3-9254-4e1b-adf2-8f70b6425674\") " pod="models-as-a-service/payload-processing-external-model-6b5d8ddb76-qp2r8" Nov 03 02:12:01.606751 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:01.606684 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/63bedac3-9254-4e1b-adf2-8f70b6425674-provider-credentials\") pod \"payload-processing-external-model-6b5d8ddb76-qp2r8\" (UID: \"63bedac3-9254-4e1b-adf2-8f70b6425674\") " pod="models-as-a-service/payload-processing-external-model-6b5d8ddb76-qp2r8" Nov 03 02:12:01.606842 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:01.606819 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xldtj\" (UniqueName: \"kubernetes.io/projected/63bedac3-9254-4e1b-adf2-8f70b6425674-kube-api-access-xldtj\") pod \"payload-processing-external-model-6b5d8ddb76-qp2r8\" (UID: \"63bedac3-9254-4e1b-adf2-8f70b6425674\") " pod="models-as-a-service/payload-processing-external-model-6b5d8ddb76-qp2r8" Nov 03 02:12:01.618533 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:01.618490 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-e895c513fa82\"" Nov 03 02:12:01.620192 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:01.620164 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b5d8ddb76-qp2r8"] Nov 03 02:12:01.663868 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:01.663832 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 03 02:12:01.663868 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:01.663852 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 03 02:12:01.663868 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:01.663872 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 03 02:12:01.664103 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:01.663852 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-kf7p8\"" Nov 03 02:12:01.664103 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:01.663901 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 03 02:12:01.720330 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:01.720282 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/63bedac3-9254-4e1b-adf2-8f70b6425674-routing-overlay-volume\") pod \"payload-processing-external-model-6b5d8ddb76-qp2r8\" (UID: \"63bedac3-9254-4e1b-adf2-8f70b6425674\") " pod="models-as-a-service/payload-processing-external-model-6b5d8ddb76-qp2r8" Nov 03 02:12:01.720576 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:01.720401 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63bedac3-9254-4e1b-adf2-8f70b6425674-plugins-config-volume\") pod \"payload-processing-external-model-6b5d8ddb76-qp2r8\" (UID: \"63bedac3-9254-4e1b-adf2-8f70b6425674\") " pod="models-as-a-service/payload-processing-external-model-6b5d8ddb76-qp2r8" Nov 03 02:12:01.720576 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:01.720445 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/63bedac3-9254-4e1b-adf2-8f70b6425674-provider-credentials\") pod \"payload-processing-external-model-6b5d8ddb76-qp2r8\" (UID: \"63bedac3-9254-4e1b-adf2-8f70b6425674\") " pod="models-as-a-service/payload-processing-external-model-6b5d8ddb76-qp2r8" Nov 03 02:12:01.720712 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:01.720580 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xldtj\" (UniqueName: \"kubernetes.io/projected/63bedac3-9254-4e1b-adf2-8f70b6425674-kube-api-access-xldtj\") pod \"payload-processing-external-model-6b5d8ddb76-qp2r8\" (UID: \"63bedac3-9254-4e1b-adf2-8f70b6425674\") " pod="models-as-a-service/payload-processing-external-model-6b5d8ddb76-qp2r8" Nov 03 02:12:01.722412 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:01.722377 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/63bedac3-9254-4e1b-adf2-8f70b6425674-routing-overlay-volume\") pod \"payload-processing-external-model-6b5d8ddb76-qp2r8\" (UID: \"63bedac3-9254-4e1b-adf2-8f70b6425674\") " pod="models-as-a-service/payload-processing-external-model-6b5d8ddb76-qp2r8" Nov 03 02:12:01.737822 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:01.737791 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/63bedac3-9254-4e1b-adf2-8f70b6425674-provider-credentials\") pod \"payload-processing-external-model-6b5d8ddb76-qp2r8\" (UID: \"63bedac3-9254-4e1b-adf2-8f70b6425674\") " pod="models-as-a-service/payload-processing-external-model-6b5d8ddb76-qp2r8" Nov 03 02:12:01.740715 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:01.740586 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63bedac3-9254-4e1b-adf2-8f70b6425674-plugins-config-volume\") pod \"payload-processing-external-model-6b5d8ddb76-qp2r8\" (UID: \"63bedac3-9254-4e1b-adf2-8f70b6425674\") " pod="models-as-a-service/payload-processing-external-model-6b5d8ddb76-qp2r8" Nov 03 02:12:01.793925 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:01.793887 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xldtj\" (UniqueName: \"kubernetes.io/projected/63bedac3-9254-4e1b-adf2-8f70b6425674-kube-api-access-xldtj\") pod \"payload-processing-external-model-6b5d8ddb76-qp2r8\" (UID: \"63bedac3-9254-4e1b-adf2-8f70b6425674\") " pod="models-as-a-service/payload-processing-external-model-6b5d8ddb76-qp2r8" Nov 03 02:12:01.880963 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:01.880874 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6b5d8ddb76-qp2r8" Nov 03 02:12:02.287880 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:02.287785 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b5d8ddb76-qp2r8"] Nov 03 02:12:02.293020 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:12:02.292991 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63bedac3_9254_4e1b_adf2_8f70b6425674.slice/crio-d2a0e07fa2fe37fe65010de3e7b9d55d5ce3de7d7ed7311ccb017ee104f97890 WatchSource:0}: Error finding container d2a0e07fa2fe37fe65010de3e7b9d55d5ce3de7d7ed7311ccb017ee104f97890: Status 404 returned error can't find the container with id d2a0e07fa2fe37fe65010de3e7b9d55d5ce3de7d7ed7311ccb017ee104f97890 Nov 03 02:12:02.555591 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:02.555546 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6b5d8ddb76-qp2r8" event={"ID":"63bedac3-9254-4e1b-adf2-8f70b6425674","Type":"ContainerStarted","Data":"43aa4ec8c029664e217a022b7163a9fdf14593dab818843a67a1363afc333be4"} Nov 03 02:12:02.555807 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:02.555737 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6b5d8ddb76-qp2r8" Nov 03 02:12:02.555807 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:02.555786 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6b5d8ddb76-qp2r8" event={"ID":"63bedac3-9254-4e1b-adf2-8f70b6425674","Type":"ContainerStarted","Data":"d2a0e07fa2fe37fe65010de3e7b9d55d5ce3de7d7ed7311ccb017ee104f97890"} Nov 03 02:12:02.662626 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:02.662575 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6b5d8ddb76-qp2r8" podStartSLOduration=1.662560439 podStartE2EDuration="1.662560439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:12:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:12:02.661955737 +0000 UTC m=+1199.921153904" watchObservedRunningTime="2026-11-03 02:12:02.662560439 +0000 UTC m=+1199.921758592" Nov 03 02:12:03.027156 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:03.027057 5605 scope.go:122] "RemoveContainer" containerID="f7fc02c64e453c882bd3c2cddfe88ed1b3936e4732e5c7f2767aee9db4ff2bec" Nov 03 02:12:03.038910 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:03.038804 5605 scope.go:122] "RemoveContainer" containerID="d041fac9df564313f4d6bc8a288173ba1843859075498a5fdd281a8bec6f98f8" Nov 03 02:12:13.313965 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:13.313925 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs"] Nov 03 02:12:13.319475 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:13.319446 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs" Nov 03 02:12:13.413623 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:13.413579 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd-istio-envoy\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs\" (UID: \"8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs" Nov 03 02:12:13.413820 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:13.413648 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd-istio-token\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs\" (UID: \"8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs" Nov 03 02:12:13.413820 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:13.413694 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd-istio-data\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs\" (UID: \"8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs" Nov 03 02:12:13.413908 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:13.413871 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd-credential-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs\" (UID: \"8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs" Nov 03 02:12:13.413949 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:13.413930 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd-istio-podinfo\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs\" (UID: \"8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs" Nov 03 02:12:13.414020 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:13.414004 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd-workload-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs\" (UID: \"8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs" Nov 03 02:12:13.414061 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:13.414045 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd-workload-certs\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs\" (UID: \"8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs" Nov 03 02:12:13.414124 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:13.414066 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvvsg\" (UniqueName: \"kubernetes.io/projected/8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd-kube-api-access-kvvsg\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs\" (UID: \"8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs" Nov 03 02:12:13.414193 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:13.414169 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd-istiod-ca-cert\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs\" (UID: \"8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs" Nov 03 02:12:13.420608 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:13.420573 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs"] Nov 03 02:12:13.515656 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:13.515613 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd-workload-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs\" (UID: \"8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs" Nov 03 02:12:13.515863 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:13.515671 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd-workload-certs\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs\" (UID: \"8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs" Nov 03 02:12:13.515863 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:13.515700 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvvsg\" (UniqueName: \"kubernetes.io/projected/8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd-kube-api-access-kvvsg\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs\" (UID: \"8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs" Nov 03 02:12:13.515863 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:13.515732 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd-istiod-ca-cert\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs\" (UID: \"8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs" Nov 03 02:12:13.515863 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:13.515800 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd-istio-envoy\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs\" (UID: \"8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs" Nov 03 02:12:13.515863 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:13.515840 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd-istio-token\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs\" (UID: \"8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs" Nov 03 02:12:13.516170 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:13.515883 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd-istio-data\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs\" (UID: \"8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs" Nov 03 02:12:13.516170 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:13.515963 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd-credential-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs\" (UID: \"8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs" Nov 03 02:12:13.516170 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:13.516007 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd-istio-podinfo\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs\" (UID: \"8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs" Nov 03 02:12:13.516320 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:13.516262 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd-workload-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs\" (UID: \"8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs" Nov 03 02:12:13.516320 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:13.516301 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd-workload-certs\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs\" (UID: \"8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs" Nov 03 02:12:13.516501 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:13.516475 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd-credential-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs\" (UID: \"8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs" Nov 03 02:12:13.516549 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:13.516528 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd-istio-data\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs\" (UID: \"8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs" Nov 03 02:12:13.516806 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:13.516786 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd-istiod-ca-cert\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs\" (UID: \"8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs" Nov 03 02:12:13.518174 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:13.518156 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd-istio-envoy\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs\" (UID: \"8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs" Nov 03 02:12:13.518388 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:13.518371 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd-istio-podinfo\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs\" (UID: \"8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs" Nov 03 02:12:13.566681 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:13.566591 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6b5d8ddb76-qp2r8" podUID="63bedac3-9254-4e1b-adf2-8f70b6425674" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:12:13.569004 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:13.568969 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd-istio-token\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs\" (UID: \"8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs" Nov 03 02:12:13.569142 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:13.569084 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvvsg\" (UniqueName: \"kubernetes.io/projected/8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd-kube-api-access-kvvsg\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs\" (UID: \"8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs" Nov 03 02:12:13.632851 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:13.632814 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs" Nov 03 02:12:14.021639 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:14.021555 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b5d8ddb76-qp2r8"] Nov 03 02:12:14.035418 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:14.034746 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6b5d8ddb76-qp2r8" podUID="63bedac3-9254-4e1b-adf2-8f70b6425674" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:12:14.176392 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:14.176339 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs"] Nov 03 02:12:14.182060 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:12:14.182022 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e17c6ad_4cc3_4cfd_a961_c14fcb4241cd.slice/crio-98478ae1e5247e8b38d3728dab4694938569da9b1d38974212b8f0ce8bb38cbe WatchSource:0}: Error finding container 98478ae1e5247e8b38d3728dab4694938569da9b1d38974212b8f0ce8bb38cbe: Status 404 returned error can't find the container with id 98478ae1e5247e8b38d3728dab4694938569da9b1d38974212b8f0ce8bb38cbe Nov 03 02:12:14.184679 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:14.184642 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:12:14.184764 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:14.184713 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:12:14.184764 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:14.184743 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:12:14.723445 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:14.712419 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs" event={"ID":"8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd","Type":"ContainerStarted","Data":"b76c79178ef2eb3be09b50b575dcb7a45a504facd15c28ba2b061f7ea421c297"} Nov 03 02:12:14.723445 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:14.712525 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs" event={"ID":"8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd","Type":"ContainerStarted","Data":"98478ae1e5247e8b38d3728dab4694938569da9b1d38974212b8f0ce8bb38cbe"} Nov 03 02:12:15.633338 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:15.633298 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs" Nov 03 02:12:15.634763 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:15.634736 5605 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 02:12:15.634881 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:15.634789 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs" podUID="8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 02:12:16.633954 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:16.633919 5605 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 02:12:16.634416 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:16.633976 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs" podUID="8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 02:12:17.633623 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:17.633589 5605 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 02:12:17.633839 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:17.633643 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs" podUID="8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 02:12:18.633162 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:18.633130 5605 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 02:12:18.633611 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:18.633182 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs" podUID="8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 02:12:19.022376 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:19.022264 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6b5d8ddb76-qp2r8" podUID="63bedac3-9254-4e1b-adf2-8f70b6425674" containerName="payload-processing" containerID="cri-o://43aa4ec8c029664e217a022b7163a9fdf14593dab818843a67a1363afc333be4" gracePeriod=25 Nov 03 02:12:19.348524 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:19.348486 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6b5d8ddb76-qp2r8" Nov 03 02:12:19.388228 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:19.388125 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63bedac3-9254-4e1b-adf2-8f70b6425674-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/63bedac3-9254-4e1b-adf2-8f70b6425674-routing-overlay-volume\") pod \"63bedac3-9254-4e1b-adf2-8f70b6425674\" (UID: \"63bedac3-9254-4e1b-adf2-8f70b6425674\") " Nov 03 02:12:19.388467 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:19.388379 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63bedac3-9254-4e1b-adf2-8f70b6425674-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63bedac3-9254-4e1b-adf2-8f70b6425674-plugins-config-volume\") pod \"63bedac3-9254-4e1b-adf2-8f70b6425674\" (UID: \"63bedac3-9254-4e1b-adf2-8f70b6425674\") " Nov 03 02:12:19.388540 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:19.388479 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63bedac3-9254-4e1b-adf2-8f70b6425674-kube-api-access-xldtj\" (UniqueName: \"kubernetes.io/projected/63bedac3-9254-4e1b-adf2-8f70b6425674-kube-api-access-xldtj\") pod \"63bedac3-9254-4e1b-adf2-8f70b6425674\" (UID: \"63bedac3-9254-4e1b-adf2-8f70b6425674\") " Nov 03 02:12:19.388540 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:19.388483 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63bedac3-9254-4e1b-adf2-8f70b6425674-routing-overlay-volume" pod "63bedac3-9254-4e1b-adf2-8f70b6425674" (UID: "63bedac3-9254-4e1b-adf2-8f70b6425674"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:12:19.388540 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:19.388526 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63bedac3-9254-4e1b-adf2-8f70b6425674-provider-credentials\" (UniqueName: \"kubernetes.io/projected/63bedac3-9254-4e1b-adf2-8f70b6425674-provider-credentials\") pod \"63bedac3-9254-4e1b-adf2-8f70b6425674\" (UID: \"63bedac3-9254-4e1b-adf2-8f70b6425674\") " Nov 03 02:12:19.388777 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:19.388749 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63bedac3-9254-4e1b-adf2-8f70b6425674-plugins-config-volume" pod "63bedac3-9254-4e1b-adf2-8f70b6425674" (UID: "63bedac3-9254-4e1b-adf2-8f70b6425674"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:12:19.389028 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:19.389009 5605 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63bedac3-9254-4e1b-adf2-8f70b6425674-plugins-config-volume\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:12:19.389096 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:19.389034 5605 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/63bedac3-9254-4e1b-adf2-8f70b6425674-routing-overlay-volume\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:12:19.391274 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:19.391250 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63bedac3-9254-4e1b-adf2-8f70b6425674-provider-credentials" pod "63bedac3-9254-4e1b-adf2-8f70b6425674" (UID: "63bedac3-9254-4e1b-adf2-8f70b6425674"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:12:19.391395 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:19.391272 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63bedac3-9254-4e1b-adf2-8f70b6425674-kube-api-access-xldtj" pod "63bedac3-9254-4e1b-adf2-8f70b6425674" (UID: "63bedac3-9254-4e1b-adf2-8f70b6425674"). InnerVolumeSpecName "kube-api-access-xldtj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:12:19.478094 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:19.478015 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs" podStartSLOduration=6.477989375 podStartE2EDuration="6.477989375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:12:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:12:14.977197339 +0000 UTC m=+1212.236395737" watchObservedRunningTime="2026-11-03 02:12:19.477989375 +0000 UTC m=+1216.737187531" Nov 03 02:12:19.490814 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:19.489885 5605 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/63bedac3-9254-4e1b-adf2-8f70b6425674-provider-credentials\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:12:19.490814 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:19.489924 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xldtj\" (UniqueName: \"kubernetes.io/projected/63bedac3-9254-4e1b-adf2-8f70b6425674-kube-api-access-xldtj\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:12:19.633745 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:19.633650 5605 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 02:12:19.633745 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:19.633714 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs" podUID="8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 02:12:19.743349 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:19.743306 5605 generic.go:356] "Generic (PLEG): container finished" podID="63bedac3-9254-4e1b-adf2-8f70b6425674" containerID="43aa4ec8c029664e217a022b7163a9fdf14593dab818843a67a1363afc333be4" exitCode=0 Nov 03 02:12:19.743551 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:19.743379 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6b5d8ddb76-qp2r8" Nov 03 02:12:19.743551 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:19.743393 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6b5d8ddb76-qp2r8" event={"ID":"63bedac3-9254-4e1b-adf2-8f70b6425674","Type":"ContainerDied","Data":"43aa4ec8c029664e217a022b7163a9fdf14593dab818843a67a1363afc333be4"} Nov 03 02:12:19.743551 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:19.743482 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6b5d8ddb76-qp2r8" event={"ID":"63bedac3-9254-4e1b-adf2-8f70b6425674","Type":"ContainerDied","Data":"d2a0e07fa2fe37fe65010de3e7b9d55d5ce3de7d7ed7311ccb017ee104f97890"} Nov 03 02:12:19.743551 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:19.743505 5605 scope.go:122] "RemoveContainer" containerID="43aa4ec8c029664e217a022b7163a9fdf14593dab818843a67a1363afc333be4" Nov 03 02:12:19.753828 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:19.753807 5605 scope.go:122] "RemoveContainer" containerID="43aa4ec8c029664e217a022b7163a9fdf14593dab818843a67a1363afc333be4" Nov 03 02:12:19.754135 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:12:19.754114 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43aa4ec8c029664e217a022b7163a9fdf14593dab818843a67a1363afc333be4\": container with ID starting with 43aa4ec8c029664e217a022b7163a9fdf14593dab818843a67a1363afc333be4 not found: ID does not exist" containerID="43aa4ec8c029664e217a022b7163a9fdf14593dab818843a67a1363afc333be4" Nov 03 02:12:19.754177 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:19.754148 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43aa4ec8c029664e217a022b7163a9fdf14593dab818843a67a1363afc333be4"} err="failed to get container status \"43aa4ec8c029664e217a022b7163a9fdf14593dab818843a67a1363afc333be4\": rpc error: code = NotFound desc = could not find container \"43aa4ec8c029664e217a022b7163a9fdf14593dab818843a67a1363afc333be4\": container with ID starting with 43aa4ec8c029664e217a022b7163a9fdf14593dab818843a67a1363afc333be4 not found: ID does not exist" Nov 03 02:12:19.898531 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:19.898434 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b5d8ddb76-qp2r8"] Nov 03 02:12:19.899776 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:19.899750 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b5d8ddb76-qp2r8"] Nov 03 02:12:20.634032 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:20.633995 5605 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 02:12:20.634444 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:20.634064 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs" podUID="8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 02:12:20.851342 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:20.851297 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63bedac3-9254-4e1b-adf2-8f70b6425674" path="/var/lib/kubelet/pods/63bedac3-9254-4e1b-adf2-8f70b6425674/volumes" Nov 03 02:12:22.061415 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:22.061346 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs" Nov 03 02:12:22.062183 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:22.062162 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs" Nov 03 02:12:22.062940 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:22.062921 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs" Nov 03 02:12:23.914689 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:12:23.914646 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:12:32.769562 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:32.769493 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a0e72e/premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv" podUID="a9b9af44-ab6d-4caf-89af-c986ea25b4c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:12:32.803655 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:32.803591 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-521f1f/simulator-w2-521f1f-kserve-5f47b86f6-mgksf" podUID="6e1b7fda-0ade-4042-ad63-5115bc66b454" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:12:32.927459 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:32.924475 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-521f1f/premium-w2-521f1f-kserve-7578fb7957-89wfb" podUID="3d988573-d8b1-43ba-a423-51a63ca12337" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:12:33.086511 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:33.086456 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1bccbd/distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg" podUID="5a71cdee-425e-4787-9217-5fb6b97e232b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:12:39.953196 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:39.953100 5605 generic.go:356] "Generic (PLEG): container finished" podID="3aa07ac7-fa23-451b-a6a6-0bea3103c9d1" containerID="15d7220b13fb52a80c5cf245b3f6ee02c489b270610086b5e918c017e3b411d6" exitCode=0 Nov 03 02:12:39.953196 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:39.953152 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf" event={"ID":"3aa07ac7-fa23-451b-a6a6-0bea3103c9d1","Type":"ContainerDied","Data":"15d7220b13fb52a80c5cf245b3f6ee02c489b270610086b5e918c017e3b411d6"} Nov 03 02:12:39.953657 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:39.953579 5605 scope.go:122] "RemoveContainer" containerID="15d7220b13fb52a80c5cf245b3f6ee02c489b270610086b5e918c017e3b411d6" Nov 03 02:12:39.955159 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:39.955121 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:12:39.955236 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:39.955217 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:12:39.955287 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:39.955271 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:12:40.987175 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:40.987134 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf" event={"ID":"3aa07ac7-fa23-451b-a6a6-0bea3103c9d1","Type":"ContainerStarted","Data":"c6a47c13c3ccc2ec6b7381ceaaff71c22e6c72edbb545caace97a64e144c4a06"} Nov 03 02:12:41.424216 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:41.424177 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf" Nov 03 02:12:41.424438 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:41.424415 5605 patch_prober.go:29] interesting pod/e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 02:12:41.424494 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:41.424463 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf" podUID="3aa07ac7-fa23-451b-a6a6-0bea3103c9d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 02:12:41.880059 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:41.880017 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3632597-bbsbx"] Nov 03 02:12:41.880828 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:41.880800 5605 cpu_manager.go:414] "containerMap: removing container" podUID="63bedac3-9254-4e1b-adf2-8f70b6425674" containerName="payload-processing" Nov 03 02:12:41.880828 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:41.880829 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63bedac3-9254-4e1b-adf2-8f70b6425674" containerName="payload-processing" Nov 03 02:12:41.881042 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:41.880989 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="63bedac3-9254-4e1b-adf2-8f70b6425674" containerName="payload-processing" Nov 03 02:12:41.884769 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:41.884751 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3632597-bbsbx" Nov 03 02:12:41.907756 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:41.907721 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krrm9\" (UniqueName: \"kubernetes.io/projected/842dabe0-fc9a-4a1f-a923-ad77169f3ae9-kube-api-access-krrm9\") pod \"maas-api-revoke-sub-b3632597-bbsbx\" (UID: \"842dabe0-fc9a-4a1f-a923-ad77169f3ae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3632597-bbsbx" Nov 03 02:12:41.907925 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:41.907870 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/842dabe0-fc9a-4a1f-a923-ad77169f3ae9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3632597-bbsbx\" (UID: \"842dabe0-fc9a-4a1f-a923-ad77169f3ae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3632597-bbsbx" Nov 03 02:12:41.907983 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:41.907936 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/842dabe0-fc9a-4a1f-a923-ad77169f3ae9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3632597-bbsbx\" (UID: \"842dabe0-fc9a-4a1f-a923-ad77169f3ae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3632597-bbsbx" Nov 03 02:12:41.933606 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:41.933565 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:12:41.933852 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:41.933828 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:12:41.933976 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:41.933898 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:12:41.934043 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:41.933977 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:12:41.934477 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:41.934445 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3632597-bbsbx"] Nov 03 02:12:42.009601 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:42.009553 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/842dabe0-fc9a-4a1f-a923-ad77169f3ae9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3632597-bbsbx\" (UID: \"842dabe0-fc9a-4a1f-a923-ad77169f3ae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3632597-bbsbx" Nov 03 02:12:42.009991 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:42.009637 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/842dabe0-fc9a-4a1f-a923-ad77169f3ae9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3632597-bbsbx\" (UID: \"842dabe0-fc9a-4a1f-a923-ad77169f3ae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3632597-bbsbx" Nov 03 02:12:42.009991 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:42.009705 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krrm9\" (UniqueName: \"kubernetes.io/projected/842dabe0-fc9a-4a1f-a923-ad77169f3ae9-kube-api-access-krrm9\") pod \"maas-api-revoke-sub-b3632597-bbsbx\" (UID: \"842dabe0-fc9a-4a1f-a923-ad77169f3ae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3632597-bbsbx" Nov 03 02:12:42.010312 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:42.010292 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/842dabe0-fc9a-4a1f-a923-ad77169f3ae9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3632597-bbsbx\" (UID: \"842dabe0-fc9a-4a1f-a923-ad77169f3ae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3632597-bbsbx" Nov 03 02:12:42.011957 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:42.011933 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/842dabe0-fc9a-4a1f-a923-ad77169f3ae9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3632597-bbsbx\" (UID: \"842dabe0-fc9a-4a1f-a923-ad77169f3ae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3632597-bbsbx" Nov 03 02:12:42.082166 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:42.082132 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krrm9\" (UniqueName: \"kubernetes.io/projected/842dabe0-fc9a-4a1f-a923-ad77169f3ae9-kube-api-access-krrm9\") pod \"maas-api-revoke-sub-b3632597-bbsbx\" (UID: \"842dabe0-fc9a-4a1f-a923-ad77169f3ae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3632597-bbsbx" Nov 03 02:12:42.198023 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:42.197934 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3632597-bbsbx" Nov 03 02:12:42.423620 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:42.423584 5605 patch_prober.go:29] interesting pod/e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 02:12:42.423797 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:42.423643 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf" podUID="3aa07ac7-fa23-451b-a6a6-0bea3103c9d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 02:12:42.601608 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:42.601566 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3632597-bbsbx"] Nov 03 02:12:42.606732 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:12:42.606696 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod842dabe0_fc9a_4a1f_a923_ad77169f3ae9.slice/crio-05b6c09d869a7d3e219b3de95c06f023a82ce05eea6ff0af37ea80c75820c5aa WatchSource:0}: Error finding container 05b6c09d869a7d3e219b3de95c06f023a82ce05eea6ff0af37ea80c75820c5aa: Status 404 returned error can't find the container with id 05b6c09d869a7d3e219b3de95c06f023a82ce05eea6ff0af37ea80c75820c5aa Nov 03 02:12:43.002906 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:43.002871 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3632597-bbsbx" event={"ID":"842dabe0-fc9a-4a1f-a923-ad77169f3ae9","Type":"ContainerStarted","Data":"57803f4c0d2e7b43ace6ac24cd0077922f6862d4c116c3e3bd131d7f27043eaf"} Nov 03 02:12:43.003116 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:43.002935 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3632597-bbsbx" event={"ID":"842dabe0-fc9a-4a1f-a923-ad77169f3ae9","Type":"ContainerStarted","Data":"05b6c09d869a7d3e219b3de95c06f023a82ce05eea6ff0af37ea80c75820c5aa"} Nov 03 02:12:43.109960 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:43.109903 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3632597-bbsbx" podStartSLOduration=2.10988796 podStartE2EDuration="2.10988796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:12:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:12:43.107480071 +0000 UTC m=+1240.366678248" watchObservedRunningTime="2026-11-03 02:12:43.10988796 +0000 UTC m=+1240.369086114" Nov 03 02:12:43.423705 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:43.423665 5605 patch_prober.go:29] interesting pod/e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 02:12:43.423889 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:43.423723 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf" podUID="3aa07ac7-fa23-451b-a6a6-0bea3103c9d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 02:12:44.045435 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:44.041375 5605 generic.go:356] "Generic (PLEG): container finished" podID="842dabe0-fc9a-4a1f-a923-ad77169f3ae9" containerID="57803f4c0d2e7b43ace6ac24cd0077922f6862d4c116c3e3bd131d7f27043eaf" exitCode=0 Nov 03 02:12:44.045435 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:44.041559 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3632597-bbsbx" event={"ID":"842dabe0-fc9a-4a1f-a923-ad77169f3ae9","Type":"ContainerDied","Data":"57803f4c0d2e7b43ace6ac24cd0077922f6862d4c116c3e3bd131d7f27043eaf"} Nov 03 02:12:44.423631 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:44.423597 5605 patch_prober.go:29] interesting pod/e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 02:12:44.423990 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:44.423654 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf" podUID="3aa07ac7-fa23-451b-a6a6-0bea3103c9d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 02:12:45.053230 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:45.053197 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:12:45.229537 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:45.229510 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3632597-bbsbx" Nov 03 02:12:45.296214 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:45.296177 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/842dabe0-fc9a-4a1f-a923-ad77169f3ae9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/842dabe0-fc9a-4a1f-a923-ad77169f3ae9-maas-api-xks-ca\") pod \"842dabe0-fc9a-4a1f-a923-ad77169f3ae9\" (UID: \"842dabe0-fc9a-4a1f-a923-ad77169f3ae9\") " Nov 03 02:12:45.296435 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:45.296262 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/842dabe0-fc9a-4a1f-a923-ad77169f3ae9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/842dabe0-fc9a-4a1f-a923-ad77169f3ae9-maas-api-openshift-ca\") pod \"842dabe0-fc9a-4a1f-a923-ad77169f3ae9\" (UID: \"842dabe0-fc9a-4a1f-a923-ad77169f3ae9\") " Nov 03 02:12:45.296435 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:45.296315 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/842dabe0-fc9a-4a1f-a923-ad77169f3ae9-kube-api-access-krrm9\" (UniqueName: \"kubernetes.io/projected/842dabe0-fc9a-4a1f-a923-ad77169f3ae9-kube-api-access-krrm9\") pod \"842dabe0-fc9a-4a1f-a923-ad77169f3ae9\" (UID: \"842dabe0-fc9a-4a1f-a923-ad77169f3ae9\") " Nov 03 02:12:45.296681 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:45.296655 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/842dabe0-fc9a-4a1f-a923-ad77169f3ae9-maas-api-openshift-ca" pod "842dabe0-fc9a-4a1f-a923-ad77169f3ae9" (UID: "842dabe0-fc9a-4a1f-a923-ad77169f3ae9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:12:45.296856 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:45.296835 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/842dabe0-fc9a-4a1f-a923-ad77169f3ae9-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:12:45.298332 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:45.298279 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/842dabe0-fc9a-4a1f-a923-ad77169f3ae9-maas-api-xks-ca" pod "842dabe0-fc9a-4a1f-a923-ad77169f3ae9" (UID: "842dabe0-fc9a-4a1f-a923-ad77169f3ae9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:12:45.298491 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:45.298476 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/842dabe0-fc9a-4a1f-a923-ad77169f3ae9-kube-api-access-krrm9" pod "842dabe0-fc9a-4a1f-a923-ad77169f3ae9" (UID: "842dabe0-fc9a-4a1f-a923-ad77169f3ae9"). InnerVolumeSpecName "kube-api-access-krrm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:12:45.397717 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:45.397673 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/842dabe0-fc9a-4a1f-a923-ad77169f3ae9-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:12:45.397717 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:45.397705 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krrm9\" (UniqueName: \"kubernetes.io/projected/842dabe0-fc9a-4a1f-a923-ad77169f3ae9-kube-api-access-krrm9\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:12:45.424533 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:45.424497 5605 patch_prober.go:29] interesting pod/e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 02:12:45.424939 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:45.424548 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf" podUID="3aa07ac7-fa23-451b-a6a6-0bea3103c9d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 02:12:46.059381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:46.057903 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3632597-bbsbx" event={"ID":"842dabe0-fc9a-4a1f-a923-ad77169f3ae9","Type":"ContainerDied","Data":"05b6c09d869a7d3e219b3de95c06f023a82ce05eea6ff0af37ea80c75820c5aa"} Nov 03 02:12:46.059381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:46.058083 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05b6c09d869a7d3e219b3de95c06f023a82ce05eea6ff0af37ea80c75820c5aa" Nov 03 02:12:46.059381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:46.058158 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3632597-bbsbx" Nov 03 02:12:46.597384 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:46.588002 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf" Nov 03 02:12:46.597384 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:46.589888 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf" Nov 03 02:12:46.611380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:46.604647 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf" Nov 03 02:12:53.912939 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:12:53.912895 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:12:54.129857 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:54.129822 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-a0e72e/distinct-w3-a0e72e-kserve-fb499cc4-2dfxc"] Nov 03 02:12:54.130432 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:54.130413 5605 cpu_manager.go:414] "containerMap: removing container" podUID="842dabe0-fc9a-4a1f-a923-ad77169f3ae9" containerName="revoke-keys" Nov 03 02:12:54.130487 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:54.130433 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="842dabe0-fc9a-4a1f-a923-ad77169f3ae9" containerName="revoke-keys" Nov 03 02:12:54.130536 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:54.130526 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="842dabe0-fc9a-4a1f-a923-ad77169f3ae9" containerName="revoke-keys" Nov 03 02:12:54.133918 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:54.133899 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a0e72e/distinct-w3-a0e72e-kserve-fb499cc4-2dfxc" Nov 03 02:12:54.231430 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:54.231314 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-a0e72e\"/\"distinct-w3-a0e72e-kserve-self-signed-certs\"" Nov 03 02:12:54.233955 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:54.233932 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-a0e72e/distinct-w3-a0e72e-kserve-fb499cc4-2dfxc"] Nov 03 02:12:54.257206 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:54.257173 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/308bca3b-9bd5-4724-9b95-acdbc978aa10-home\") pod \"distinct-w3-a0e72e-kserve-fb499cc4-2dfxc\" (UID: \"308bca3b-9bd5-4724-9b95-acdbc978aa10\") " pod="e2e-models-e2e-worker-w3-a0e72e/distinct-w3-a0e72e-kserve-fb499cc4-2dfxc" Nov 03 02:12:54.257345 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:54.257236 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/308bca3b-9bd5-4724-9b95-acdbc978aa10-model-cache\") pod \"distinct-w3-a0e72e-kserve-fb499cc4-2dfxc\" (UID: \"308bca3b-9bd5-4724-9b95-acdbc978aa10\") " pod="e2e-models-e2e-worker-w3-a0e72e/distinct-w3-a0e72e-kserve-fb499cc4-2dfxc" Nov 03 02:12:54.257434 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:54.257403 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx6w7\" (UniqueName: \"kubernetes.io/projected/308bca3b-9bd5-4724-9b95-acdbc978aa10-kube-api-access-wx6w7\") pod \"distinct-w3-a0e72e-kserve-fb499cc4-2dfxc\" (UID: \"308bca3b-9bd5-4724-9b95-acdbc978aa10\") " pod="e2e-models-e2e-worker-w3-a0e72e/distinct-w3-a0e72e-kserve-fb499cc4-2dfxc" Nov 03 02:12:54.257475 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:54.257438 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/308bca3b-9bd5-4724-9b95-acdbc978aa10-dshm\") pod \"distinct-w3-a0e72e-kserve-fb499cc4-2dfxc\" (UID: \"308bca3b-9bd5-4724-9b95-acdbc978aa10\") " pod="e2e-models-e2e-worker-w3-a0e72e/distinct-w3-a0e72e-kserve-fb499cc4-2dfxc" Nov 03 02:12:54.257475 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:54.257459 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/308bca3b-9bd5-4724-9b95-acdbc978aa10-tmp-dir\") pod \"distinct-w3-a0e72e-kserve-fb499cc4-2dfxc\" (UID: \"308bca3b-9bd5-4724-9b95-acdbc978aa10\") " pod="e2e-models-e2e-worker-w3-a0e72e/distinct-w3-a0e72e-kserve-fb499cc4-2dfxc" Nov 03 02:12:54.257604 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:54.257590 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/308bca3b-9bd5-4724-9b95-acdbc978aa10-tls-certs\") pod \"distinct-w3-a0e72e-kserve-fb499cc4-2dfxc\" (UID: \"308bca3b-9bd5-4724-9b95-acdbc978aa10\") " pod="e2e-models-e2e-worker-w3-a0e72e/distinct-w3-a0e72e-kserve-fb499cc4-2dfxc" Nov 03 02:12:54.359411 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:54.359330 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/308bca3b-9bd5-4724-9b95-acdbc978aa10-tls-certs\") pod \"distinct-w3-a0e72e-kserve-fb499cc4-2dfxc\" (UID: \"308bca3b-9bd5-4724-9b95-acdbc978aa10\") " pod="e2e-models-e2e-worker-w3-a0e72e/distinct-w3-a0e72e-kserve-fb499cc4-2dfxc" Nov 03 02:12:54.360886 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:54.360859 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/308bca3b-9bd5-4724-9b95-acdbc978aa10-home\") pod \"distinct-w3-a0e72e-kserve-fb499cc4-2dfxc\" (UID: \"308bca3b-9bd5-4724-9b95-acdbc978aa10\") " pod="e2e-models-e2e-worker-w3-a0e72e/distinct-w3-a0e72e-kserve-fb499cc4-2dfxc" Nov 03 02:12:54.361108 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:54.361092 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/308bca3b-9bd5-4724-9b95-acdbc978aa10-model-cache\") pod \"distinct-w3-a0e72e-kserve-fb499cc4-2dfxc\" (UID: \"308bca3b-9bd5-4724-9b95-acdbc978aa10\") " pod="e2e-models-e2e-worker-w3-a0e72e/distinct-w3-a0e72e-kserve-fb499cc4-2dfxc" Nov 03 02:12:54.363990 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:54.361339 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx6w7\" (UniqueName: \"kubernetes.io/projected/308bca3b-9bd5-4724-9b95-acdbc978aa10-kube-api-access-wx6w7\") pod \"distinct-w3-a0e72e-kserve-fb499cc4-2dfxc\" (UID: \"308bca3b-9bd5-4724-9b95-acdbc978aa10\") " pod="e2e-models-e2e-worker-w3-a0e72e/distinct-w3-a0e72e-kserve-fb499cc4-2dfxc" Nov 03 02:12:54.364419 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:54.364389 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/308bca3b-9bd5-4724-9b95-acdbc978aa10-dshm\") pod \"distinct-w3-a0e72e-kserve-fb499cc4-2dfxc\" (UID: \"308bca3b-9bd5-4724-9b95-acdbc978aa10\") " pod="e2e-models-e2e-worker-w3-a0e72e/distinct-w3-a0e72e-kserve-fb499cc4-2dfxc" Nov 03 02:12:54.364972 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:54.364950 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/308bca3b-9bd5-4724-9b95-acdbc978aa10-tmp-dir\") pod \"distinct-w3-a0e72e-kserve-fb499cc4-2dfxc\" (UID: \"308bca3b-9bd5-4724-9b95-acdbc978aa10\") " pod="e2e-models-e2e-worker-w3-a0e72e/distinct-w3-a0e72e-kserve-fb499cc4-2dfxc" Nov 03 02:12:54.371648 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:54.361771 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/308bca3b-9bd5-4724-9b95-acdbc978aa10-model-cache\") pod \"distinct-w3-a0e72e-kserve-fb499cc4-2dfxc\" (UID: \"308bca3b-9bd5-4724-9b95-acdbc978aa10\") " pod="e2e-models-e2e-worker-w3-a0e72e/distinct-w3-a0e72e-kserve-fb499cc4-2dfxc" Nov 03 02:12:54.371805 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:54.371474 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/308bca3b-9bd5-4724-9b95-acdbc978aa10-tmp-dir\") pod \"distinct-w3-a0e72e-kserve-fb499cc4-2dfxc\" (UID: \"308bca3b-9bd5-4724-9b95-acdbc978aa10\") " pod="e2e-models-e2e-worker-w3-a0e72e/distinct-w3-a0e72e-kserve-fb499cc4-2dfxc" Nov 03 02:12:54.386104 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:54.362622 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/308bca3b-9bd5-4724-9b95-acdbc978aa10-home\") pod \"distinct-w3-a0e72e-kserve-fb499cc4-2dfxc\" (UID: \"308bca3b-9bd5-4724-9b95-acdbc978aa10\") " pod="e2e-models-e2e-worker-w3-a0e72e/distinct-w3-a0e72e-kserve-fb499cc4-2dfxc" Nov 03 02:12:54.386323 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:54.384999 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/308bca3b-9bd5-4724-9b95-acdbc978aa10-tls-certs\") pod \"distinct-w3-a0e72e-kserve-fb499cc4-2dfxc\" (UID: \"308bca3b-9bd5-4724-9b95-acdbc978aa10\") " pod="e2e-models-e2e-worker-w3-a0e72e/distinct-w3-a0e72e-kserve-fb499cc4-2dfxc" Nov 03 02:12:54.404772 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:54.394726 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/308bca3b-9bd5-4724-9b95-acdbc978aa10-dshm\") pod \"distinct-w3-a0e72e-kserve-fb499cc4-2dfxc\" (UID: \"308bca3b-9bd5-4724-9b95-acdbc978aa10\") " pod="e2e-models-e2e-worker-w3-a0e72e/distinct-w3-a0e72e-kserve-fb499cc4-2dfxc" Nov 03 02:12:54.415672 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:54.415638 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx6w7\" (UniqueName: \"kubernetes.io/projected/308bca3b-9bd5-4724-9b95-acdbc978aa10-kube-api-access-wx6w7\") pod \"distinct-w3-a0e72e-kserve-fb499cc4-2dfxc\" (UID: \"308bca3b-9bd5-4724-9b95-acdbc978aa10\") " pod="e2e-models-e2e-worker-w3-a0e72e/distinct-w3-a0e72e-kserve-fb499cc4-2dfxc" Nov 03 02:12:54.445928 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:54.445890 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a0e72e/distinct-w3-a0e72e-kserve-fb499cc4-2dfxc" Nov 03 02:12:54.989741 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:54.973965 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-a0e72e/distinct-w3-a0e72e-kserve-fb499cc4-2dfxc"] Nov 03 02:12:55.203381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:55.201034 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a0e72e/distinct-w3-a0e72e-kserve-fb499cc4-2dfxc" event={"ID":"308bca3b-9bd5-4724-9b95-acdbc978aa10","Type":"ContainerStarted","Data":"1aa09c6571bd9571b9f8b05e504c098aab7c3adeae9acd73269599620cb9d477"} Nov 03 02:12:55.203381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:55.201095 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a0e72e/distinct-w3-a0e72e-kserve-fb499cc4-2dfxc" event={"ID":"308bca3b-9bd5-4724-9b95-acdbc978aa10","Type":"ContainerStarted","Data":"1a8528c05512f2b4fab4bab9ec5aaab81ef8be9d968ec01b1deb2806c911771a"} Nov 03 02:12:55.310233 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:55.310180 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-a0e72e/distinct-w3-a0e72e-kserve-fb499cc4-2dfxc" podStartSLOduration=1.3101665 podStartE2EDuration="1.3101665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:12:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:12:55.308588161 +0000 UTC m=+1252.567786316" watchObservedRunningTime="2026-11-03 02:12:55.3101665 +0000 UTC m=+1252.569364653" Nov 03 02:12:55.447523 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:12:55.447454 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a0e72e/distinct-w3-a0e72e-kserve-fb499cc4-2dfxc" Nov 03 02:13:00.205153 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:00.205104 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbceb3f9-kvzfx"] Nov 03 02:13:00.209313 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:00.209288 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbceb3f9-kvzfx" Nov 03 02:13:00.259538 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:00.259494 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:13:00.306207 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:00.306165 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:13:00.306207 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:00.306190 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:13:00.306207 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:00.306178 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:13:00.309137 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:00.309106 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbceb3f9-kvzfx"] Nov 03 02:13:00.319459 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:00.319417 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l4qj\" (UniqueName: \"kubernetes.io/projected/0d3aea59-b051-4377-8108-73a7d958922f-kube-api-access-2l4qj\") pod \"maas-api-revoke-sub-cbceb3f9-kvzfx\" (UID: \"0d3aea59-b051-4377-8108-73a7d958922f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbceb3f9-kvzfx" Nov 03 02:13:00.319636 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:00.319619 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d3aea59-b051-4377-8108-73a7d958922f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbceb3f9-kvzfx\" (UID: \"0d3aea59-b051-4377-8108-73a7d958922f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbceb3f9-kvzfx" Nov 03 02:13:00.319753 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:00.319733 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d3aea59-b051-4377-8108-73a7d958922f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbceb3f9-kvzfx\" (UID: \"0d3aea59-b051-4377-8108-73a7d958922f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbceb3f9-kvzfx" Nov 03 02:13:00.421297 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:00.421254 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d3aea59-b051-4377-8108-73a7d958922f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbceb3f9-kvzfx\" (UID: \"0d3aea59-b051-4377-8108-73a7d958922f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbceb3f9-kvzfx" Nov 03 02:13:00.421548 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:00.421419 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l4qj\" (UniqueName: \"kubernetes.io/projected/0d3aea59-b051-4377-8108-73a7d958922f-kube-api-access-2l4qj\") pod \"maas-api-revoke-sub-cbceb3f9-kvzfx\" (UID: \"0d3aea59-b051-4377-8108-73a7d958922f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbceb3f9-kvzfx" Nov 03 02:13:00.421618 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:00.421564 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d3aea59-b051-4377-8108-73a7d958922f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbceb3f9-kvzfx\" (UID: \"0d3aea59-b051-4377-8108-73a7d958922f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbceb3f9-kvzfx" Nov 03 02:13:00.421946 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:00.421922 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d3aea59-b051-4377-8108-73a7d958922f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbceb3f9-kvzfx\" (UID: \"0d3aea59-b051-4377-8108-73a7d958922f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbceb3f9-kvzfx" Nov 03 02:13:00.423935 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:00.423907 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d3aea59-b051-4377-8108-73a7d958922f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbceb3f9-kvzfx\" (UID: \"0d3aea59-b051-4377-8108-73a7d958922f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbceb3f9-kvzfx" Nov 03 02:13:00.474736 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:00.474653 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l4qj\" (UniqueName: \"kubernetes.io/projected/0d3aea59-b051-4377-8108-73a7d958922f-kube-api-access-2l4qj\") pod \"maas-api-revoke-sub-cbceb3f9-kvzfx\" (UID: \"0d3aea59-b051-4377-8108-73a7d958922f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbceb3f9-kvzfx" Nov 03 02:13:00.524088 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:00.524049 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbceb3f9-kvzfx" Nov 03 02:13:00.899499 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:00.899465 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbceb3f9-kvzfx"] Nov 03 02:13:00.905567 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:13:00.905514 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d3aea59_b051_4377_8108_73a7d958922f.slice/crio-f54742c6a6e4aa24e9dfe208aafa7d97911273b6b500a6fde6f5801f31aebaff WatchSource:0}: Error finding container f54742c6a6e4aa24e9dfe208aafa7d97911273b6b500a6fde6f5801f31aebaff: Status 404 returned error can't find the container with id f54742c6a6e4aa24e9dfe208aafa7d97911273b6b500a6fde6f5801f31aebaff Nov 03 02:13:01.245289 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:01.245235 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbceb3f9-kvzfx" event={"ID":"0d3aea59-b051-4377-8108-73a7d958922f","Type":"ContainerStarted","Data":"d791d936e02ed1fda40b7fbfa99f5f55a66ed1de8573878bda394a3846a9b3c0"} Nov 03 02:13:01.245700 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:01.245305 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbceb3f9-kvzfx" event={"ID":"0d3aea59-b051-4377-8108-73a7d958922f","Type":"ContainerStarted","Data":"f54742c6a6e4aa24e9dfe208aafa7d97911273b6b500a6fde6f5801f31aebaff"} Nov 03 02:13:01.359848 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:01.359779 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbceb3f9-kvzfx" podStartSLOduration=1.359756493 podStartE2EDuration="1.359756493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:13:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:13:01.356095087 +0000 UTC m=+1258.615293241" watchObservedRunningTime="2026-11-03 02:13:01.359756493 +0000 UTC m=+1258.618954649" Nov 03 02:13:02.252998 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:02.252962 5605 generic.go:356] "Generic (PLEG): container finished" podID="0d3aea59-b051-4377-8108-73a7d958922f" containerID="d791d936e02ed1fda40b7fbfa99f5f55a66ed1de8573878bda394a3846a9b3c0" exitCode=0 Nov 03 02:13:02.253474 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:02.253052 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbceb3f9-kvzfx" event={"ID":"0d3aea59-b051-4377-8108-73a7d958922f","Type":"ContainerDied","Data":"d791d936e02ed1fda40b7fbfa99f5f55a66ed1de8573878bda394a3846a9b3c0"} Nov 03 02:13:03.663117 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:03.663087 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbceb3f9-kvzfx" Nov 03 02:13:03.705955 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:03.705921 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d3aea59-b051-4377-8108-73a7d958922f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d3aea59-b051-4377-8108-73a7d958922f-maas-api-openshift-ca\") pod \"0d3aea59-b051-4377-8108-73a7d958922f\" (UID: \"0d3aea59-b051-4377-8108-73a7d958922f\") " Nov 03 02:13:03.706125 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:03.706018 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d3aea59-b051-4377-8108-73a7d958922f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d3aea59-b051-4377-8108-73a7d958922f-maas-api-xks-ca\") pod \"0d3aea59-b051-4377-8108-73a7d958922f\" (UID: \"0d3aea59-b051-4377-8108-73a7d958922f\") " Nov 03 02:13:03.706125 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:03.706055 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d3aea59-b051-4377-8108-73a7d958922f-kube-api-access-2l4qj\" (UniqueName: \"kubernetes.io/projected/0d3aea59-b051-4377-8108-73a7d958922f-kube-api-access-2l4qj\") pod \"0d3aea59-b051-4377-8108-73a7d958922f\" (UID: \"0d3aea59-b051-4377-8108-73a7d958922f\") " Nov 03 02:13:03.706374 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:03.706325 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d3aea59-b051-4377-8108-73a7d958922f-maas-api-openshift-ca" pod "0d3aea59-b051-4377-8108-73a7d958922f" (UID: "0d3aea59-b051-4377-8108-73a7d958922f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:13:03.706486 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:03.706472 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d3aea59-b051-4377-8108-73a7d958922f-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:13:03.708137 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:03.708110 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d3aea59-b051-4377-8108-73a7d958922f-maas-api-xks-ca" pod "0d3aea59-b051-4377-8108-73a7d958922f" (UID: "0d3aea59-b051-4377-8108-73a7d958922f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:13:03.708235 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:03.708161 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d3aea59-b051-4377-8108-73a7d958922f-kube-api-access-2l4qj" pod "0d3aea59-b051-4377-8108-73a7d958922f" (UID: "0d3aea59-b051-4377-8108-73a7d958922f"). InnerVolumeSpecName "kube-api-access-2l4qj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:13:03.807929 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:03.807884 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d3aea59-b051-4377-8108-73a7d958922f-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:13:03.807929 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:03.807929 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2l4qj\" (UniqueName: \"kubernetes.io/projected/0d3aea59-b051-4377-8108-73a7d958922f-kube-api-access-2l4qj\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:13:04.266123 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:04.266091 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbceb3f9-kvzfx" Nov 03 02:13:04.266318 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:04.266124 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbceb3f9-kvzfx" event={"ID":"0d3aea59-b051-4377-8108-73a7d958922f","Type":"ContainerDied","Data":"f54742c6a6e4aa24e9dfe208aafa7d97911273b6b500a6fde6f5801f31aebaff"} Nov 03 02:13:04.266318 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:04.266211 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f54742c6a6e4aa24e9dfe208aafa7d97911273b6b500a6fde6f5801f31aebaff" Nov 03 02:13:04.470391 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:04.465325 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-a0e72e/distinct-w3-a0e72e-kserve-fb499cc4-2dfxc" Nov 03 02:13:04.511384 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:04.510916 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-a0e72e/distinct-w3-a0e72e-kserve-fb499cc4-2dfxc" Nov 03 02:13:05.283930 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:05.283893 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a0e72e/distinct-w3-a0e72e-kserve-fb499cc4-2dfxc" Nov 03 02:13:22.302441 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:22.302350 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54abb9d6-5tzw2"] Nov 03 02:13:22.302864 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:22.302850 5605 cpu_manager.go:414] "containerMap: removing container" podUID="0d3aea59-b051-4377-8108-73a7d958922f" containerName="revoke-keys" Nov 03 02:13:22.302919 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:22.302866 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d3aea59-b051-4377-8108-73a7d958922f" containerName="revoke-keys" Nov 03 02:13:22.302960 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:22.302955 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d3aea59-b051-4377-8108-73a7d958922f" containerName="revoke-keys" Nov 03 02:13:22.307597 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:22.307563 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54abb9d6-5tzw2" Nov 03 02:13:22.360506 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:22.360477 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54abb9d6-5tzw2"] Nov 03 02:13:22.385588 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:22.385554 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd2b7\" (UniqueName: \"kubernetes.io/projected/a60fd0a2-31e0-4549-98d9-74c013efb7cf-kube-api-access-qd2b7\") pod \"maas-api-revoke-sub-54abb9d6-5tzw2\" (UID: \"a60fd0a2-31e0-4549-98d9-74c013efb7cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54abb9d6-5tzw2" Nov 03 02:13:22.385723 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:22.385606 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a60fd0a2-31e0-4549-98d9-74c013efb7cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54abb9d6-5tzw2\" (UID: \"a60fd0a2-31e0-4549-98d9-74c013efb7cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54abb9d6-5tzw2" Nov 03 02:13:22.385831 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:22.385812 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a60fd0a2-31e0-4549-98d9-74c013efb7cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54abb9d6-5tzw2\" (UID: \"a60fd0a2-31e0-4549-98d9-74c013efb7cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54abb9d6-5tzw2" Nov 03 02:13:22.403370 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:22.403341 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:13:22.403496 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:22.403402 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:13:22.403496 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:22.403439 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:13:22.403626 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:22.403611 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:13:22.487266 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:22.487242 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd2b7\" (UniqueName: \"kubernetes.io/projected/a60fd0a2-31e0-4549-98d9-74c013efb7cf-kube-api-access-qd2b7\") pod \"maas-api-revoke-sub-54abb9d6-5tzw2\" (UID: \"a60fd0a2-31e0-4549-98d9-74c013efb7cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54abb9d6-5tzw2" Nov 03 02:13:22.487421 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:22.487274 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a60fd0a2-31e0-4549-98d9-74c013efb7cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54abb9d6-5tzw2\" (UID: \"a60fd0a2-31e0-4549-98d9-74c013efb7cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54abb9d6-5tzw2" Nov 03 02:13:22.487421 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:22.487392 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a60fd0a2-31e0-4549-98d9-74c013efb7cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54abb9d6-5tzw2\" (UID: \"a60fd0a2-31e0-4549-98d9-74c013efb7cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54abb9d6-5tzw2" Nov 03 02:13:22.488164 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:22.488134 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a60fd0a2-31e0-4549-98d9-74c013efb7cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54abb9d6-5tzw2\" (UID: \"a60fd0a2-31e0-4549-98d9-74c013efb7cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54abb9d6-5tzw2" Nov 03 02:13:22.489625 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:22.489607 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a60fd0a2-31e0-4549-98d9-74c013efb7cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54abb9d6-5tzw2\" (UID: \"a60fd0a2-31e0-4549-98d9-74c013efb7cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54abb9d6-5tzw2" Nov 03 02:13:22.540425 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:22.540392 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd2b7\" (UniqueName: \"kubernetes.io/projected/a60fd0a2-31e0-4549-98d9-74c013efb7cf-kube-api-access-qd2b7\") pod \"maas-api-revoke-sub-54abb9d6-5tzw2\" (UID: \"a60fd0a2-31e0-4549-98d9-74c013efb7cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54abb9d6-5tzw2" Nov 03 02:13:22.625302 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:22.625248 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54abb9d6-5tzw2" Nov 03 02:13:22.992609 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:22.992538 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54abb9d6-5tzw2"] Nov 03 02:13:22.997884 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:13:22.997855 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda60fd0a2_31e0_4549_98d9_74c013efb7cf.slice/crio-f6cca9df3d6ed0fcf2c530173851fee85b4bf677dd7ef731186e08184a4657d9 WatchSource:0}: Error finding container f6cca9df3d6ed0fcf2c530173851fee85b4bf677dd7ef731186e08184a4657d9: Status 404 returned error can't find the container with id f6cca9df3d6ed0fcf2c530173851fee85b4bf677dd7ef731186e08184a4657d9 Nov 03 02:13:23.408220 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:23.408160 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54abb9d6-5tzw2" event={"ID":"a60fd0a2-31e0-4549-98d9-74c013efb7cf","Type":"ContainerStarted","Data":"7eb277de983218730123bc8d9b26000ad4673693adb651326599fe03981597bd"} Nov 03 02:13:23.408985 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:23.408960 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54abb9d6-5tzw2" event={"ID":"a60fd0a2-31e0-4549-98d9-74c013efb7cf","Type":"ContainerStarted","Data":"f6cca9df3d6ed0fcf2c530173851fee85b4bf677dd7ef731186e08184a4657d9"} Nov 03 02:13:23.516519 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:23.516420 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54abb9d6-5tzw2" podStartSLOduration=1.516397099 podStartE2EDuration="1.516397099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:13:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:13:23.51556848 +0000 UTC m=+1280.774766635" watchObservedRunningTime="2026-11-03 02:13:23.516397099 +0000 UTC m=+1280.775595252" Nov 03 02:13:23.915519 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:13:23.915485 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:13:24.416549 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:24.416514 5605 generic.go:356] "Generic (PLEG): container finished" podID="a60fd0a2-31e0-4549-98d9-74c013efb7cf" containerID="7eb277de983218730123bc8d9b26000ad4673693adb651326599fe03981597bd" exitCode=0 Nov 03 02:13:24.416924 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:24.416563 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54abb9d6-5tzw2" event={"ID":"a60fd0a2-31e0-4549-98d9-74c013efb7cf","Type":"ContainerDied","Data":"7eb277de983218730123bc8d9b26000ad4673693adb651326599fe03981597bd"} Nov 03 02:13:25.572413 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:25.572380 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54abb9d6-5tzw2" Nov 03 02:13:25.621584 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:25.621556 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a60fd0a2-31e0-4549-98d9-74c013efb7cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a60fd0a2-31e0-4549-98d9-74c013efb7cf-maas-api-openshift-ca\") pod \"a60fd0a2-31e0-4549-98d9-74c013efb7cf\" (UID: \"a60fd0a2-31e0-4549-98d9-74c013efb7cf\") " Nov 03 02:13:25.621723 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:25.621628 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a60fd0a2-31e0-4549-98d9-74c013efb7cf-kube-api-access-qd2b7\" (UniqueName: \"kubernetes.io/projected/a60fd0a2-31e0-4549-98d9-74c013efb7cf-kube-api-access-qd2b7\") pod \"a60fd0a2-31e0-4549-98d9-74c013efb7cf\" (UID: \"a60fd0a2-31e0-4549-98d9-74c013efb7cf\") " Nov 03 02:13:25.621723 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:25.621694 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a60fd0a2-31e0-4549-98d9-74c013efb7cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a60fd0a2-31e0-4549-98d9-74c013efb7cf-maas-api-xks-ca\") pod \"a60fd0a2-31e0-4549-98d9-74c013efb7cf\" (UID: \"a60fd0a2-31e0-4549-98d9-74c013efb7cf\") " Nov 03 02:13:25.621910 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:25.621882 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a60fd0a2-31e0-4549-98d9-74c013efb7cf-maas-api-openshift-ca" pod "a60fd0a2-31e0-4549-98d9-74c013efb7cf" (UID: "a60fd0a2-31e0-4549-98d9-74c013efb7cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:13:25.622159 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:25.622138 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a60fd0a2-31e0-4549-98d9-74c013efb7cf-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:13:25.623579 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:25.623551 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a60fd0a2-31e0-4549-98d9-74c013efb7cf-maas-api-xks-ca" pod "a60fd0a2-31e0-4549-98d9-74c013efb7cf" (UID: "a60fd0a2-31e0-4549-98d9-74c013efb7cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:13:25.623694 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:25.623666 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a60fd0a2-31e0-4549-98d9-74c013efb7cf-kube-api-access-qd2b7" pod "a60fd0a2-31e0-4549-98d9-74c013efb7cf" (UID: "a60fd0a2-31e0-4549-98d9-74c013efb7cf"). InnerVolumeSpecName "kube-api-access-qd2b7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:13:25.723732 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:25.723664 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qd2b7\" (UniqueName: \"kubernetes.io/projected/a60fd0a2-31e0-4549-98d9-74c013efb7cf-kube-api-access-qd2b7\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:13:25.723732 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:25.723690 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a60fd0a2-31e0-4549-98d9-74c013efb7cf-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:13:26.207057 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:26.207022 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:13:26.429213 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:26.429177 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54abb9d6-5tzw2" event={"ID":"a60fd0a2-31e0-4549-98d9-74c013efb7cf","Type":"ContainerDied","Data":"f6cca9df3d6ed0fcf2c530173851fee85b4bf677dd7ef731186e08184a4657d9"} Nov 03 02:13:26.429393 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:26.429228 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54abb9d6-5tzw2" Nov 03 02:13:26.429393 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:26.429232 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6cca9df3d6ed0fcf2c530173851fee85b4bf677dd7ef731186e08184a4657d9" Nov 03 02:13:27.102099 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:27.102064 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-701ba30f/e2e-lifecycle-model-701ba30f-kserve-64469b6665-bmqzq"] Nov 03 02:13:27.102589 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:27.102572 5605 cpu_manager.go:414] "containerMap: removing container" podUID="a60fd0a2-31e0-4549-98d9-74c013efb7cf" containerName="revoke-keys" Nov 03 02:13:27.102642 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:27.102590 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a60fd0a2-31e0-4549-98d9-74c013efb7cf" containerName="revoke-keys" Nov 03 02:13:27.102703 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:27.102693 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="a60fd0a2-31e0-4549-98d9-74c013efb7cf" containerName="revoke-keys" Nov 03 02:13:27.106240 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:27.106219 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-701ba30f/e2e-lifecycle-model-701ba30f-kserve-64469b6665-bmqzq" Nov 03 02:13:27.139081 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:27.139055 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce-tls-certs\") pod \"e2e-lifecycle-model-701ba30f-kserve-64469b6665-bmqzq\" (UID: \"451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce\") " pod="ai-tenant-e2e-mt-701ba30f/e2e-lifecycle-model-701ba30f-kserve-64469b6665-bmqzq" Nov 03 02:13:27.139211 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:27.139124 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce-home\") pod \"e2e-lifecycle-model-701ba30f-kserve-64469b6665-bmqzq\" (UID: \"451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce\") " pod="ai-tenant-e2e-mt-701ba30f/e2e-lifecycle-model-701ba30f-kserve-64469b6665-bmqzq" Nov 03 02:13:27.139402 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:27.139336 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce-tmp-dir\") pod \"e2e-lifecycle-model-701ba30f-kserve-64469b6665-bmqzq\" (UID: \"451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce\") " pod="ai-tenant-e2e-mt-701ba30f/e2e-lifecycle-model-701ba30f-kserve-64469b6665-bmqzq" Nov 03 02:13:27.139529 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:27.139436 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgpdk\" (UniqueName: \"kubernetes.io/projected/451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce-kube-api-access-hgpdk\") pod \"e2e-lifecycle-model-701ba30f-kserve-64469b6665-bmqzq\" (UID: \"451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce\") " pod="ai-tenant-e2e-mt-701ba30f/e2e-lifecycle-model-701ba30f-kserve-64469b6665-bmqzq" Nov 03 02:13:27.139529 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:27.139482 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce-model-cache\") pod \"e2e-lifecycle-model-701ba30f-kserve-64469b6665-bmqzq\" (UID: \"451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce\") " pod="ai-tenant-e2e-mt-701ba30f/e2e-lifecycle-model-701ba30f-kserve-64469b6665-bmqzq" Nov 03 02:13:27.139685 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:27.139664 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce-dshm\") pod \"e2e-lifecycle-model-701ba30f-kserve-64469b6665-bmqzq\" (UID: \"451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce\") " pod="ai-tenant-e2e-mt-701ba30f/e2e-lifecycle-model-701ba30f-kserve-64469b6665-bmqzq" Nov 03 02:13:27.202859 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:27.202818 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-701ba30f\"/\"openshift-service-ca.crt\"" Nov 03 02:13:27.202859 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:27.202839 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-701ba30f\"/\"kube-root-ca.crt\"" Nov 03 02:13:27.203068 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:27.202889 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-701ba30f\"/\"e2e-lifecycle-model-701ba30f-kserve-self-signed-certs\"" Nov 03 02:13:27.203068 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:27.202973 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-701ba30f\"/\"default-dockercfg-9r26v\"" Nov 03 02:13:27.205825 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:27.205802 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-701ba30f/e2e-lifecycle-model-701ba30f-kserve-64469b6665-bmqzq"] Nov 03 02:13:27.240664 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:27.240638 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce-tmp-dir\") pod \"e2e-lifecycle-model-701ba30f-kserve-64469b6665-bmqzq\" (UID: \"451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce\") " pod="ai-tenant-e2e-mt-701ba30f/e2e-lifecycle-model-701ba30f-kserve-64469b6665-bmqzq" Nov 03 02:13:27.240788 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:27.240682 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgpdk\" (UniqueName: \"kubernetes.io/projected/451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce-kube-api-access-hgpdk\") pod \"e2e-lifecycle-model-701ba30f-kserve-64469b6665-bmqzq\" (UID: \"451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce\") " pod="ai-tenant-e2e-mt-701ba30f/e2e-lifecycle-model-701ba30f-kserve-64469b6665-bmqzq" Nov 03 02:13:27.240788 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:27.240731 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce-model-cache\") pod \"e2e-lifecycle-model-701ba30f-kserve-64469b6665-bmqzq\" (UID: \"451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce\") " pod="ai-tenant-e2e-mt-701ba30f/e2e-lifecycle-model-701ba30f-kserve-64469b6665-bmqzq" Nov 03 02:13:27.240885 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:27.240810 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce-dshm\") pod \"e2e-lifecycle-model-701ba30f-kserve-64469b6665-bmqzq\" (UID: \"451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce\") " pod="ai-tenant-e2e-mt-701ba30f/e2e-lifecycle-model-701ba30f-kserve-64469b6665-bmqzq" Nov 03 02:13:27.240885 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:27.240844 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce-tls-certs\") pod \"e2e-lifecycle-model-701ba30f-kserve-64469b6665-bmqzq\" (UID: \"451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce\") " pod="ai-tenant-e2e-mt-701ba30f/e2e-lifecycle-model-701ba30f-kserve-64469b6665-bmqzq" Nov 03 02:13:27.241024 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:27.240926 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce-home\") pod \"e2e-lifecycle-model-701ba30f-kserve-64469b6665-bmqzq\" (UID: \"451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce\") " pod="ai-tenant-e2e-mt-701ba30f/e2e-lifecycle-model-701ba30f-kserve-64469b6665-bmqzq" Nov 03 02:13:27.241140 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:27.241092 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce-tmp-dir\") pod \"e2e-lifecycle-model-701ba30f-kserve-64469b6665-bmqzq\" (UID: \"451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce\") " pod="ai-tenant-e2e-mt-701ba30f/e2e-lifecycle-model-701ba30f-kserve-64469b6665-bmqzq" Nov 03 02:13:27.241257 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:27.241142 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce-model-cache\") pod \"e2e-lifecycle-model-701ba30f-kserve-64469b6665-bmqzq\" (UID: \"451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce\") " pod="ai-tenant-e2e-mt-701ba30f/e2e-lifecycle-model-701ba30f-kserve-64469b6665-bmqzq" Nov 03 02:13:27.241316 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:27.241294 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce-home\") pod \"e2e-lifecycle-model-701ba30f-kserve-64469b6665-bmqzq\" (UID: \"451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce\") " pod="ai-tenant-e2e-mt-701ba30f/e2e-lifecycle-model-701ba30f-kserve-64469b6665-bmqzq" Nov 03 02:13:27.243068 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:27.243049 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce-dshm\") pod \"e2e-lifecycle-model-701ba30f-kserve-64469b6665-bmqzq\" (UID: \"451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce\") " pod="ai-tenant-e2e-mt-701ba30f/e2e-lifecycle-model-701ba30f-kserve-64469b6665-bmqzq" Nov 03 02:13:27.243443 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:27.243426 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce-tls-certs\") pod \"e2e-lifecycle-model-701ba30f-kserve-64469b6665-bmqzq\" (UID: \"451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce\") " pod="ai-tenant-e2e-mt-701ba30f/e2e-lifecycle-model-701ba30f-kserve-64469b6665-bmqzq" Nov 03 02:13:27.313379 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:27.311664 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgpdk\" (UniqueName: \"kubernetes.io/projected/451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce-kube-api-access-hgpdk\") pod \"e2e-lifecycle-model-701ba30f-kserve-64469b6665-bmqzq\" (UID: \"451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce\") " pod="ai-tenant-e2e-mt-701ba30f/e2e-lifecycle-model-701ba30f-kserve-64469b6665-bmqzq" Nov 03 02:13:27.427436 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:27.426608 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-701ba30f/e2e-lifecycle-model-701ba30f-kserve-64469b6665-bmqzq" Nov 03 02:13:28.116740 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:28.108448 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-701ba30f/e2e-lifecycle-model-701ba30f-kserve-64469b6665-bmqzq"] Nov 03 02:13:28.460646 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:28.460551 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-701ba30f/e2e-lifecycle-model-701ba30f-kserve-64469b6665-bmqzq" event={"ID":"451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce","Type":"ContainerStarted","Data":"df9ddefd0f1c905c028fd7143bb5cca734774745782ee46a0d3a6017c3d2cd21"} Nov 03 02:13:28.460646 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:28.460610 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-701ba30f/e2e-lifecycle-model-701ba30f-kserve-64469b6665-bmqzq" event={"ID":"451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce","Type":"ContainerStarted","Data":"5bfc1fd4e55296e8caf37ce9e38bbee008d69353a0024183d28bd9686b9d043f"} Nov 03 02:13:28.567139 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:28.567079 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-701ba30f/e2e-lifecycle-model-701ba30f-kserve-64469b6665-bmqzq" podStartSLOduration=1.567063984 podStartE2EDuration="1.567063984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:13:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:13:28.56676035 +0000 UTC m=+1285.825958504" watchObservedRunningTime="2026-11-03 02:13:28.567063984 +0000 UTC m=+1285.826262138" Nov 03 02:13:29.428500 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:29.428462 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-701ba30f/e2e-lifecycle-model-701ba30f-kserve-64469b6665-bmqzq" Nov 03 02:13:35.546206 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:35.546163 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76"] Nov 03 02:13:35.550546 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:35.550523 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76" Nov 03 02:13:35.577810 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:35.577777 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bd56b15-0fc7-4a81-a569-4eba9ffaf645-istio-podinfo\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76\" (UID: \"4bd56b15-0fc7-4a81-a569-4eba9ffaf645\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76" Nov 03 02:13:35.577949 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:35.577834 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bd56b15-0fc7-4a81-a569-4eba9ffaf645-istio-token\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76\" (UID: \"4bd56b15-0fc7-4a81-a569-4eba9ffaf645\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76" Nov 03 02:13:35.577949 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:35.577924 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqqjm\" (UniqueName: \"kubernetes.io/projected/4bd56b15-0fc7-4a81-a569-4eba9ffaf645-kube-api-access-dqqjm\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76\" (UID: \"4bd56b15-0fc7-4a81-a569-4eba9ffaf645\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76" Nov 03 02:13:35.578069 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:35.578029 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bd56b15-0fc7-4a81-a569-4eba9ffaf645-workload-certs\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76\" (UID: \"4bd56b15-0fc7-4a81-a569-4eba9ffaf645\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76" Nov 03 02:13:35.578137 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:35.578118 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bd56b15-0fc7-4a81-a569-4eba9ffaf645-istio-data\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76\" (UID: \"4bd56b15-0fc7-4a81-a569-4eba9ffaf645\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76" Nov 03 02:13:35.578198 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:35.578179 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bd56b15-0fc7-4a81-a569-4eba9ffaf645-workload-socket\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76\" (UID: \"4bd56b15-0fc7-4a81-a569-4eba9ffaf645\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76" Nov 03 02:13:35.578315 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:35.578293 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bd56b15-0fc7-4a81-a569-4eba9ffaf645-credential-socket\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76\" (UID: \"4bd56b15-0fc7-4a81-a569-4eba9ffaf645\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76" Nov 03 02:13:35.578391 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:35.578328 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bd56b15-0fc7-4a81-a569-4eba9ffaf645-istiod-ca-cert\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76\" (UID: \"4bd56b15-0fc7-4a81-a569-4eba9ffaf645\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76" Nov 03 02:13:35.578450 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:35.578391 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bd56b15-0fc7-4a81-a569-4eba9ffaf645-istio-envoy\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76\" (UID: \"4bd56b15-0fc7-4a81-a569-4eba9ffaf645\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76" Nov 03 02:13:35.622393 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:35.622330 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76"] Nov 03 02:13:35.648306 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:35.647614 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-a0e72e-openshift-default-dockercfg-cskrh\"" Nov 03 02:13:35.701396 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:35.687095 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bd56b15-0fc7-4a81-a569-4eba9ffaf645-istio-token\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76\" (UID: \"4bd56b15-0fc7-4a81-a569-4eba9ffaf645\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76" Nov 03 02:13:35.701733 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:35.701702 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqqjm\" (UniqueName: \"kubernetes.io/projected/4bd56b15-0fc7-4a81-a569-4eba9ffaf645-kube-api-access-dqqjm\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76\" (UID: \"4bd56b15-0fc7-4a81-a569-4eba9ffaf645\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76" Nov 03 02:13:35.711550 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:35.711502 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bd56b15-0fc7-4a81-a569-4eba9ffaf645-workload-certs\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76\" (UID: \"4bd56b15-0fc7-4a81-a569-4eba9ffaf645\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76" Nov 03 02:13:35.720416 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:35.711827 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bd56b15-0fc7-4a81-a569-4eba9ffaf645-istio-data\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76\" (UID: \"4bd56b15-0fc7-4a81-a569-4eba9ffaf645\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76" Nov 03 02:13:35.720416 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:35.720167 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bd56b15-0fc7-4a81-a569-4eba9ffaf645-workload-socket\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76\" (UID: \"4bd56b15-0fc7-4a81-a569-4eba9ffaf645\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76" Nov 03 02:13:35.720416 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:35.720272 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bd56b15-0fc7-4a81-a569-4eba9ffaf645-credential-socket\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76\" (UID: \"4bd56b15-0fc7-4a81-a569-4eba9ffaf645\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76" Nov 03 02:13:35.720416 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:35.720299 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bd56b15-0fc7-4a81-a569-4eba9ffaf645-istiod-ca-cert\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76\" (UID: \"4bd56b15-0fc7-4a81-a569-4eba9ffaf645\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76" Nov 03 02:13:35.720926 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:35.720866 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bd56b15-0fc7-4a81-a569-4eba9ffaf645-istio-envoy\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76\" (UID: \"4bd56b15-0fc7-4a81-a569-4eba9ffaf645\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76" Nov 03 02:13:35.721144 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:35.721122 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bd56b15-0fc7-4a81-a569-4eba9ffaf645-istio-podinfo\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76\" (UID: \"4bd56b15-0fc7-4a81-a569-4eba9ffaf645\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76" Nov 03 02:13:35.723714 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:35.723682 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bd56b15-0fc7-4a81-a569-4eba9ffaf645-workload-socket\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76\" (UID: \"4bd56b15-0fc7-4a81-a569-4eba9ffaf645\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76" Nov 03 02:13:35.723820 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:35.723713 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bd56b15-0fc7-4a81-a569-4eba9ffaf645-credential-socket\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76\" (UID: \"4bd56b15-0fc7-4a81-a569-4eba9ffaf645\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76" Nov 03 02:13:35.724020 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:35.723996 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bd56b15-0fc7-4a81-a569-4eba9ffaf645-workload-certs\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76\" (UID: \"4bd56b15-0fc7-4a81-a569-4eba9ffaf645\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76" Nov 03 02:13:35.724193 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:35.724169 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bd56b15-0fc7-4a81-a569-4eba9ffaf645-istiod-ca-cert\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76\" (UID: \"4bd56b15-0fc7-4a81-a569-4eba9ffaf645\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76" Nov 03 02:13:35.724278 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:35.724255 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bd56b15-0fc7-4a81-a569-4eba9ffaf645-istio-data\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76\" (UID: \"4bd56b15-0fc7-4a81-a569-4eba9ffaf645\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76" Nov 03 02:13:35.725190 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:35.725169 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bd56b15-0fc7-4a81-a569-4eba9ffaf645-istio-podinfo\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76\" (UID: \"4bd56b15-0fc7-4a81-a569-4eba9ffaf645\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76" Nov 03 02:13:35.727325 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:35.727307 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bd56b15-0fc7-4a81-a569-4eba9ffaf645-istio-envoy\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76\" (UID: \"4bd56b15-0fc7-4a81-a569-4eba9ffaf645\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76" Nov 03 02:13:35.804642 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:35.785851 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqqjm\" (UniqueName: \"kubernetes.io/projected/4bd56b15-0fc7-4a81-a569-4eba9ffaf645-kube-api-access-dqqjm\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76\" (UID: \"4bd56b15-0fc7-4a81-a569-4eba9ffaf645\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76" Nov 03 02:13:35.804642 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:35.799328 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bd56b15-0fc7-4a81-a569-4eba9ffaf645-istio-token\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76\" (UID: \"4bd56b15-0fc7-4a81-a569-4eba9ffaf645\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76" Nov 03 02:13:35.872386 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:35.872053 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76" Nov 03 02:13:36.336693 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:36.336659 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76"] Nov 03 02:13:36.343598 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:13:36.343568 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bd56b15_0fc7_4a81_a569_4eba9ffaf645.slice/crio-c65707b594fb5f4346bd4491ca654467845ce8c7cd6aa7caa9f9fb38d6a0213a WatchSource:0}: Error finding container c65707b594fb5f4346bd4491ca654467845ce8c7cd6aa7caa9f9fb38d6a0213a: Status 404 returned error can't find the container with id c65707b594fb5f4346bd4491ca654467845ce8c7cd6aa7caa9f9fb38d6a0213a Nov 03 02:13:36.346113 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:36.346080 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:13:36.346207 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:36.346150 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:13:36.346207 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:36.346180 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:13:36.567133 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:36.567100 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76" event={"ID":"4bd56b15-0fc7-4a81-a569-4eba9ffaf645","Type":"ContainerStarted","Data":"b3fcbcb6fb615e86c873dacad12dec42505fcfae737f0f08a8b91503ece900b5"} Nov 03 02:13:36.567524 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:36.567164 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76" event={"ID":"4bd56b15-0fc7-4a81-a569-4eba9ffaf645","Type":"ContainerStarted","Data":"c65707b594fb5f4346bd4491ca654467845ce8c7cd6aa7caa9f9fb38d6a0213a"} Nov 03 02:13:36.676974 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:36.676873 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76" podStartSLOduration=1.6768584930000001 podStartE2EDuration="1.676858493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:13:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:13:36.675078106 +0000 UTC m=+1293.934276261" watchObservedRunningTime="2026-11-03 02:13:36.676858493 +0000 UTC m=+1293.936056646" Nov 03 02:13:36.876381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:36.872459 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76" Nov 03 02:13:37.428977 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:37.428940 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-701ba30f/e2e-lifecycle-model-701ba30f-kserve-64469b6665-bmqzq" Nov 03 02:13:37.442059 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:37.442032 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-701ba30f/e2e-lifecycle-model-701ba30f-kserve-64469b6665-bmqzq" Nov 03 02:13:37.594149 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:37.594113 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-701ba30f/e2e-lifecycle-model-701ba30f-kserve-64469b6665-bmqzq" Nov 03 02:13:37.880467 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:37.880433 5605 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 02:13:37.880668 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:37.880504 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76" podUID="4bd56b15-0fc7-4a81-a569-4eba9ffaf645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 02:13:38.879517 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:38.879480 5605 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 02:13:38.879943 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:38.879577 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76" podUID="4bd56b15-0fc7-4a81-a569-4eba9ffaf645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 02:13:39.878380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:39.877656 5605 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 02:13:39.878380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:39.877717 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76" podUID="4bd56b15-0fc7-4a81-a569-4eba9ffaf645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 02:13:40.873391 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:40.872670 5605 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 02:13:40.873391 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:40.872741 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76" podUID="4bd56b15-0fc7-4a81-a569-4eba9ffaf645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 02:13:41.887817 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:41.887774 5605 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 02:13:41.888508 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:41.888473 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76" podUID="4bd56b15-0fc7-4a81-a569-4eba9ffaf645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 02:13:41.989651 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:41.989595 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a772c3-6cc589fd8c-pf6hh"] Nov 03 02:13:41.996102 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:41.996070 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a772c3-6cc589fd8c-pf6hh" Nov 03 02:13:42.043252 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:42.043224 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a772c3-6cc589fd8c-pf6hh"] Nov 03 02:13:42.045081 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:42.045053 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-a772c3\"" Nov 03 02:13:42.045228 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:42.045104 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-a772c3-dockercfg-vx7pt\"" Nov 03 02:13:42.052120 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:42.052096 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqnrs\" (UniqueName: \"kubernetes.io/projected/3e782231-a2f8-45d9-8507-fcc038f1a368-kube-api-access-qqnrs\") pod \"payload-pre-processing-e2e-api-a-a772c3-6cc589fd8c-pf6hh\" (UID: \"3e782231-a2f8-45d9-8507-fcc038f1a368\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a772c3-6cc589fd8c-pf6hh" Nov 03 02:13:42.052228 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:42.052180 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e782231-a2f8-45d9-8507-fcc038f1a368-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a772c3-6cc589fd8c-pf6hh\" (UID: \"3e782231-a2f8-45d9-8507-fcc038f1a368\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a772c3-6cc589fd8c-pf6hh" Nov 03 02:13:42.229155 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:42.229045 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqnrs\" (UniqueName: \"kubernetes.io/projected/3e782231-a2f8-45d9-8507-fcc038f1a368-kube-api-access-qqnrs\") pod \"payload-pre-processing-e2e-api-a-a772c3-6cc589fd8c-pf6hh\" (UID: \"3e782231-a2f8-45d9-8507-fcc038f1a368\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a772c3-6cc589fd8c-pf6hh" Nov 03 02:13:42.229559 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:42.229530 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e782231-a2f8-45d9-8507-fcc038f1a368-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a772c3-6cc589fd8c-pf6hh\" (UID: \"3e782231-a2f8-45d9-8507-fcc038f1a368\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a772c3-6cc589fd8c-pf6hh" Nov 03 02:13:42.246411 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:42.246332 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e782231-a2f8-45d9-8507-fcc038f1a368-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a772c3-6cc589fd8c-pf6hh\" (UID: \"3e782231-a2f8-45d9-8507-fcc038f1a368\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a772c3-6cc589fd8c-pf6hh" Nov 03 02:13:42.385632 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:42.385445 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqnrs\" (UniqueName: \"kubernetes.io/projected/3e782231-a2f8-45d9-8507-fcc038f1a368-kube-api-access-qqnrs\") pod \"payload-pre-processing-e2e-api-a-a772c3-6cc589fd8c-pf6hh\" (UID: \"3e782231-a2f8-45d9-8507-fcc038f1a368\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a772c3-6cc589fd8c-pf6hh" Nov 03 02:13:42.609916 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:42.609885 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a772c3-6cc589fd8c-pf6hh" Nov 03 02:13:42.872609 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:42.872539 5605 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 02:13:42.872609 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:42.872588 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76" podUID="4bd56b15-0fc7-4a81-a569-4eba9ffaf645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 02:13:43.023445 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:43.022561 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a772c3-6cc589fd8c-pf6hh"] Nov 03 02:13:43.622935 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:43.622893 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a772c3-6cc589fd8c-pf6hh" event={"ID":"3e782231-a2f8-45d9-8507-fcc038f1a368","Type":"ContainerStarted","Data":"87835394c46ddbe897479e341429a5b51ea93bdb5b2cf2311c2b0e052ed5dad9"} Nov 03 02:13:43.623133 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:43.622963 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a772c3-6cc589fd8c-pf6hh" event={"ID":"3e782231-a2f8-45d9-8507-fcc038f1a368","Type":"ContainerStarted","Data":"68e9f608b5472623cf64ce8fe88dedd3018a6d4aeb2f0ee99c0627ba54d8480a"} Nov 03 02:13:43.623133 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:43.623121 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a772c3-6cc589fd8c-pf6hh" Nov 03 02:13:43.735073 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:43.735014 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a772c3-6cc589fd8c-pf6hh" podStartSLOduration=2.734999991 podStartE2EDuration="2.734999991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:13:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:13:43.734090428 +0000 UTC m=+1300.993288583" watchObservedRunningTime="2026-11-03 02:13:43.734999991 +0000 UTC m=+1300.994198157" Nov 03 02:13:43.876951 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:43.876881 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76" Nov 03 02:13:43.877401 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:43.877380 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76" Nov 03 02:13:43.878006 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:43.877967 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76" Nov 03 02:13:53.913327 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:13:53.913291 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:13:54.698312 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:54.698282 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a772c3-6cc589fd8c-pf6hh" Nov 03 02:13:59.270447 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:59.270413 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-a0e72e/distinct-2-w3-a0e72e-kserve-59f58c59bf-jz5p8"] Nov 03 02:13:59.274585 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:59.274559 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a0e72e/distinct-2-w3-a0e72e-kserve-59f58c59bf-jz5p8" Nov 03 02:13:59.371574 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:59.371542 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-a0e72e\"/\"distinct-2-w3-a0e72e-kserve-self-signed-certs\"" Nov 03 02:13:59.375391 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:59.375346 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-a0e72e/distinct-2-w3-a0e72e-kserve-59f58c59bf-jz5p8"] Nov 03 02:13:59.375863 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:59.375843 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70bd98e0-d8ac-47c2-901e-83f301d59a57-dshm\") pod \"distinct-2-w3-a0e72e-kserve-59f58c59bf-jz5p8\" (UID: \"70bd98e0-d8ac-47c2-901e-83f301d59a57\") " pod="e2e-models-e2e-worker-w3-a0e72e/distinct-2-w3-a0e72e-kserve-59f58c59bf-jz5p8" Nov 03 02:13:59.375915 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:59.375876 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9dcn\" (UniqueName: \"kubernetes.io/projected/70bd98e0-d8ac-47c2-901e-83f301d59a57-kube-api-access-h9dcn\") pod \"distinct-2-w3-a0e72e-kserve-59f58c59bf-jz5p8\" (UID: \"70bd98e0-d8ac-47c2-901e-83f301d59a57\") " pod="e2e-models-e2e-worker-w3-a0e72e/distinct-2-w3-a0e72e-kserve-59f58c59bf-jz5p8" Nov 03 02:13:59.375915 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:59.375905 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70bd98e0-d8ac-47c2-901e-83f301d59a57-home\") pod \"distinct-2-w3-a0e72e-kserve-59f58c59bf-jz5p8\" (UID: \"70bd98e0-d8ac-47c2-901e-83f301d59a57\") " pod="e2e-models-e2e-worker-w3-a0e72e/distinct-2-w3-a0e72e-kserve-59f58c59bf-jz5p8" Nov 03 02:13:59.376123 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:59.376103 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70bd98e0-d8ac-47c2-901e-83f301d59a57-tmp-dir\") pod \"distinct-2-w3-a0e72e-kserve-59f58c59bf-jz5p8\" (UID: \"70bd98e0-d8ac-47c2-901e-83f301d59a57\") " pod="e2e-models-e2e-worker-w3-a0e72e/distinct-2-w3-a0e72e-kserve-59f58c59bf-jz5p8" Nov 03 02:13:59.376186 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:59.376137 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70bd98e0-d8ac-47c2-901e-83f301d59a57-model-cache\") pod \"distinct-2-w3-a0e72e-kserve-59f58c59bf-jz5p8\" (UID: \"70bd98e0-d8ac-47c2-901e-83f301d59a57\") " pod="e2e-models-e2e-worker-w3-a0e72e/distinct-2-w3-a0e72e-kserve-59f58c59bf-jz5p8" Nov 03 02:13:59.376249 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:59.376187 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70bd98e0-d8ac-47c2-901e-83f301d59a57-tls-certs\") pod \"distinct-2-w3-a0e72e-kserve-59f58c59bf-jz5p8\" (UID: \"70bd98e0-d8ac-47c2-901e-83f301d59a57\") " pod="e2e-models-e2e-worker-w3-a0e72e/distinct-2-w3-a0e72e-kserve-59f58c59bf-jz5p8" Nov 03 02:13:59.482633 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:59.482596 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70bd98e0-d8ac-47c2-901e-83f301d59a57-tmp-dir\") pod \"distinct-2-w3-a0e72e-kserve-59f58c59bf-jz5p8\" (UID: \"70bd98e0-d8ac-47c2-901e-83f301d59a57\") " pod="e2e-models-e2e-worker-w3-a0e72e/distinct-2-w3-a0e72e-kserve-59f58c59bf-jz5p8" Nov 03 02:13:59.482883 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:59.482801 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70bd98e0-d8ac-47c2-901e-83f301d59a57-tmp-dir\") pod \"distinct-2-w3-a0e72e-kserve-59f58c59bf-jz5p8\" (UID: \"70bd98e0-d8ac-47c2-901e-83f301d59a57\") " pod="e2e-models-e2e-worker-w3-a0e72e/distinct-2-w3-a0e72e-kserve-59f58c59bf-jz5p8" Nov 03 02:13:59.483074 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:59.483052 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70bd98e0-d8ac-47c2-901e-83f301d59a57-model-cache\") pod \"distinct-2-w3-a0e72e-kserve-59f58c59bf-jz5p8\" (UID: \"70bd98e0-d8ac-47c2-901e-83f301d59a57\") " pod="e2e-models-e2e-worker-w3-a0e72e/distinct-2-w3-a0e72e-kserve-59f58c59bf-jz5p8" Nov 03 02:13:59.483231 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:59.483212 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70bd98e0-d8ac-47c2-901e-83f301d59a57-tls-certs\") pod \"distinct-2-w3-a0e72e-kserve-59f58c59bf-jz5p8\" (UID: \"70bd98e0-d8ac-47c2-901e-83f301d59a57\") " pod="e2e-models-e2e-worker-w3-a0e72e/distinct-2-w3-a0e72e-kserve-59f58c59bf-jz5p8" Nov 03 02:13:59.483560 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:59.483536 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70bd98e0-d8ac-47c2-901e-83f301d59a57-dshm\") pod \"distinct-2-w3-a0e72e-kserve-59f58c59bf-jz5p8\" (UID: \"70bd98e0-d8ac-47c2-901e-83f301d59a57\") " pod="e2e-models-e2e-worker-w3-a0e72e/distinct-2-w3-a0e72e-kserve-59f58c59bf-jz5p8" Nov 03 02:13:59.483672 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:59.483643 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9dcn\" (UniqueName: \"kubernetes.io/projected/70bd98e0-d8ac-47c2-901e-83f301d59a57-kube-api-access-h9dcn\") pod \"distinct-2-w3-a0e72e-kserve-59f58c59bf-jz5p8\" (UID: \"70bd98e0-d8ac-47c2-901e-83f301d59a57\") " pod="e2e-models-e2e-worker-w3-a0e72e/distinct-2-w3-a0e72e-kserve-59f58c59bf-jz5p8" Nov 03 02:13:59.483742 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:59.483718 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70bd98e0-d8ac-47c2-901e-83f301d59a57-home\") pod \"distinct-2-w3-a0e72e-kserve-59f58c59bf-jz5p8\" (UID: \"70bd98e0-d8ac-47c2-901e-83f301d59a57\") " pod="e2e-models-e2e-worker-w3-a0e72e/distinct-2-w3-a0e72e-kserve-59f58c59bf-jz5p8" Nov 03 02:13:59.483796 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:59.483741 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70bd98e0-d8ac-47c2-901e-83f301d59a57-model-cache\") pod \"distinct-2-w3-a0e72e-kserve-59f58c59bf-jz5p8\" (UID: \"70bd98e0-d8ac-47c2-901e-83f301d59a57\") " pod="e2e-models-e2e-worker-w3-a0e72e/distinct-2-w3-a0e72e-kserve-59f58c59bf-jz5p8" Nov 03 02:13:59.484880 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:59.484839 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70bd98e0-d8ac-47c2-901e-83f301d59a57-home\") pod \"distinct-2-w3-a0e72e-kserve-59f58c59bf-jz5p8\" (UID: \"70bd98e0-d8ac-47c2-901e-83f301d59a57\") " pod="e2e-models-e2e-worker-w3-a0e72e/distinct-2-w3-a0e72e-kserve-59f58c59bf-jz5p8" Nov 03 02:13:59.488743 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:59.488716 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70bd98e0-d8ac-47c2-901e-83f301d59a57-dshm\") pod \"distinct-2-w3-a0e72e-kserve-59f58c59bf-jz5p8\" (UID: \"70bd98e0-d8ac-47c2-901e-83f301d59a57\") " pod="e2e-models-e2e-worker-w3-a0e72e/distinct-2-w3-a0e72e-kserve-59f58c59bf-jz5p8" Nov 03 02:13:59.513171 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:59.513098 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70bd98e0-d8ac-47c2-901e-83f301d59a57-tls-certs\") pod \"distinct-2-w3-a0e72e-kserve-59f58c59bf-jz5p8\" (UID: \"70bd98e0-d8ac-47c2-901e-83f301d59a57\") " pod="e2e-models-e2e-worker-w3-a0e72e/distinct-2-w3-a0e72e-kserve-59f58c59bf-jz5p8" Nov 03 02:13:59.538589 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:59.538499 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9dcn\" (UniqueName: \"kubernetes.io/projected/70bd98e0-d8ac-47c2-901e-83f301d59a57-kube-api-access-h9dcn\") pod \"distinct-2-w3-a0e72e-kserve-59f58c59bf-jz5p8\" (UID: \"70bd98e0-d8ac-47c2-901e-83f301d59a57\") " pod="e2e-models-e2e-worker-w3-a0e72e/distinct-2-w3-a0e72e-kserve-59f58c59bf-jz5p8" Nov 03 02:13:59.587735 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:59.587694 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a0e72e/distinct-2-w3-a0e72e-kserve-59f58c59bf-jz5p8" Nov 03 02:13:59.999271 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:13:59.999233 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-a0e72e/distinct-2-w3-a0e72e-kserve-59f58c59bf-jz5p8"] Nov 03 02:14:00.004681 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:14:00.004645 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod70bd98e0_d8ac_47c2_901e_83f301d59a57.slice/crio-1fe42bab462c3b011ebc822c0a5a50a303d8d41e8e5a40d2640df82c94211b90 WatchSource:0}: Error finding container 1fe42bab462c3b011ebc822c0a5a50a303d8d41e8e5a40d2640df82c94211b90: Status 404 returned error can't find the container with id 1fe42bab462c3b011ebc822c0a5a50a303d8d41e8e5a40d2640df82c94211b90 Nov 03 02:14:00.880011 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:00.879969 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a0e72e/distinct-2-w3-a0e72e-kserve-59f58c59bf-jz5p8" event={"ID":"70bd98e0-d8ac-47c2-901e-83f301d59a57","Type":"ContainerStarted","Data":"81d3f5c16c98c27bdb8dfd4a2e963704733d3775640740eec7d25422c78230c9"} Nov 03 02:14:00.880569 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:00.880020 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a0e72e/distinct-2-w3-a0e72e-kserve-59f58c59bf-jz5p8" event={"ID":"70bd98e0-d8ac-47c2-901e-83f301d59a57","Type":"ContainerStarted","Data":"1fe42bab462c3b011ebc822c0a5a50a303d8d41e8e5a40d2640df82c94211b90"} Nov 03 02:14:00.989430 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:00.989351 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-a0e72e/distinct-2-w3-a0e72e-kserve-59f58c59bf-jz5p8" podStartSLOduration=1.9893321309999998 podStartE2EDuration="1.989332131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:13:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:14:00.986047319 +0000 UTC m=+1318.245245474" watchObservedRunningTime="2026-11-03 02:14:00.989332131 +0000 UTC m=+1318.248530286" Nov 03 02:14:01.589033 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:01.588992 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a0e72e/distinct-2-w3-a0e72e-kserve-59f58c59bf-jz5p8" Nov 03 02:14:09.392290 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:09.392248 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:14:09.588655 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:09.588616 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-a0e72e/distinct-2-w3-a0e72e-kserve-59f58c59bf-jz5p8" Nov 03 02:14:09.601846 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:09.601815 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-a0e72e/distinct-2-w3-a0e72e-kserve-59f58c59bf-jz5p8" Nov 03 02:14:09.946814 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:09.946785 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a0e72e/distinct-2-w3-a0e72e-kserve-59f58c59bf-jz5p8" Nov 03 02:14:22.452670 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:22.452627 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8"] Nov 03 02:14:22.456865 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:22.456838 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8" Nov 03 02:14:22.505797 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:22.505763 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-c617de-openshift-default-dockercfg-nd496\"" Nov 03 02:14:22.557180 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:22.557148 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8"] Nov 03 02:14:22.605626 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:22.605579 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87ac6502-78a2-41ed-9033-77eecf21086f-istio-data\") pod \"e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8\" (UID: \"87ac6502-78a2-41ed-9033-77eecf21086f\") " pod="openshift-ingress/e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8" Nov 03 02:14:22.674382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:22.665644 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4559\" (UniqueName: \"kubernetes.io/projected/87ac6502-78a2-41ed-9033-77eecf21086f-kube-api-access-l4559\") pod \"e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8\" (UID: \"87ac6502-78a2-41ed-9033-77eecf21086f\") " pod="openshift-ingress/e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8" Nov 03 02:14:22.674382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:22.665709 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87ac6502-78a2-41ed-9033-77eecf21086f-istio-token\") pod \"e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8\" (UID: \"87ac6502-78a2-41ed-9033-77eecf21086f\") " pod="openshift-ingress/e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8" Nov 03 02:14:22.674382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:22.665752 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87ac6502-78a2-41ed-9033-77eecf21086f-istiod-ca-cert\") pod \"e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8\" (UID: \"87ac6502-78a2-41ed-9033-77eecf21086f\") " pod="openshift-ingress/e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8" Nov 03 02:14:22.674382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:22.665887 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87ac6502-78a2-41ed-9033-77eecf21086f-istio-envoy\") pod \"e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8\" (UID: \"87ac6502-78a2-41ed-9033-77eecf21086f\") " pod="openshift-ingress/e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8" Nov 03 02:14:22.674382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:22.665925 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87ac6502-78a2-41ed-9033-77eecf21086f-istio-podinfo\") pod \"e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8\" (UID: \"87ac6502-78a2-41ed-9033-77eecf21086f\") " pod="openshift-ingress/e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8" Nov 03 02:14:22.674382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:22.665954 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87ac6502-78a2-41ed-9033-77eecf21086f-workload-certs\") pod \"e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8\" (UID: \"87ac6502-78a2-41ed-9033-77eecf21086f\") " pod="openshift-ingress/e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8" Nov 03 02:14:22.674382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:22.665983 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87ac6502-78a2-41ed-9033-77eecf21086f-credential-socket\") pod \"e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8\" (UID: \"87ac6502-78a2-41ed-9033-77eecf21086f\") " pod="openshift-ingress/e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8" Nov 03 02:14:22.674382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:22.666103 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87ac6502-78a2-41ed-9033-77eecf21086f-workload-socket\") pod \"e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8\" (UID: \"87ac6502-78a2-41ed-9033-77eecf21086f\") " pod="openshift-ingress/e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8" Nov 03 02:14:22.781730 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:22.778283 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87ac6502-78a2-41ed-9033-77eecf21086f-istio-data\") pod \"e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8\" (UID: \"87ac6502-78a2-41ed-9033-77eecf21086f\") " pod="openshift-ingress/e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8" Nov 03 02:14:22.781730 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:22.778376 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4559\" (UniqueName: \"kubernetes.io/projected/87ac6502-78a2-41ed-9033-77eecf21086f-kube-api-access-l4559\") pod \"e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8\" (UID: \"87ac6502-78a2-41ed-9033-77eecf21086f\") " pod="openshift-ingress/e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8" Nov 03 02:14:22.781730 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:22.778420 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87ac6502-78a2-41ed-9033-77eecf21086f-istio-token\") pod \"e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8\" (UID: \"87ac6502-78a2-41ed-9033-77eecf21086f\") " pod="openshift-ingress/e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8" Nov 03 02:14:22.781730 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:22.778455 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87ac6502-78a2-41ed-9033-77eecf21086f-istiod-ca-cert\") pod \"e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8\" (UID: \"87ac6502-78a2-41ed-9033-77eecf21086f\") " pod="openshift-ingress/e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8" Nov 03 02:14:22.781730 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:22.778560 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87ac6502-78a2-41ed-9033-77eecf21086f-istio-envoy\") pod \"e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8\" (UID: \"87ac6502-78a2-41ed-9033-77eecf21086f\") " pod="openshift-ingress/e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8" Nov 03 02:14:22.781730 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:22.778596 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87ac6502-78a2-41ed-9033-77eecf21086f-istio-podinfo\") pod \"e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8\" (UID: \"87ac6502-78a2-41ed-9033-77eecf21086f\") " pod="openshift-ingress/e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8" Nov 03 02:14:22.781730 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:22.778625 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87ac6502-78a2-41ed-9033-77eecf21086f-workload-certs\") pod \"e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8\" (UID: \"87ac6502-78a2-41ed-9033-77eecf21086f\") " pod="openshift-ingress/e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8" Nov 03 02:14:22.781730 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:22.778652 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87ac6502-78a2-41ed-9033-77eecf21086f-credential-socket\") pod \"e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8\" (UID: \"87ac6502-78a2-41ed-9033-77eecf21086f\") " pod="openshift-ingress/e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8" Nov 03 02:14:22.781730 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:22.778747 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87ac6502-78a2-41ed-9033-77eecf21086f-workload-socket\") pod \"e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8\" (UID: \"87ac6502-78a2-41ed-9033-77eecf21086f\") " pod="openshift-ingress/e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8" Nov 03 02:14:22.781730 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:22.779269 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87ac6502-78a2-41ed-9033-77eecf21086f-workload-socket\") pod \"e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8\" (UID: \"87ac6502-78a2-41ed-9033-77eecf21086f\") " pod="openshift-ingress/e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8" Nov 03 02:14:22.781730 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:22.779544 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87ac6502-78a2-41ed-9033-77eecf21086f-istio-data\") pod \"e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8\" (UID: \"87ac6502-78a2-41ed-9033-77eecf21086f\") " pod="openshift-ingress/e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8" Nov 03 02:14:22.799376 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:22.798026 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87ac6502-78a2-41ed-9033-77eecf21086f-workload-certs\") pod \"e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8\" (UID: \"87ac6502-78a2-41ed-9033-77eecf21086f\") " pod="openshift-ingress/e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8" Nov 03 02:14:22.799376 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:22.798646 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87ac6502-78a2-41ed-9033-77eecf21086f-credential-socket\") pod \"e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8\" (UID: \"87ac6502-78a2-41ed-9033-77eecf21086f\") " pod="openshift-ingress/e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8" Nov 03 02:14:22.811388 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:22.810294 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87ac6502-78a2-41ed-9033-77eecf21086f-istiod-ca-cert\") pod \"e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8\" (UID: \"87ac6502-78a2-41ed-9033-77eecf21086f\") " pod="openshift-ingress/e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8" Nov 03 02:14:22.837377 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:22.836697 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4559\" (UniqueName: \"kubernetes.io/projected/87ac6502-78a2-41ed-9033-77eecf21086f-kube-api-access-l4559\") pod \"e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8\" (UID: \"87ac6502-78a2-41ed-9033-77eecf21086f\") " pod="openshift-ingress/e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8" Nov 03 02:14:22.853232 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:22.849162 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87ac6502-78a2-41ed-9033-77eecf21086f-istio-podinfo\") pod \"e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8\" (UID: \"87ac6502-78a2-41ed-9033-77eecf21086f\") " pod="openshift-ingress/e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8" Nov 03 02:14:22.866080 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:22.866046 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87ac6502-78a2-41ed-9033-77eecf21086f-istio-envoy\") pod \"e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8\" (UID: \"87ac6502-78a2-41ed-9033-77eecf21086f\") " pod="openshift-ingress/e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8" Nov 03 02:14:22.867863 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:22.867814 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87ac6502-78a2-41ed-9033-77eecf21086f-istio-token\") pod \"e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8\" (UID: \"87ac6502-78a2-41ed-9033-77eecf21086f\") " pod="openshift-ingress/e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8" Nov 03 02:14:23.070799 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:23.070763 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8" Nov 03 02:14:23.500020 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:23.499906 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8"] Nov 03 02:14:23.504717 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:14:23.504681 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87ac6502_78a2_41ed_9033_77eecf21086f.slice/crio-8e2b810cfdc89c37625989c2e6aa943702902b51a648bbff6f21b66694017edf WatchSource:0}: Error finding container 8e2b810cfdc89c37625989c2e6aa943702902b51a648bbff6f21b66694017edf: Status 404 returned error can't find the container with id 8e2b810cfdc89c37625989c2e6aa943702902b51a648bbff6f21b66694017edf Nov 03 02:14:23.507302 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:23.507256 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:14:23.507418 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:23.507373 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:14:23.507473 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:23.507422 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:14:23.916591 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:14:23.916549 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:14:24.064431 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:24.064386 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8" event={"ID":"87ac6502-78a2-41ed-9033-77eecf21086f","Type":"ContainerStarted","Data":"80fa1b08040a5dd9081261734bb32061db14746eb9c9c04e4d5e91f90881f114"} Nov 03 02:14:24.064635 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:24.064478 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8" event={"ID":"87ac6502-78a2-41ed-9033-77eecf21086f","Type":"ContainerStarted","Data":"8e2b810cfdc89c37625989c2e6aa943702902b51a648bbff6f21b66694017edf"} Nov 03 02:14:24.071860 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:24.071835 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8" Nov 03 02:14:24.077208 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:24.077184 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8" Nov 03 02:14:24.220036 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:24.219900 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8" podStartSLOduration=2.219885218 podStartE2EDuration="2.219885218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:14:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:14:24.219833839 +0000 UTC m=+1341.479032011" watchObservedRunningTime="2026-11-03 02:14:24.219885218 +0000 UTC m=+1341.479083372" Nov 03 02:14:25.070935 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:25.070902 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8" Nov 03 02:14:25.072044 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:25.072021 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8" Nov 03 02:14:48.333288 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:48.333211 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d705e3c-z46sq"] Nov 03 02:14:48.338581 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:48.338558 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d705e3c-z46sq" Nov 03 02:14:48.390687 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:48.390652 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d705e3c-z46sq"] Nov 03 02:14:48.433923 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:48.433887 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:14:48.434083 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:48.433922 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:14:48.434083 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:48.433939 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:14:48.434758 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:48.434736 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:14:48.468346 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:48.468315 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27c5d\" (UniqueName: \"kubernetes.io/projected/ba867d73-335f-4cdd-9dea-beec4c925dd3-kube-api-access-27c5d\") pod \"maas-api-revoke-sub-4d705e3c-z46sq\" (UID: \"ba867d73-335f-4cdd-9dea-beec4c925dd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d705e3c-z46sq" Nov 03 02:14:48.468514 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:48.468435 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba867d73-335f-4cdd-9dea-beec4c925dd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d705e3c-z46sq\" (UID: \"ba867d73-335f-4cdd-9dea-beec4c925dd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d705e3c-z46sq" Nov 03 02:14:48.468573 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:48.468558 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba867d73-335f-4cdd-9dea-beec4c925dd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d705e3c-z46sq\" (UID: \"ba867d73-335f-4cdd-9dea-beec4c925dd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d705e3c-z46sq" Nov 03 02:14:48.569322 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:48.569287 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba867d73-335f-4cdd-9dea-beec4c925dd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d705e3c-z46sq\" (UID: \"ba867d73-335f-4cdd-9dea-beec4c925dd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d705e3c-z46sq" Nov 03 02:14:48.569534 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:48.569423 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27c5d\" (UniqueName: \"kubernetes.io/projected/ba867d73-335f-4cdd-9dea-beec4c925dd3-kube-api-access-27c5d\") pod \"maas-api-revoke-sub-4d705e3c-z46sq\" (UID: \"ba867d73-335f-4cdd-9dea-beec4c925dd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d705e3c-z46sq" Nov 03 02:14:48.569534 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:48.569480 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba867d73-335f-4cdd-9dea-beec4c925dd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d705e3c-z46sq\" (UID: \"ba867d73-335f-4cdd-9dea-beec4c925dd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d705e3c-z46sq" Nov 03 02:14:48.570073 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:48.570053 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba867d73-335f-4cdd-9dea-beec4c925dd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d705e3c-z46sq\" (UID: \"ba867d73-335f-4cdd-9dea-beec4c925dd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d705e3c-z46sq" Nov 03 02:14:48.571577 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:48.571558 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba867d73-335f-4cdd-9dea-beec4c925dd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d705e3c-z46sq\" (UID: \"ba867d73-335f-4cdd-9dea-beec4c925dd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d705e3c-z46sq" Nov 03 02:14:48.622769 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:48.622698 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27c5d\" (UniqueName: \"kubernetes.io/projected/ba867d73-335f-4cdd-9dea-beec4c925dd3-kube-api-access-27c5d\") pod \"maas-api-revoke-sub-4d705e3c-z46sq\" (UID: \"ba867d73-335f-4cdd-9dea-beec4c925dd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d705e3c-z46sq" Nov 03 02:14:48.653049 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:48.653017 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d705e3c-z46sq" Nov 03 02:14:49.027829 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:49.027731 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d705e3c-z46sq"] Nov 03 02:14:49.032588 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:14:49.032556 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba867d73_335f_4cdd_9dea_beec4c925dd3.slice/crio-9c9430ebbef159c35becafcb6ac3af91dfcc0761a7676a7e9bdb14af8cfcbeb3 WatchSource:0}: Error finding container 9c9430ebbef159c35becafcb6ac3af91dfcc0761a7676a7e9bdb14af8cfcbeb3: Status 404 returned error can't find the container with id 9c9430ebbef159c35becafcb6ac3af91dfcc0761a7676a7e9bdb14af8cfcbeb3 Nov 03 02:14:49.263938 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:49.263901 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d705e3c-z46sq" event={"ID":"ba867d73-335f-4cdd-9dea-beec4c925dd3","Type":"ContainerStarted","Data":"9c9430ebbef159c35becafcb6ac3af91dfcc0761a7676a7e9bdb14af8cfcbeb3"} Nov 03 02:14:50.271666 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:50.271573 5605 generic.go:356] "Generic (PLEG): container finished" podID="ba867d73-335f-4cdd-9dea-beec4c925dd3" containerID="d05d459b581769ef02f92eb25ba6c209e2cf53e3e781552a0f0ffa0b0b0df867" exitCode=0 Nov 03 02:14:50.272050 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:50.271660 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d705e3c-z46sq" event={"ID":"ba867d73-335f-4cdd-9dea-beec4c925dd3","Type":"ContainerDied","Data":"d05d459b581769ef02f92eb25ba6c209e2cf53e3e781552a0f0ffa0b0b0df867"} Nov 03 02:14:51.431835 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:51.431811 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d705e3c-z46sq" Nov 03 02:14:51.508098 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:51.508066 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba867d73-335f-4cdd-9dea-beec4c925dd3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba867d73-335f-4cdd-9dea-beec4c925dd3-maas-api-openshift-ca\") pod \"ba867d73-335f-4cdd-9dea-beec4c925dd3\" (UID: \"ba867d73-335f-4cdd-9dea-beec4c925dd3\") " Nov 03 02:14:51.508298 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:51.508242 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba867d73-335f-4cdd-9dea-beec4c925dd3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba867d73-335f-4cdd-9dea-beec4c925dd3-maas-api-xks-ca\") pod \"ba867d73-335f-4cdd-9dea-beec4c925dd3\" (UID: \"ba867d73-335f-4cdd-9dea-beec4c925dd3\") " Nov 03 02:14:51.508400 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:51.508319 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba867d73-335f-4cdd-9dea-beec4c925dd3-kube-api-access-27c5d\" (UniqueName: \"kubernetes.io/projected/ba867d73-335f-4cdd-9dea-beec4c925dd3-kube-api-access-27c5d\") pod \"ba867d73-335f-4cdd-9dea-beec4c925dd3\" (UID: \"ba867d73-335f-4cdd-9dea-beec4c925dd3\") " Nov 03 02:14:51.508530 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:51.508500 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba867d73-335f-4cdd-9dea-beec4c925dd3-maas-api-openshift-ca" pod "ba867d73-335f-4cdd-9dea-beec4c925dd3" (UID: "ba867d73-335f-4cdd-9dea-beec4c925dd3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:14:51.508849 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:51.508824 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba867d73-335f-4cdd-9dea-beec4c925dd3-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:14:51.510506 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:51.510479 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba867d73-335f-4cdd-9dea-beec4c925dd3-maas-api-xks-ca" pod "ba867d73-335f-4cdd-9dea-beec4c925dd3" (UID: "ba867d73-335f-4cdd-9dea-beec4c925dd3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:14:51.510590 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:51.510515 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba867d73-335f-4cdd-9dea-beec4c925dd3-kube-api-access-27c5d" pod "ba867d73-335f-4cdd-9dea-beec4c925dd3" (UID: "ba867d73-335f-4cdd-9dea-beec4c925dd3"). InnerVolumeSpecName "kube-api-access-27c5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:14:51.609828 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:51.609788 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba867d73-335f-4cdd-9dea-beec4c925dd3-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:14:51.609828 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:51.609821 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27c5d\" (UniqueName: \"kubernetes.io/projected/ba867d73-335f-4cdd-9dea-beec4c925dd3-kube-api-access-27c5d\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:14:52.286340 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:52.286308 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d705e3c-z46sq" Nov 03 02:14:52.286340 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:52.286320 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d705e3c-z46sq" event={"ID":"ba867d73-335f-4cdd-9dea-beec4c925dd3","Type":"ContainerDied","Data":"9c9430ebbef159c35becafcb6ac3af91dfcc0761a7676a7e9bdb14af8cfcbeb3"} Nov 03 02:14:52.286577 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:14:52.286387 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c9430ebbef159c35becafcb6ac3af91dfcc0761a7676a7e9bdb14af8cfcbeb3" Nov 03 02:14:53.914706 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:14:53.914662 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:15:00.157802 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:00.157767 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894535-4sw2s"] Nov 03 02:15:00.158500 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:00.158469 5605 cpu_manager.go:414] "containerMap: removing container" podUID="ba867d73-335f-4cdd-9dea-beec4c925dd3" containerName="revoke-keys" Nov 03 02:15:00.158500 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:00.158494 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba867d73-335f-4cdd-9dea-beec4c925dd3" containerName="revoke-keys" Nov 03 02:15:00.158684 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:00.158607 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba867d73-335f-4cdd-9dea-beec4c925dd3" containerName="revoke-keys" Nov 03 02:15:00.162140 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:00.162124 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a772c3-29894535-72559"] Nov 03 02:15:00.162278 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:00.162254 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894535-4sw2s" Nov 03 02:15:00.166142 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:00.166121 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894535-gplpr"] Nov 03 02:15:00.166263 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:00.166230 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a772c3-29894535-72559" Nov 03 02:15:00.169827 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:00.169806 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894535-gplpr" Nov 03 02:15:00.204327 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:00.204300 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7f332d-29894535-mn8qq"] Nov 03 02:15:00.207986 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:00.207969 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894535-nj7sx"] Nov 03 02:15:00.208092 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:00.208076 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7f332d-29894535-mn8qq" Nov 03 02:15:00.211241 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:00.211218 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:15:00.211349 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:00.211300 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:15:00.212464 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:00.212448 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894535-nj7sx" Nov 03 02:15:00.234624 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:00.234598 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d32fcb0-b25a-4a1c-a480-b32cc423d1bb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-a772c3-29894535-72559\" (UID: \"9d32fcb0-b25a-4a1c-a480-b32cc423d1bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a772c3-29894535-72559" Nov 03 02:15:00.234732 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:00.234667 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec4f26be-fd2d-4f37-8200-eee788394037-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c7d758-29894535-4sw2s\" (UID: \"ec4f26be-fd2d-4f37-8200-eee788394037\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894535-4sw2s" Nov 03 02:15:00.234786 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:00.234765 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4thgz\" (UniqueName: \"kubernetes.io/projected/ec4f26be-fd2d-4f37-8200-eee788394037-kube-api-access-4thgz\") pod \"maas-api-key-cleanup-e2e-worker-w4-c7d758-29894535-4sw2s\" (UID: \"ec4f26be-fd2d-4f37-8200-eee788394037\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894535-4sw2s" Nov 03 02:15:00.234932 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:00.234919 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcgh2\" (UniqueName: \"kubernetes.io/projected/9d32fcb0-b25a-4a1c-a480-b32cc423d1bb-kube-api-access-kcgh2\") pod \"maas-api-key-cleanup-e2e-api-a-a772c3-29894535-72559\" (UID: \"9d32fcb0-b25a-4a1c-a480-b32cc423d1bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a772c3-29894535-72559" Nov 03 02:15:00.248033 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:00.247998 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:15:00.263556 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:00.263515 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a772c3-29894535-72559"] Nov 03 02:15:00.263744 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:00.263626 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894535-4sw2s"] Nov 03 02:15:00.267322 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:00.267292 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894535-gplpr"] Nov 03 02:15:00.304240 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:00.304206 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894535-nj7sx"] Nov 03 02:15:00.335990 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:00.335957 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/988f453a-1c54-4410-85a1-aff8a5e66af4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-7f332d-29894535-mn8qq\" (UID: \"988f453a-1c54-4410-85a1-aff8a5e66af4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7f332d-29894535-mn8qq" Nov 03 02:15:00.336162 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:00.336010 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcgh2\" (UniqueName: \"kubernetes.io/projected/9d32fcb0-b25a-4a1c-a480-b32cc423d1bb-kube-api-access-kcgh2\") pod \"maas-api-key-cleanup-e2e-api-a-a772c3-29894535-72559\" (UID: \"9d32fcb0-b25a-4a1c-a480-b32cc423d1bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a772c3-29894535-72559" Nov 03 02:15:00.336162 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:00.336071 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x67kc\" (UniqueName: \"kubernetes.io/projected/988f453a-1c54-4410-85a1-aff8a5e66af4-kube-api-access-x67kc\") pod \"maas-api-key-cleanup-e2e-api-a-7f332d-29894535-mn8qq\" (UID: \"988f453a-1c54-4410-85a1-aff8a5e66af4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7f332d-29894535-mn8qq" Nov 03 02:15:00.336162 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:00.336154 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d32fcb0-b25a-4a1c-a480-b32cc423d1bb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-a772c3-29894535-72559\" (UID: \"9d32fcb0-b25a-4a1c-a480-b32cc423d1bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a772c3-29894535-72559" Nov 03 02:15:00.336296 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:00.336244 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d38408a-0e73-4d42-b78c-6ce430d3927c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-521f1f-29894535-gplpr\" (UID: \"7d38408a-0e73-4d42-b78c-6ce430d3927c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894535-gplpr" Nov 03 02:15:00.336409 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:00.336393 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec4f26be-fd2d-4f37-8200-eee788394037-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c7d758-29894535-4sw2s\" (UID: \"ec4f26be-fd2d-4f37-8200-eee788394037\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894535-4sw2s" Nov 03 02:15:00.336474 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:00.336434 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vzzg\" (UniqueName: \"kubernetes.io/projected/7d38408a-0e73-4d42-b78c-6ce430d3927c-kube-api-access-5vzzg\") pod \"maas-api-key-cleanup-e2e-worker-w2-521f1f-29894535-gplpr\" (UID: \"7d38408a-0e73-4d42-b78c-6ce430d3927c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894535-gplpr" Nov 03 02:15:00.336538 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:00.336475 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4thgz\" (UniqueName: \"kubernetes.io/projected/ec4f26be-fd2d-4f37-8200-eee788394037-kube-api-access-4thgz\") pod \"maas-api-key-cleanup-e2e-worker-w4-c7d758-29894535-4sw2s\" (UID: \"ec4f26be-fd2d-4f37-8200-eee788394037\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894535-4sw2s" Nov 03 02:15:00.336612 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:00.336594 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4a0e603-b309-4a5b-861b-f1780b71f6e0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894535-nj7sx\" (UID: \"b4a0e603-b309-4a5b-861b-f1780b71f6e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894535-nj7sx" Nov 03 02:15:00.336663 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:00.336621 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm5qn\" (UniqueName: \"kubernetes.io/projected/b4a0e603-b309-4a5b-861b-f1780b71f6e0-kube-api-access-jm5qn\") pod \"maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894535-nj7sx\" (UID: \"b4a0e603-b309-4a5b-861b-f1780b71f6e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894535-nj7sx" Nov 03 02:15:00.336965 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:00.336942 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d32fcb0-b25a-4a1c-a480-b32cc423d1bb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-a772c3-29894535-72559\" (UID: \"9d32fcb0-b25a-4a1c-a480-b32cc423d1bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a772c3-29894535-72559" Nov 03 02:15:00.337204 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:00.337182 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec4f26be-fd2d-4f37-8200-eee788394037-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c7d758-29894535-4sw2s\" (UID: \"ec4f26be-fd2d-4f37-8200-eee788394037\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894535-4sw2s" Nov 03 02:15:00.352048 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:00.352023 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7f332d-29894535-mn8qq"] Nov 03 02:15:00.444219 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:00.444128 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vzzg\" (UniqueName: \"kubernetes.io/projected/7d38408a-0e73-4d42-b78c-6ce430d3927c-kube-api-access-5vzzg\") pod \"maas-api-key-cleanup-e2e-worker-w2-521f1f-29894535-gplpr\" (UID: \"7d38408a-0e73-4d42-b78c-6ce430d3927c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894535-gplpr" Nov 03 02:15:00.444441 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:00.444276 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4a0e603-b309-4a5b-861b-f1780b71f6e0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894535-nj7sx\" (UID: \"b4a0e603-b309-4a5b-861b-f1780b71f6e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894535-nj7sx" Nov 03 02:15:00.444441 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:00.444308 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm5qn\" (UniqueName: \"kubernetes.io/projected/b4a0e603-b309-4a5b-861b-f1780b71f6e0-kube-api-access-jm5qn\") pod \"maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894535-nj7sx\" (UID: \"b4a0e603-b309-4a5b-861b-f1780b71f6e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894535-nj7sx" Nov 03 02:15:00.444441 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:00.444400 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/988f453a-1c54-4410-85a1-aff8a5e66af4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-7f332d-29894535-mn8qq\" (UID: \"988f453a-1c54-4410-85a1-aff8a5e66af4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7f332d-29894535-mn8qq" Nov 03 02:15:00.444620 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:00.444446 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x67kc\" (UniqueName: \"kubernetes.io/projected/988f453a-1c54-4410-85a1-aff8a5e66af4-kube-api-access-x67kc\") pod \"maas-api-key-cleanup-e2e-api-a-7f332d-29894535-mn8qq\" (UID: \"988f453a-1c54-4410-85a1-aff8a5e66af4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7f332d-29894535-mn8qq" Nov 03 02:15:00.444620 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:00.444590 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d38408a-0e73-4d42-b78c-6ce430d3927c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-521f1f-29894535-gplpr\" (UID: \"7d38408a-0e73-4d42-b78c-6ce430d3927c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894535-gplpr" Nov 03 02:15:00.445319 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:00.445291 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d38408a-0e73-4d42-b78c-6ce430d3927c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-521f1f-29894535-gplpr\" (UID: \"7d38408a-0e73-4d42-b78c-6ce430d3927c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894535-gplpr" Nov 03 02:15:00.446275 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:00.446252 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4a0e603-b309-4a5b-861b-f1780b71f6e0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894535-nj7sx\" (UID: \"b4a0e603-b309-4a5b-861b-f1780b71f6e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894535-nj7sx" Nov 03 02:15:00.446645 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:00.446624 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/988f453a-1c54-4410-85a1-aff8a5e66af4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-7f332d-29894535-mn8qq\" (UID: \"988f453a-1c54-4410-85a1-aff8a5e66af4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7f332d-29894535-mn8qq" Nov 03 02:15:00.470375 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:00.461384 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcgh2\" (UniqueName: \"kubernetes.io/projected/9d32fcb0-b25a-4a1c-a480-b32cc423d1bb-kube-api-access-kcgh2\") pod \"maas-api-key-cleanup-e2e-api-a-a772c3-29894535-72559\" (UID: \"9d32fcb0-b25a-4a1c-a480-b32cc423d1bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a772c3-29894535-72559" Nov 03 02:15:00.485055 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:00.485001 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4thgz\" (UniqueName: \"kubernetes.io/projected/ec4f26be-fd2d-4f37-8200-eee788394037-kube-api-access-4thgz\") pod \"maas-api-key-cleanup-e2e-worker-w4-c7d758-29894535-4sw2s\" (UID: \"ec4f26be-fd2d-4f37-8200-eee788394037\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894535-4sw2s" Nov 03 02:15:00.485794 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:00.485758 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a772c3-29894535-72559" Nov 03 02:15:00.498445 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:00.498415 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vzzg\" (UniqueName: \"kubernetes.io/projected/7d38408a-0e73-4d42-b78c-6ce430d3927c-kube-api-access-5vzzg\") pod \"maas-api-key-cleanup-e2e-worker-w2-521f1f-29894535-gplpr\" (UID: \"7d38408a-0e73-4d42-b78c-6ce430d3927c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894535-gplpr" Nov 03 02:15:00.498739 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:00.498717 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm5qn\" (UniqueName: \"kubernetes.io/projected/b4a0e603-b309-4a5b-861b-f1780b71f6e0-kube-api-access-jm5qn\") pod \"maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894535-nj7sx\" (UID: \"b4a0e603-b309-4a5b-861b-f1780b71f6e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894535-nj7sx" Nov 03 02:15:00.499494 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:00.499475 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x67kc\" (UniqueName: \"kubernetes.io/projected/988f453a-1c54-4410-85a1-aff8a5e66af4-kube-api-access-x67kc\") pod \"maas-api-key-cleanup-e2e-api-a-7f332d-29894535-mn8qq\" (UID: \"988f453a-1c54-4410-85a1-aff8a5e66af4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7f332d-29894535-mn8qq" Nov 03 02:15:00.526407 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:00.526375 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7f332d-29894535-mn8qq" Nov 03 02:15:00.532306 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:00.532278 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894535-nj7sx" Nov 03 02:15:00.776498 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:00.776413 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894535-4sw2s" Nov 03 02:15:00.792764 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:00.792741 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894535-gplpr" Nov 03 02:15:00.860041 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:00.860001 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a772c3-29894535-72559"] Nov 03 02:15:00.867653 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:15:00.867623 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d32fcb0_b25a_4a1c_a480_b32cc423d1bb.slice/crio-6f0e64211273e2d26e974bac658ec49665bef8f5748321e37e0de5c42470c70c WatchSource:0}: Error finding container 6f0e64211273e2d26e974bac658ec49665bef8f5748321e37e0de5c42470c70c: Status 404 returned error can't find the container with id 6f0e64211273e2d26e974bac658ec49665bef8f5748321e37e0de5c42470c70c Nov 03 02:15:00.915154 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:00.915126 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7f332d-29894535-mn8qq"] Nov 03 02:15:00.924812 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:15:00.924786 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod988f453a_1c54_4410_85a1_aff8a5e66af4.slice/crio-c7d4fed4362f9ce0730ffabe5f5193ead6d4df19a1b6edb58c451aa331e379c2 WatchSource:0}: Error finding container c7d4fed4362f9ce0730ffabe5f5193ead6d4df19a1b6edb58c451aa331e379c2: Status 404 returned error can't find the container with id c7d4fed4362f9ce0730ffabe5f5193ead6d4df19a1b6edb58c451aa331e379c2 Nov 03 02:15:00.962918 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:00.962897 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894535-nj7sx"] Nov 03 02:15:00.966964 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:15:00.966938 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4a0e603_b309_4a5b_861b_f1780b71f6e0.slice/crio-6aa8d91438b399d71734e9d627ee174072c4933e0b4cf233899a180977e8b78c WatchSource:0}: Error finding container 6aa8d91438b399d71734e9d627ee174072c4933e0b4cf233899a180977e8b78c: Status 404 returned error can't find the container with id 6aa8d91438b399d71734e9d627ee174072c4933e0b4cf233899a180977e8b78c Nov 03 02:15:01.146420 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:01.146383 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894535-4sw2s"] Nov 03 02:15:01.157015 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:15:01.156989 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec4f26be_fd2d_4f37_8200_eee788394037.slice/crio-a769008b15cc2b0e7fefde668ab70135e15edfe1ea98a4d7ea3e78cdcb9abba2 WatchSource:0}: Error finding container a769008b15cc2b0e7fefde668ab70135e15edfe1ea98a4d7ea3e78cdcb9abba2: Status 404 returned error can't find the container with id a769008b15cc2b0e7fefde668ab70135e15edfe1ea98a4d7ea3e78cdcb9abba2 Nov 03 02:15:01.180975 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:01.180949 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894535-gplpr"] Nov 03 02:15:01.184734 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:15:01.184708 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d38408a_0e73_4d42_b78c_6ce430d3927c.slice/crio-b2ee9b61a1ef716cc4d5bf23eb12ca686d30363e847883c1d135a1f78ee8b4e8 WatchSource:0}: Error finding container b2ee9b61a1ef716cc4d5bf23eb12ca686d30363e847883c1d135a1f78ee8b4e8: Status 404 returned error can't find the container with id b2ee9b61a1ef716cc4d5bf23eb12ca686d30363e847883c1d135a1f78ee8b4e8 Nov 03 02:15:01.342814 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:01.342731 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7f332d-29894535-mn8qq" event={"ID":"988f453a-1c54-4410-85a1-aff8a5e66af4","Type":"ContainerStarted","Data":"c7d4fed4362f9ce0730ffabe5f5193ead6d4df19a1b6edb58c451aa331e379c2"} Nov 03 02:15:01.344433 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:01.344397 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a772c3-29894535-72559" event={"ID":"9d32fcb0-b25a-4a1c-a480-b32cc423d1bb","Type":"ContainerStarted","Data":"1e0453e17bc42ab1ce8e52688fdd2d750e67a2f820edeaab216c3768f71a96ae"} Nov 03 02:15:01.344574 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:01.344459 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a772c3-29894535-72559" event={"ID":"9d32fcb0-b25a-4a1c-a480-b32cc423d1bb","Type":"ContainerStarted","Data":"6f0e64211273e2d26e974bac658ec49665bef8f5748321e37e0de5c42470c70c"} Nov 03 02:15:01.345725 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:01.345704 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894535-4sw2s" event={"ID":"ec4f26be-fd2d-4f37-8200-eee788394037","Type":"ContainerStarted","Data":"a769008b15cc2b0e7fefde668ab70135e15edfe1ea98a4d7ea3e78cdcb9abba2"} Nov 03 02:15:01.347249 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:01.347226 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894535-nj7sx" event={"ID":"b4a0e603-b309-4a5b-861b-f1780b71f6e0","Type":"ContainerStarted","Data":"5bd87b69b4f0e03bca70123f2fe63305bac93000f5f31f2a490f1ea60e61d2fd"} Nov 03 02:15:01.347352 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:01.347274 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894535-nj7sx" event={"ID":"b4a0e603-b309-4a5b-861b-f1780b71f6e0","Type":"ContainerStarted","Data":"6aa8d91438b399d71734e9d627ee174072c4933e0b4cf233899a180977e8b78c"} Nov 03 02:15:01.348406 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:01.348382 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894535-gplpr" event={"ID":"7d38408a-0e73-4d42-b78c-6ce430d3927c","Type":"ContainerStarted","Data":"b2ee9b61a1ef716cc4d5bf23eb12ca686d30363e847883c1d135a1f78ee8b4e8"} Nov 03 02:15:01.451436 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:01.451377 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a772c3-29894535-72559" podStartSLOduration=1.451350348 podStartE2EDuration="1.451350348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:15:01.449071184 +0000 UTC m=+1378.708269349" watchObservedRunningTime="2026-11-03 02:15:01.451350348 +0000 UTC m=+1378.710548502" Nov 03 02:15:01.560127 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:01.560077 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894535-nj7sx" podStartSLOduration=1.560062565 podStartE2EDuration="1.560062565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:15:01.558300674 +0000 UTC m=+1378.817498831" watchObservedRunningTime="2026-11-03 02:15:01.560062565 +0000 UTC m=+1378.819260719" Nov 03 02:15:02.358156 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:02.358122 5605 generic.go:356] "Generic (PLEG): container finished" podID="988f453a-1c54-4410-85a1-aff8a5e66af4" containerID="a7bbf8979f432afb69f7234ba9b9004cfcb2c1ea4301206ccf163de3e824f78f" exitCode=60 Nov 03 02:15:02.358638 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:02.358193 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7f332d-29894535-mn8qq" event={"ID":"988f453a-1c54-4410-85a1-aff8a5e66af4","Type":"ContainerDied","Data":"a7bbf8979f432afb69f7234ba9b9004cfcb2c1ea4301206ccf163de3e824f78f"} Nov 03 02:15:02.358638 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:02.358535 5605 scope.go:122] "RemoveContainer" containerID="a7bbf8979f432afb69f7234ba9b9004cfcb2c1ea4301206ccf163de3e824f78f" Nov 03 02:15:02.359879 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:02.359857 5605 generic.go:356] "Generic (PLEG): container finished" podID="9d32fcb0-b25a-4a1c-a480-b32cc423d1bb" containerID="1e0453e17bc42ab1ce8e52688fdd2d750e67a2f820edeaab216c3768f71a96ae" exitCode=60 Nov 03 02:15:02.360010 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:02.359951 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a772c3-29894535-72559" event={"ID":"9d32fcb0-b25a-4a1c-a480-b32cc423d1bb","Type":"ContainerDied","Data":"1e0453e17bc42ab1ce8e52688fdd2d750e67a2f820edeaab216c3768f71a96ae"} Nov 03 02:15:02.360232 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:02.360151 5605 scope.go:122] "RemoveContainer" containerID="1e0453e17bc42ab1ce8e52688fdd2d750e67a2f820edeaab216c3768f71a96ae" Nov 03 02:15:02.361938 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:02.361797 5605 generic.go:356] "Generic (PLEG): container finished" podID="ec4f26be-fd2d-4f37-8200-eee788394037" containerID="686886b4dbe254bd7cb737d9e7f6335658c1952cd34850a3dc40160f9a026e8f" exitCode=60 Nov 03 02:15:02.361938 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:02.361858 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894535-4sw2s" event={"ID":"ec4f26be-fd2d-4f37-8200-eee788394037","Type":"ContainerDied","Data":"686886b4dbe254bd7cb737d9e7f6335658c1952cd34850a3dc40160f9a026e8f"} Nov 03 02:15:02.362120 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:02.362102 5605 scope.go:122] "RemoveContainer" containerID="686886b4dbe254bd7cb737d9e7f6335658c1952cd34850a3dc40160f9a026e8f" Nov 03 02:15:02.363750 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:02.363729 5605 generic.go:356] "Generic (PLEG): container finished" podID="b4a0e603-b309-4a5b-861b-f1780b71f6e0" containerID="5bd87b69b4f0e03bca70123f2fe63305bac93000f5f31f2a490f1ea60e61d2fd" exitCode=60 Nov 03 02:15:02.363848 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:02.363835 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894535-nj7sx" event={"ID":"b4a0e603-b309-4a5b-861b-f1780b71f6e0","Type":"ContainerDied","Data":"5bd87b69b4f0e03bca70123f2fe63305bac93000f5f31f2a490f1ea60e61d2fd"} Nov 03 02:15:02.364151 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:02.364137 5605 scope.go:122] "RemoveContainer" containerID="5bd87b69b4f0e03bca70123f2fe63305bac93000f5f31f2a490f1ea60e61d2fd" Nov 03 02:15:02.366762 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:02.366741 5605 generic.go:356] "Generic (PLEG): container finished" podID="7d38408a-0e73-4d42-b78c-6ce430d3927c" containerID="73d630e56445661f467bb3e27e87cd3d7dc527ed3bdef6ee720a18b70767e148" exitCode=60 Nov 03 02:15:02.366847 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:02.366813 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894535-gplpr" event={"ID":"7d38408a-0e73-4d42-b78c-6ce430d3927c","Type":"ContainerDied","Data":"73d630e56445661f467bb3e27e87cd3d7dc527ed3bdef6ee720a18b70767e148"} Nov 03 02:15:02.366976 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:02.366964 5605 scope.go:122] "RemoveContainer" containerID="73d630e56445661f467bb3e27e87cd3d7dc527ed3bdef6ee720a18b70767e148" Nov 03 02:15:03.376432 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:03.376325 5605 generic.go:356] "Generic (PLEG): container finished" podID="988f453a-1c54-4410-85a1-aff8a5e66af4" containerID="7782212b623193f38c78cb995d7b70d0637bce523299a46dc7f5011aab6cd77f" exitCode=60 Nov 03 02:15:03.376802 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:03.376423 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7f332d-29894535-mn8qq" event={"ID":"988f453a-1c54-4410-85a1-aff8a5e66af4","Type":"ContainerDied","Data":"7782212b623193f38c78cb995d7b70d0637bce523299a46dc7f5011aab6cd77f"} Nov 03 02:15:03.376802 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:03.376500 5605 scope.go:122] "RemoveContainer" containerID="a7bbf8979f432afb69f7234ba9b9004cfcb2c1ea4301206ccf163de3e824f78f" Nov 03 02:15:03.376802 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:03.376796 5605 scope.go:122] "RemoveContainer" containerID="7782212b623193f38c78cb995d7b70d0637bce523299a46dc7f5011aab6cd77f" Nov 03 02:15:03.377220 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:15:03.377194 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-7f332d-29894535-mn8qq_odh-ai-gateway-infra(988f453a-1c54-4410-85a1-aff8a5e66af4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7f332d-29894535-mn8qq" podUID="988f453a-1c54-4410-85a1-aff8a5e66af4" Nov 03 02:15:03.379020 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:03.379001 5605 generic.go:356] "Generic (PLEG): container finished" podID="9d32fcb0-b25a-4a1c-a480-b32cc423d1bb" containerID="8ee392b8f0224e58ff06fa8cda676918d86db283711c6e82b50538a9d6e53791" exitCode=60 Nov 03 02:15:03.379134 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:03.379080 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a772c3-29894535-72559" event={"ID":"9d32fcb0-b25a-4a1c-a480-b32cc423d1bb","Type":"ContainerDied","Data":"8ee392b8f0224e58ff06fa8cda676918d86db283711c6e82b50538a9d6e53791"} Nov 03 02:15:03.379396 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:03.379375 5605 scope.go:122] "RemoveContainer" containerID="8ee392b8f0224e58ff06fa8cda676918d86db283711c6e82b50538a9d6e53791" Nov 03 02:15:03.379683 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:15:03.379660 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-a772c3-29894535-72559_odh-ai-gateway-infra(9d32fcb0-b25a-4a1c-a480-b32cc423d1bb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a772c3-29894535-72559" podUID="9d32fcb0-b25a-4a1c-a480-b32cc423d1bb" Nov 03 02:15:03.381181 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:03.381158 5605 generic.go:356] "Generic (PLEG): container finished" podID="ec4f26be-fd2d-4f37-8200-eee788394037" containerID="0c6b4d10930b7f5d123e3a3dcd55446722816a032f10bbfdbbf7cd0a5d5987e7" exitCode=60 Nov 03 02:15:03.381295 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:03.381225 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894535-4sw2s" event={"ID":"ec4f26be-fd2d-4f37-8200-eee788394037","Type":"ContainerDied","Data":"0c6b4d10930b7f5d123e3a3dcd55446722816a032f10bbfdbbf7cd0a5d5987e7"} Nov 03 02:15:03.381498 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:03.381482 5605 scope.go:122] "RemoveContainer" containerID="0c6b4d10930b7f5d123e3a3dcd55446722816a032f10bbfdbbf7cd0a5d5987e7" Nov 03 02:15:03.381779 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:15:03.381761 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-c7d758-29894535-4sw2s_odh-ai-gateway-infra(ec4f26be-fd2d-4f37-8200-eee788394037)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894535-4sw2s" podUID="ec4f26be-fd2d-4f37-8200-eee788394037" Nov 03 02:15:03.383290 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:03.383271 5605 generic.go:356] "Generic (PLEG): container finished" podID="b4a0e603-b309-4a5b-861b-f1780b71f6e0" containerID="285dd362dc4629efd82491ee04368fb636d17b7d8c34bc45a74f2c47b419ab4a" exitCode=60 Nov 03 02:15:03.383410 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:03.383389 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894535-nj7sx" event={"ID":"b4a0e603-b309-4a5b-861b-f1780b71f6e0","Type":"ContainerDied","Data":"285dd362dc4629efd82491ee04368fb636d17b7d8c34bc45a74f2c47b419ab4a"} Nov 03 02:15:03.383674 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:03.383654 5605 scope.go:122] "RemoveContainer" containerID="285dd362dc4629efd82491ee04368fb636d17b7d8c34bc45a74f2c47b419ab4a" Nov 03 02:15:03.383903 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:15:03.383883 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894535-nj7sx_odh-ai-gateway-infra(b4a0e603-b309-4a5b-861b-f1780b71f6e0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894535-nj7sx" podUID="b4a0e603-b309-4a5b-861b-f1780b71f6e0" Nov 03 02:15:03.385760 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:03.385737 5605 generic.go:356] "Generic (PLEG): container finished" podID="7d38408a-0e73-4d42-b78c-6ce430d3927c" containerID="9385734ecf070908fcafaf3f40796be1c0114aee52a135e29bb39735865d0a69" exitCode=60 Nov 03 02:15:03.385843 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:03.385791 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894535-gplpr" event={"ID":"7d38408a-0e73-4d42-b78c-6ce430d3927c","Type":"ContainerDied","Data":"9385734ecf070908fcafaf3f40796be1c0114aee52a135e29bb39735865d0a69"} Nov 03 02:15:03.386152 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:03.386139 5605 scope.go:122] "RemoveContainer" containerID="9385734ecf070908fcafaf3f40796be1c0114aee52a135e29bb39735865d0a69" Nov 03 02:15:03.386394 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:15:03.386344 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-521f1f-29894535-gplpr_odh-ai-gateway-infra(7d38408a-0e73-4d42-b78c-6ce430d3927c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894535-gplpr" podUID="7d38408a-0e73-4d42-b78c-6ce430d3927c" Nov 03 02:15:03.558602 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:03.558571 5605 scope.go:122] "RemoveContainer" containerID="1e0453e17bc42ab1ce8e52688fdd2d750e67a2f820edeaab216c3768f71a96ae" Nov 03 02:15:03.571925 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:03.571902 5605 scope.go:122] "RemoveContainer" containerID="686886b4dbe254bd7cb737d9e7f6335658c1952cd34850a3dc40160f9a026e8f" Nov 03 02:15:03.587613 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:03.587588 5605 scope.go:122] "RemoveContainer" containerID="5bd87b69b4f0e03bca70123f2fe63305bac93000f5f31f2a490f1ea60e61d2fd" Nov 03 02:15:03.600718 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:03.600697 5605 scope.go:122] "RemoveContainer" containerID="73d630e56445661f467bb3e27e87cd3d7dc527ed3bdef6ee720a18b70767e148" Nov 03 02:15:04.294865 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:04.294828 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:15:04.393297 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:04.393262 5605 scope.go:122] "RemoveContainer" containerID="9385734ecf070908fcafaf3f40796be1c0114aee52a135e29bb39735865d0a69" Nov 03 02:15:04.393745 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:15:04.393595 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-521f1f-29894535-gplpr_odh-ai-gateway-infra(7d38408a-0e73-4d42-b78c-6ce430d3927c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894535-gplpr" podUID="7d38408a-0e73-4d42-b78c-6ce430d3927c" Nov 03 02:15:04.394809 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:04.394790 5605 scope.go:122] "RemoveContainer" containerID="7782212b623193f38c78cb995d7b70d0637bce523299a46dc7f5011aab6cd77f" Nov 03 02:15:04.395099 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:15:04.395079 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-7f332d-29894535-mn8qq_odh-ai-gateway-infra(988f453a-1c54-4410-85a1-aff8a5e66af4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7f332d-29894535-mn8qq" podUID="988f453a-1c54-4410-85a1-aff8a5e66af4" Nov 03 02:15:04.396165 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:04.396147 5605 scope.go:122] "RemoveContainer" containerID="8ee392b8f0224e58ff06fa8cda676918d86db283711c6e82b50538a9d6e53791" Nov 03 02:15:04.396469 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:15:04.396446 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-a772c3-29894535-72559_odh-ai-gateway-infra(9d32fcb0-b25a-4a1c-a480-b32cc423d1bb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a772c3-29894535-72559" podUID="9d32fcb0-b25a-4a1c-a480-b32cc423d1bb" Nov 03 02:15:04.397942 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:04.397922 5605 scope.go:122] "RemoveContainer" containerID="0c6b4d10930b7f5d123e3a3dcd55446722816a032f10bbfdbbf7cd0a5d5987e7" Nov 03 02:15:04.398189 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:15:04.398167 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-c7d758-29894535-4sw2s_odh-ai-gateway-infra(ec4f26be-fd2d-4f37-8200-eee788394037)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894535-4sw2s" podUID="ec4f26be-fd2d-4f37-8200-eee788394037" Nov 03 02:15:04.399422 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:04.399404 5605 scope.go:122] "RemoveContainer" containerID="285dd362dc4629efd82491ee04368fb636d17b7d8c34bc45a74f2c47b419ab4a" Nov 03 02:15:04.399600 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:15:04.399584 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894535-nj7sx_odh-ai-gateway-infra(b4a0e603-b309-4a5b-861b-f1780b71f6e0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894535-nj7sx" podUID="b4a0e603-b309-4a5b-861b-f1780b71f6e0" Nov 03 02:15:11.188885 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:11.188783 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d"] Nov 03 02:15:11.194297 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:11.194273 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d" Nov 03 02:15:11.330599 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:11.328907 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2225e2b8-fca7-4a8a-8c47-a687a0d9b214-workload-certs\") pod \"e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d\" (UID: \"2225e2b8-fca7-4a8a-8c47-a687a0d9b214\") " pod="openshift-ingress/e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d" Nov 03 02:15:11.330599 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:11.328959 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2225e2b8-fca7-4a8a-8c47-a687a0d9b214-istio-podinfo\") pod \"e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d\" (UID: \"2225e2b8-fca7-4a8a-8c47-a687a0d9b214\") " pod="openshift-ingress/e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d" Nov 03 02:15:11.330599 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:11.328996 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2225e2b8-fca7-4a8a-8c47-a687a0d9b214-istio-data\") pod \"e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d\" (UID: \"2225e2b8-fca7-4a8a-8c47-a687a0d9b214\") " pod="openshift-ingress/e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d" Nov 03 02:15:11.330599 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:11.329027 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2225e2b8-fca7-4a8a-8c47-a687a0d9b214-istiod-ca-cert\") pod \"e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d\" (UID: \"2225e2b8-fca7-4a8a-8c47-a687a0d9b214\") " pod="openshift-ingress/e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d" Nov 03 02:15:11.330599 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:11.329063 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2225e2b8-fca7-4a8a-8c47-a687a0d9b214-credential-socket\") pod \"e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d\" (UID: \"2225e2b8-fca7-4a8a-8c47-a687a0d9b214\") " pod="openshift-ingress/e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d" Nov 03 02:15:11.330599 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:11.329099 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2225e2b8-fca7-4a8a-8c47-a687a0d9b214-istio-envoy\") pod \"e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d\" (UID: \"2225e2b8-fca7-4a8a-8c47-a687a0d9b214\") " pod="openshift-ingress/e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d" Nov 03 02:15:11.330599 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:11.329219 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2225e2b8-fca7-4a8a-8c47-a687a0d9b214-workload-socket\") pod \"e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d\" (UID: \"2225e2b8-fca7-4a8a-8c47-a687a0d9b214\") " pod="openshift-ingress/e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d" Nov 03 02:15:11.330599 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:11.329245 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2225e2b8-fca7-4a8a-8c47-a687a0d9b214-istio-token\") pod \"e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d\" (UID: \"2225e2b8-fca7-4a8a-8c47-a687a0d9b214\") " pod="openshift-ingress/e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d" Nov 03 02:15:11.330599 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:11.329271 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc2k7\" (UniqueName: \"kubernetes.io/projected/2225e2b8-fca7-4a8a-8c47-a687a0d9b214-kube-api-access-sc2k7\") pod \"e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d\" (UID: \"2225e2b8-fca7-4a8a-8c47-a687a0d9b214\") " pod="openshift-ingress/e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d" Nov 03 02:15:11.330599 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:11.330010 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-aa82d2-openshift-default-dockercfg-wh4dg\"" Nov 03 02:15:11.351102 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:11.341650 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d"] Nov 03 02:15:11.436380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:11.430669 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2225e2b8-fca7-4a8a-8c47-a687a0d9b214-workload-socket\") pod \"e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d\" (UID: \"2225e2b8-fca7-4a8a-8c47-a687a0d9b214\") " pod="openshift-ingress/e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d" Nov 03 02:15:11.436380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:11.430716 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2225e2b8-fca7-4a8a-8c47-a687a0d9b214-istio-token\") pod \"e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d\" (UID: \"2225e2b8-fca7-4a8a-8c47-a687a0d9b214\") " pod="openshift-ingress/e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d" Nov 03 02:15:11.436380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:11.430744 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc2k7\" (UniqueName: \"kubernetes.io/projected/2225e2b8-fca7-4a8a-8c47-a687a0d9b214-kube-api-access-sc2k7\") pod \"e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d\" (UID: \"2225e2b8-fca7-4a8a-8c47-a687a0d9b214\") " pod="openshift-ingress/e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d" Nov 03 02:15:11.436380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:11.430873 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2225e2b8-fca7-4a8a-8c47-a687a0d9b214-workload-certs\") pod \"e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d\" (UID: \"2225e2b8-fca7-4a8a-8c47-a687a0d9b214\") " pod="openshift-ingress/e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d" Nov 03 02:15:11.436380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:11.430916 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2225e2b8-fca7-4a8a-8c47-a687a0d9b214-istio-podinfo\") pod \"e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d\" (UID: \"2225e2b8-fca7-4a8a-8c47-a687a0d9b214\") " pod="openshift-ingress/e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d" Nov 03 02:15:11.436380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:11.430954 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2225e2b8-fca7-4a8a-8c47-a687a0d9b214-istio-data\") pod \"e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d\" (UID: \"2225e2b8-fca7-4a8a-8c47-a687a0d9b214\") " pod="openshift-ingress/e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d" Nov 03 02:15:11.436380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:11.430988 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2225e2b8-fca7-4a8a-8c47-a687a0d9b214-istiod-ca-cert\") pod \"e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d\" (UID: \"2225e2b8-fca7-4a8a-8c47-a687a0d9b214\") " pod="openshift-ingress/e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d" Nov 03 02:15:11.436380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:11.431029 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2225e2b8-fca7-4a8a-8c47-a687a0d9b214-credential-socket\") pod \"e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d\" (UID: \"2225e2b8-fca7-4a8a-8c47-a687a0d9b214\") " pod="openshift-ingress/e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d" Nov 03 02:15:11.436380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:11.431071 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2225e2b8-fca7-4a8a-8c47-a687a0d9b214-istio-envoy\") pod \"e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d\" (UID: \"2225e2b8-fca7-4a8a-8c47-a687a0d9b214\") " pod="openshift-ingress/e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d" Nov 03 02:15:11.436380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:11.431886 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2225e2b8-fca7-4a8a-8c47-a687a0d9b214-workload-certs\") pod \"e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d\" (UID: \"2225e2b8-fca7-4a8a-8c47-a687a0d9b214\") " pod="openshift-ingress/e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d" Nov 03 02:15:11.436380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:11.432215 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2225e2b8-fca7-4a8a-8c47-a687a0d9b214-workload-socket\") pod \"e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d\" (UID: \"2225e2b8-fca7-4a8a-8c47-a687a0d9b214\") " pod="openshift-ingress/e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d" Nov 03 02:15:11.436380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:11.432999 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2225e2b8-fca7-4a8a-8c47-a687a0d9b214-istio-data\") pod \"e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d\" (UID: \"2225e2b8-fca7-4a8a-8c47-a687a0d9b214\") " pod="openshift-ingress/e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d" Nov 03 02:15:11.441411 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:11.441309 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2225e2b8-fca7-4a8a-8c47-a687a0d9b214-istiod-ca-cert\") pod \"e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d\" (UID: \"2225e2b8-fca7-4a8a-8c47-a687a0d9b214\") " pod="openshift-ingress/e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d" Nov 03 02:15:11.441978 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:11.441879 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2225e2b8-fca7-4a8a-8c47-a687a0d9b214-credential-socket\") pod \"e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d\" (UID: \"2225e2b8-fca7-4a8a-8c47-a687a0d9b214\") " pod="openshift-ingress/e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d" Nov 03 02:15:11.498947 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:11.497981 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2225e2b8-fca7-4a8a-8c47-a687a0d9b214-istio-envoy\") pod \"e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d\" (UID: \"2225e2b8-fca7-4a8a-8c47-a687a0d9b214\") " pod="openshift-ingress/e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d" Nov 03 02:15:11.502599 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:11.501490 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2225e2b8-fca7-4a8a-8c47-a687a0d9b214-istio-podinfo\") pod \"e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d\" (UID: \"2225e2b8-fca7-4a8a-8c47-a687a0d9b214\") " pod="openshift-ingress/e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d" Nov 03 02:15:11.510509 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:11.510326 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2225e2b8-fca7-4a8a-8c47-a687a0d9b214-istio-token\") pod \"e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d\" (UID: \"2225e2b8-fca7-4a8a-8c47-a687a0d9b214\") " pod="openshift-ingress/e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d" Nov 03 02:15:11.512738 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:11.511998 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc2k7\" (UniqueName: \"kubernetes.io/projected/2225e2b8-fca7-4a8a-8c47-a687a0d9b214-kube-api-access-sc2k7\") pod \"e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d\" (UID: \"2225e2b8-fca7-4a8a-8c47-a687a0d9b214\") " pod="openshift-ingress/e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d" Nov 03 02:15:11.572056 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:11.549659 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d" Nov 03 02:15:12.160744 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:12.160706 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d"] Nov 03 02:15:12.255922 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:12.255879 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:15:12.256476 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:12.256439 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:15:12.256599 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:12.256503 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:15:12.501343 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:12.501249 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d" event={"ID":"2225e2b8-fca7-4a8a-8c47-a687a0d9b214","Type":"ContainerStarted","Data":"8938b6aa86b546ebb49211ec1e640193f4f45bf3962d90615893c4f209f900aa"} Nov 03 02:15:12.501343 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:12.501319 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d" event={"ID":"2225e2b8-fca7-4a8a-8c47-a687a0d9b214","Type":"ContainerStarted","Data":"b66a681f5848a92322af904862fdbd1d4cb98049bb4a72c35a5b1ccde2dd43d9"} Nov 03 02:15:12.551151 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:12.551122 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d" Nov 03 02:15:12.612211 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:12.612154 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d" podStartSLOduration=1.6121392079999999 podStartE2EDuration="1.612139208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:15:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:15:12.610196012 +0000 UTC m=+1389.869394166" watchObservedRunningTime="2026-11-03 02:15:12.612139208 +0000 UTC m=+1389.871337399" Nov 03 02:15:13.557048 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:13.557017 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d" Nov 03 02:15:14.050924 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:14.050894 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2df4dda6-mds7w"] Nov 03 02:15:14.054704 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:14.054680 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2df4dda6-mds7w"] Nov 03 02:15:14.514436 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:14.514343 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d" Nov 03 02:15:14.515427 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:14.515408 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d" Nov 03 02:15:14.851502 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:14.851466 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abd0f382-d3ad-4fbc-93da-69b7c9c98715" path="/var/lib/kubelet/pods/abd0f382-d3ad-4fbc-93da-69b7c9c98715/volumes" Nov 03 02:15:15.844006 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:15.843977 5605 scope.go:122] "RemoveContainer" containerID="9385734ecf070908fcafaf3f40796be1c0114aee52a135e29bb39735865d0a69" Nov 03 02:15:15.844152 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:15.844045 5605 scope.go:122] "RemoveContainer" containerID="285dd362dc4629efd82491ee04368fb636d17b7d8c34bc45a74f2c47b419ab4a" Nov 03 02:15:16.528495 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:16.528405 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894535-nj7sx" event={"ID":"b4a0e603-b309-4a5b-861b-f1780b71f6e0","Type":"ContainerStarted","Data":"1c3890d7f96d0b436da436f24b4b3c6c9a81c52a05a3792f22447474a331c5ca"} Nov 03 02:15:16.530411 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:16.530384 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894535-gplpr" event={"ID":"7d38408a-0e73-4d42-b78c-6ce430d3927c","Type":"ContainerStarted","Data":"827bbc02cc27b7315bd077d7cfdb3d45d95675eec260eece07e46db537a951ab"} Nov 03 02:15:16.739785 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:16.739719 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894535-gplpr" podStartSLOduration=16.739700637 podStartE2EDuration="16.739700637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:15:16.739290879 +0000 UTC m=+1393.998489033" watchObservedRunningTime="2026-11-03 02:15:16.739700637 +0000 UTC m=+1393.998898792" Nov 03 02:15:16.843597 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:16.843567 5605 scope.go:122] "RemoveContainer" containerID="7782212b623193f38c78cb995d7b70d0637bce523299a46dc7f5011aab6cd77f" Nov 03 02:15:17.539480 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:17.539388 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7f332d-29894535-mn8qq" event={"ID":"988f453a-1c54-4410-85a1-aff8a5e66af4","Type":"ContainerStarted","Data":"ffaf5e2508e9b9174ac493b1ac6cfd2b9205db07ee5a699e5afbf445572f2cd2"} Nov 03 02:15:17.541349 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:17.541325 5605 generic.go:356] "Generic (PLEG): container finished" podID="b4a0e603-b309-4a5b-861b-f1780b71f6e0" containerID="1c3890d7f96d0b436da436f24b4b3c6c9a81c52a05a3792f22447474a331c5ca" exitCode=60 Nov 03 02:15:17.541478 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:17.541385 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894535-nj7sx" event={"ID":"b4a0e603-b309-4a5b-861b-f1780b71f6e0","Type":"ContainerDied","Data":"1c3890d7f96d0b436da436f24b4b3c6c9a81c52a05a3792f22447474a331c5ca"} Nov 03 02:15:17.541478 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:17.541436 5605 scope.go:122] "RemoveContainer" containerID="285dd362dc4629efd82491ee04368fb636d17b7d8c34bc45a74f2c47b419ab4a" Nov 03 02:15:17.541745 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:17.541727 5605 scope.go:122] "RemoveContainer" containerID="1c3890d7f96d0b436da436f24b4b3c6c9a81c52a05a3792f22447474a331c5ca" Nov 03 02:15:17.542049 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:15:17.542025 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894535-nj7sx_odh-ai-gateway-infra(b4a0e603-b309-4a5b-861b-f1780b71f6e0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894535-nj7sx" podUID="b4a0e603-b309-4a5b-861b-f1780b71f6e0" Nov 03 02:15:17.543479 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:17.543460 5605 generic.go:356] "Generic (PLEG): container finished" podID="7d38408a-0e73-4d42-b78c-6ce430d3927c" containerID="827bbc02cc27b7315bd077d7cfdb3d45d95675eec260eece07e46db537a951ab" exitCode=60 Nov 03 02:15:17.543570 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:17.543485 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894535-gplpr" event={"ID":"7d38408a-0e73-4d42-b78c-6ce430d3927c","Type":"ContainerDied","Data":"827bbc02cc27b7315bd077d7cfdb3d45d95675eec260eece07e46db537a951ab"} Nov 03 02:15:17.543759 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:17.543746 5605 scope.go:122] "RemoveContainer" containerID="827bbc02cc27b7315bd077d7cfdb3d45d95675eec260eece07e46db537a951ab" Nov 03 02:15:17.543986 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:15:17.543969 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-521f1f-29894535-gplpr_odh-ai-gateway-infra(7d38408a-0e73-4d42-b78c-6ce430d3927c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894535-gplpr" podUID="7d38408a-0e73-4d42-b78c-6ce430d3927c" Nov 03 02:15:17.553421 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:17.553376 5605 scope.go:122] "RemoveContainer" containerID="9385734ecf070908fcafaf3f40796be1c0114aee52a135e29bb39735865d0a69" Nov 03 02:15:17.644133 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:17.644099 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894535-nj7sx"] Nov 03 02:15:17.646838 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:17.646787 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7f332d-29894535-mn8qq" podStartSLOduration=17.646772772 podStartE2EDuration="17.646772772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:15:17.644981336 +0000 UTC m=+1394.904179490" watchObservedRunningTime="2026-11-03 02:15:17.646772772 +0000 UTC m=+1394.905970925" Nov 03 02:15:17.749766 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:17.749734 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894535-gplpr"] Nov 03 02:15:17.843588 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:17.843565 5605 scope.go:122] "RemoveContainer" containerID="8ee392b8f0224e58ff06fa8cda676918d86db283711c6e82b50538a9d6e53791" Nov 03 02:15:17.843732 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:17.843651 5605 scope.go:122] "RemoveContainer" containerID="0c6b4d10930b7f5d123e3a3dcd55446722816a032f10bbfdbbf7cd0a5d5987e7" Nov 03 02:15:18.553407 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:18.553148 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894535-4sw2s" event={"ID":"ec4f26be-fd2d-4f37-8200-eee788394037","Type":"ContainerStarted","Data":"4d007e31b73df38fa6963c102ecd2a35e3f3760345b07572f152a0fa8c5d959b"} Nov 03 02:15:18.559943 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:18.559909 5605 generic.go:356] "Generic (PLEG): container finished" podID="988f453a-1c54-4410-85a1-aff8a5e66af4" containerID="ffaf5e2508e9b9174ac493b1ac6cfd2b9205db07ee5a699e5afbf445572f2cd2" exitCode=60 Nov 03 02:15:18.560067 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:18.559969 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7f332d-29894535-mn8qq" event={"ID":"988f453a-1c54-4410-85a1-aff8a5e66af4","Type":"ContainerDied","Data":"ffaf5e2508e9b9174ac493b1ac6cfd2b9205db07ee5a699e5afbf445572f2cd2"} Nov 03 02:15:18.560067 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:18.560032 5605 scope.go:122] "RemoveContainer" containerID="7782212b623193f38c78cb995d7b70d0637bce523299a46dc7f5011aab6cd77f" Nov 03 02:15:18.562384 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:18.560460 5605 scope.go:122] "RemoveContainer" containerID="ffaf5e2508e9b9174ac493b1ac6cfd2b9205db07ee5a699e5afbf445572f2cd2" Nov 03 02:15:18.562384 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:15:18.560763 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-7f332d-29894535-mn8qq_odh-ai-gateway-infra(988f453a-1c54-4410-85a1-aff8a5e66af4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7f332d-29894535-mn8qq" podUID="988f453a-1c54-4410-85a1-aff8a5e66af4" Nov 03 02:15:18.564101 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:18.563351 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a772c3-29894535-72559" event={"ID":"9d32fcb0-b25a-4a1c-a480-b32cc423d1bb","Type":"ContainerStarted","Data":"e523e1cf76d7a8e14f21c5ad2ec8ae448f7c20fd1610b9522056a88730f3f3e3"} Nov 03 02:15:18.655809 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:18.655588 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7f332d-29894535-mn8qq"] Nov 03 02:15:18.667393 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:18.667314 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894535-4sw2s" podStartSLOduration=18.667299086 podStartE2EDuration="18.667299086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:15:18.663198393 +0000 UTC m=+1395.922396548" watchObservedRunningTime="2026-11-03 02:15:18.667299086 +0000 UTC m=+1395.926497240" Nov 03 02:15:18.769822 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:18.769793 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894535-nj7sx" Nov 03 02:15:18.801135 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:18.801099 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894535-gplpr" Nov 03 02:15:18.853165 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:18.853074 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d38408a-0e73-4d42-b78c-6ce430d3927c-kube-api-access-5vzzg\" (UniqueName: \"kubernetes.io/projected/7d38408a-0e73-4d42-b78c-6ce430d3927c-kube-api-access-5vzzg\") pod \"7d38408a-0e73-4d42-b78c-6ce430d3927c\" (UID: \"7d38408a-0e73-4d42-b78c-6ce430d3927c\") " Nov 03 02:15:18.853165 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:18.853136 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4a0e603-b309-4a5b-861b-f1780b71f6e0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4a0e603-b309-4a5b-861b-f1780b71f6e0-maas-api-service-ca\") pod \"b4a0e603-b309-4a5b-861b-f1780b71f6e0\" (UID: \"b4a0e603-b309-4a5b-861b-f1780b71f6e0\") " Nov 03 02:15:18.853405 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:18.853186 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d38408a-0e73-4d42-b78c-6ce430d3927c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d38408a-0e73-4d42-b78c-6ce430d3927c-maas-api-service-ca\") pod \"7d38408a-0e73-4d42-b78c-6ce430d3927c\" (UID: \"7d38408a-0e73-4d42-b78c-6ce430d3927c\") " Nov 03 02:15:18.853405 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:18.853322 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4a0e603-b309-4a5b-861b-f1780b71f6e0-kube-api-access-jm5qn\" (UniqueName: \"kubernetes.io/projected/b4a0e603-b309-4a5b-861b-f1780b71f6e0-kube-api-access-jm5qn\") pod \"b4a0e603-b309-4a5b-861b-f1780b71f6e0\" (UID: \"b4a0e603-b309-4a5b-861b-f1780b71f6e0\") " Nov 03 02:15:18.853866 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:18.853836 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4a0e603-b309-4a5b-861b-f1780b71f6e0-maas-api-service-ca" pod "b4a0e603-b309-4a5b-861b-f1780b71f6e0" (UID: "b4a0e603-b309-4a5b-861b-f1780b71f6e0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:15:18.854131 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:18.854112 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4a0e603-b309-4a5b-861b-f1780b71f6e0-maas-api-service-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:15:18.854426 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:18.854406 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d38408a-0e73-4d42-b78c-6ce430d3927c-maas-api-service-ca" pod "7d38408a-0e73-4d42-b78c-6ce430d3927c" (UID: "7d38408a-0e73-4d42-b78c-6ce430d3927c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:15:18.859865 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:18.859835 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d38408a-0e73-4d42-b78c-6ce430d3927c-kube-api-access-5vzzg" pod "7d38408a-0e73-4d42-b78c-6ce430d3927c" (UID: "7d38408a-0e73-4d42-b78c-6ce430d3927c"). InnerVolumeSpecName "kube-api-access-5vzzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:15:18.860151 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:18.860131 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4a0e603-b309-4a5b-861b-f1780b71f6e0-kube-api-access-jm5qn" pod "b4a0e603-b309-4a5b-861b-f1780b71f6e0" (UID: "b4a0e603-b309-4a5b-861b-f1780b71f6e0"). InnerVolumeSpecName "kube-api-access-jm5qn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:15:18.955025 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:18.954989 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jm5qn\" (UniqueName: \"kubernetes.io/projected/b4a0e603-b309-4a5b-861b-f1780b71f6e0-kube-api-access-jm5qn\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:15:18.955025 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:18.955024 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vzzg\" (UniqueName: \"kubernetes.io/projected/7d38408a-0e73-4d42-b78c-6ce430d3927c-kube-api-access-5vzzg\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:15:18.955226 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:18.955041 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d38408a-0e73-4d42-b78c-6ce430d3927c-maas-api-service-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:15:18.979141 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:15:18.979112 5605 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec4f26be_fd2d_4f37_8200_eee788394037.slice/crio-4d007e31b73df38fa6963c102ecd2a35e3f3760345b07572f152a0fa8c5d959b.scope\": RecentStats: unable to find data in memory cache]" Nov 03 02:15:19.688396 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:19.685260 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894535-4sw2s"] Nov 03 02:15:19.695402 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:19.690642 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894535-gplpr" event={"ID":"7d38408a-0e73-4d42-b78c-6ce430d3927c","Type":"ContainerDied","Data":"b2ee9b61a1ef716cc4d5bf23eb12ca686d30363e847883c1d135a1f78ee8b4e8"} Nov 03 02:15:19.695402 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:19.690712 5605 scope.go:122] "RemoveContainer" containerID="827bbc02cc27b7315bd077d7cfdb3d45d95675eec260eece07e46db537a951ab" Nov 03 02:15:19.695402 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:19.690870 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894535-gplpr" Nov 03 02:15:19.702671 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:19.702586 5605 generic.go:356] "Generic (PLEG): container finished" podID="9d32fcb0-b25a-4a1c-a480-b32cc423d1bb" containerID="e523e1cf76d7a8e14f21c5ad2ec8ae448f7c20fd1610b9522056a88730f3f3e3" exitCode=60 Nov 03 02:15:19.702983 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:19.702641 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a772c3-29894535-72559" event={"ID":"9d32fcb0-b25a-4a1c-a480-b32cc423d1bb","Type":"ContainerDied","Data":"e523e1cf76d7a8e14f21c5ad2ec8ae448f7c20fd1610b9522056a88730f3f3e3"} Nov 03 02:15:19.703659 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:19.703640 5605 scope.go:122] "RemoveContainer" containerID="e523e1cf76d7a8e14f21c5ad2ec8ae448f7c20fd1610b9522056a88730f3f3e3" Nov 03 02:15:19.704171 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:15:19.704146 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-a772c3-29894535-72559_odh-ai-gateway-infra(9d32fcb0-b25a-4a1c-a480-b32cc423d1bb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a772c3-29894535-72559" podUID="9d32fcb0-b25a-4a1c-a480-b32cc423d1bb" Nov 03 02:15:19.709585 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:19.709493 5605 scope.go:122] "RemoveContainer" containerID="8ee392b8f0224e58ff06fa8cda676918d86db283711c6e82b50538a9d6e53791" Nov 03 02:15:19.709684 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:19.709656 5605 generic.go:356] "Generic (PLEG): container finished" podID="ec4f26be-fd2d-4f37-8200-eee788394037" containerID="4d007e31b73df38fa6963c102ecd2a35e3f3760345b07572f152a0fa8c5d959b" exitCode=60 Nov 03 02:15:19.709875 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:19.709834 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894535-4sw2s" event={"ID":"ec4f26be-fd2d-4f37-8200-eee788394037","Type":"ContainerDied","Data":"4d007e31b73df38fa6963c102ecd2a35e3f3760345b07572f152a0fa8c5d959b"} Nov 03 02:15:19.711648 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:19.711622 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894535-nj7sx" event={"ID":"b4a0e603-b309-4a5b-861b-f1780b71f6e0","Type":"ContainerDied","Data":"6aa8d91438b399d71734e9d627ee174072c4933e0b4cf233899a180977e8b78c"} Nov 03 02:15:19.711750 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:19.711735 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894535-nj7sx" Nov 03 02:15:19.726171 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:19.726147 5605 scope.go:122] "RemoveContainer" containerID="0c6b4d10930b7f5d123e3a3dcd55446722816a032f10bbfdbbf7cd0a5d5987e7" Nov 03 02:15:19.746432 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:19.746295 5605 scope.go:122] "RemoveContainer" containerID="1c3890d7f96d0b436da436f24b4b3c6c9a81c52a05a3792f22447474a331c5ca" Nov 03 02:15:19.859742 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:19.859678 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894535-gplpr"] Nov 03 02:15:19.867088 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:19.867057 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894535-gplpr"] Nov 03 02:15:19.887388 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:19.887342 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7f332d-29894535-mn8qq" Nov 03 02:15:19.902981 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:19.902949 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a772c3-29894535-72559"] Nov 03 02:15:19.927021 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:19.926993 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894535-4sw2s" Nov 03 02:15:20.063497 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:20.063465 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/988f453a-1c54-4410-85a1-aff8a5e66af4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/988f453a-1c54-4410-85a1-aff8a5e66af4-maas-api-service-ca\") pod \"988f453a-1c54-4410-85a1-aff8a5e66af4\" (UID: \"988f453a-1c54-4410-85a1-aff8a5e66af4\") " Nov 03 02:15:20.063666 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:20.063545 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec4f26be-fd2d-4f37-8200-eee788394037-kube-api-access-4thgz\" (UniqueName: \"kubernetes.io/projected/ec4f26be-fd2d-4f37-8200-eee788394037-kube-api-access-4thgz\") pod \"ec4f26be-fd2d-4f37-8200-eee788394037\" (UID: \"ec4f26be-fd2d-4f37-8200-eee788394037\") " Nov 03 02:15:20.063666 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:20.063651 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/988f453a-1c54-4410-85a1-aff8a5e66af4-kube-api-access-x67kc\" (UniqueName: \"kubernetes.io/projected/988f453a-1c54-4410-85a1-aff8a5e66af4-kube-api-access-x67kc\") pod \"988f453a-1c54-4410-85a1-aff8a5e66af4\" (UID: \"988f453a-1c54-4410-85a1-aff8a5e66af4\") " Nov 03 02:15:20.063756 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:20.063747 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec4f26be-fd2d-4f37-8200-eee788394037-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec4f26be-fd2d-4f37-8200-eee788394037-maas-api-service-ca\") pod \"ec4f26be-fd2d-4f37-8200-eee788394037\" (UID: \"ec4f26be-fd2d-4f37-8200-eee788394037\") " Nov 03 02:15:20.063896 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:20.063867 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/988f453a-1c54-4410-85a1-aff8a5e66af4-maas-api-service-ca" pod "988f453a-1c54-4410-85a1-aff8a5e66af4" (UID: "988f453a-1c54-4410-85a1-aff8a5e66af4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:15:20.064093 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:20.064073 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec4f26be-fd2d-4f37-8200-eee788394037-maas-api-service-ca" pod "ec4f26be-fd2d-4f37-8200-eee788394037" (UID: "ec4f26be-fd2d-4f37-8200-eee788394037"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:15:20.064257 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:20.064240 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec4f26be-fd2d-4f37-8200-eee788394037-maas-api-service-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:15:20.064392 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:20.064263 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/988f453a-1c54-4410-85a1-aff8a5e66af4-maas-api-service-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:15:20.065887 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:20.065864 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/988f453a-1c54-4410-85a1-aff8a5e66af4-kube-api-access-x67kc" pod "988f453a-1c54-4410-85a1-aff8a5e66af4" (UID: "988f453a-1c54-4410-85a1-aff8a5e66af4"). InnerVolumeSpecName "kube-api-access-x67kc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:15:20.066015 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:20.065915 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec4f26be-fd2d-4f37-8200-eee788394037-kube-api-access-4thgz" pod "ec4f26be-fd2d-4f37-8200-eee788394037" (UID: "ec4f26be-fd2d-4f37-8200-eee788394037"). InnerVolumeSpecName "kube-api-access-4thgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:15:20.069164 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:20.069140 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894535-nj7sx"] Nov 03 02:15:20.072319 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:20.072294 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894535-nj7sx"] Nov 03 02:15:20.165631 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:20.165598 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4thgz\" (UniqueName: \"kubernetes.io/projected/ec4f26be-fd2d-4f37-8200-eee788394037-kube-api-access-4thgz\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:15:20.165631 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:20.165624 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x67kc\" (UniqueName: \"kubernetes.io/projected/988f453a-1c54-4410-85a1-aff8a5e66af4-kube-api-access-x67kc\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:15:20.719854 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:20.719822 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7f332d-29894535-mn8qq" Nov 03 02:15:20.720344 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:20.719822 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7f332d-29894535-mn8qq" event={"ID":"988f453a-1c54-4410-85a1-aff8a5e66af4","Type":"ContainerDied","Data":"c7d4fed4362f9ce0730ffabe5f5193ead6d4df19a1b6edb58c451aa331e379c2"} Nov 03 02:15:20.720344 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:20.719956 5605 scope.go:122] "RemoveContainer" containerID="ffaf5e2508e9b9174ac493b1ac6cfd2b9205db07ee5a699e5afbf445572f2cd2" Nov 03 02:15:20.723020 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:20.722995 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894535-4sw2s" event={"ID":"ec4f26be-fd2d-4f37-8200-eee788394037","Type":"ContainerDied","Data":"a769008b15cc2b0e7fefde668ab70135e15edfe1ea98a4d7ea3e78cdcb9abba2"} Nov 03 02:15:20.723119 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:20.723085 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894535-4sw2s" Nov 03 02:15:20.740612 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:20.740587 5605 scope.go:122] "RemoveContainer" containerID="4d007e31b73df38fa6963c102ecd2a35e3f3760345b07572f152a0fa8c5d959b" Nov 03 02:15:20.851335 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:20.851300 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d38408a-0e73-4d42-b78c-6ce430d3927c" path="/var/lib/kubelet/pods/7d38408a-0e73-4d42-b78c-6ce430d3927c/volumes" Nov 03 02:15:20.852394 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:20.851944 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4a0e603-b309-4a5b-861b-f1780b71f6e0" path="/var/lib/kubelet/pods/b4a0e603-b309-4a5b-861b-f1780b71f6e0/volumes" Nov 03 02:15:20.893238 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:20.893207 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a772c3-29894535-72559" Nov 03 02:15:21.005131 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:20.994995 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d32fcb0-b25a-4a1c-a480-b32cc423d1bb-kube-api-access-kcgh2\" (UniqueName: \"kubernetes.io/projected/9d32fcb0-b25a-4a1c-a480-b32cc423d1bb-kube-api-access-kcgh2\") pod \"9d32fcb0-b25a-4a1c-a480-b32cc423d1bb\" (UID: \"9d32fcb0-b25a-4a1c-a480-b32cc423d1bb\") " Nov 03 02:15:21.005131 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:20.995180 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d32fcb0-b25a-4a1c-a480-b32cc423d1bb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d32fcb0-b25a-4a1c-a480-b32cc423d1bb-maas-api-service-ca\") pod \"9d32fcb0-b25a-4a1c-a480-b32cc423d1bb\" (UID: \"9d32fcb0-b25a-4a1c-a480-b32cc423d1bb\") " Nov 03 02:15:21.007639 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:21.007578 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d32fcb0-b25a-4a1c-a480-b32cc423d1bb-maas-api-service-ca" pod "9d32fcb0-b25a-4a1c-a480-b32cc423d1bb" (UID: "9d32fcb0-b25a-4a1c-a480-b32cc423d1bb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:15:21.013917 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:21.012453 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894535-4sw2s"] Nov 03 02:15:21.016798 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:21.016766 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894535-4sw2s"] Nov 03 02:15:21.035827 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:21.032898 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d32fcb0-b25a-4a1c-a480-b32cc423d1bb-kube-api-access-kcgh2" pod "9d32fcb0-b25a-4a1c-a480-b32cc423d1bb" (UID: "9d32fcb0-b25a-4a1c-a480-b32cc423d1bb"). InnerVolumeSpecName "kube-api-access-kcgh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:15:21.035827 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:21.033298 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d32fcb0-b25a-4a1c-a480-b32cc423d1bb-maas-api-service-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:15:21.100205 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:21.100147 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7f332d-29894535-mn8qq"] Nov 03 02:15:21.106726 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:21.106690 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7f332d-29894535-mn8qq"] Nov 03 02:15:21.134315 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:21.134274 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcgh2\" (UniqueName: \"kubernetes.io/projected/9d32fcb0-b25a-4a1c-a480-b32cc423d1bb-kube-api-access-kcgh2\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:15:21.735085 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:21.735055 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a772c3-29894535-72559" Nov 03 02:15:21.735649 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:21.735081 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a772c3-29894535-72559" event={"ID":"9d32fcb0-b25a-4a1c-a480-b32cc423d1bb","Type":"ContainerDied","Data":"6f0e64211273e2d26e974bac658ec49665bef8f5748321e37e0de5c42470c70c"} Nov 03 02:15:21.735649 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:21.735133 5605 scope.go:122] "RemoveContainer" containerID="e523e1cf76d7a8e14f21c5ad2ec8ae448f7c20fd1610b9522056a88730f3f3e3" Nov 03 02:15:21.889012 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:21.888973 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a772c3-29894535-72559"] Nov 03 02:15:21.896037 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:21.895997 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a772c3-29894535-72559"] Nov 03 02:15:22.850660 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:22.850629 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="988f453a-1c54-4410-85a1-aff8a5e66af4" path="/var/lib/kubelet/pods/988f453a-1c54-4410-85a1-aff8a5e66af4/volumes" Nov 03 02:15:22.851090 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:22.851076 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d32fcb0-b25a-4a1c-a480-b32cc423d1bb" path="/var/lib/kubelet/pods/9d32fcb0-b25a-4a1c-a480-b32cc423d1bb/volumes" Nov 03 02:15:22.851573 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:22.851556 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec4f26be-fd2d-4f37-8200-eee788394037" path="/var/lib/kubelet/pods/ec4f26be-fd2d-4f37-8200-eee788394037/volumes" Nov 03 02:15:23.914250 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:15:23.914212 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:15:26.058387 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:26.055845 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7241-42734b"] Nov 03 02:15:26.058387 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:26.056848 5605 cpu_manager.go:414] "containerMap: removing container" podUID="7d38408a-0e73-4d42-b78c-6ce430d3927c" containerName="cleanup" Nov 03 02:15:26.058387 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:26.056872 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d38408a-0e73-4d42-b78c-6ce430d3927c" containerName="cleanup" Nov 03 02:15:26.058387 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:26.056896 5605 cpu_manager.go:414] "containerMap: removing container" podUID="988f453a-1c54-4410-85a1-aff8a5e66af4" containerName="cleanup" Nov 03 02:15:26.058387 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:26.056904 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="988f453a-1c54-4410-85a1-aff8a5e66af4" containerName="cleanup" Nov 03 02:15:26.058387 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:26.056921 5605 cpu_manager.go:414] "containerMap: removing container" podUID="b4a0e603-b309-4a5b-861b-f1780b71f6e0" containerName="cleanup" Nov 03 02:15:26.058387 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:26.056928 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4a0e603-b309-4a5b-861b-f1780b71f6e0" containerName="cleanup" Nov 03 02:15:26.058387 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:26.056938 5605 cpu_manager.go:414] "containerMap: removing container" podUID="b4a0e603-b309-4a5b-861b-f1780b71f6e0" containerName="cleanup" Nov 03 02:15:26.058387 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:26.056945 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4a0e603-b309-4a5b-861b-f1780b71f6e0" containerName="cleanup" Nov 03 02:15:26.058387 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:26.056977 5605 cpu_manager.go:414] "containerMap: removing container" podUID="9d32fcb0-b25a-4a1c-a480-b32cc423d1bb" containerName="cleanup" Nov 03 02:15:26.058387 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:26.056985 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d32fcb0-b25a-4a1c-a480-b32cc423d1bb" containerName="cleanup" Nov 03 02:15:26.058387 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:26.057003 5605 cpu_manager.go:414] "containerMap: removing container" podUID="ec4f26be-fd2d-4f37-8200-eee788394037" containerName="cleanup" Nov 03 02:15:26.058387 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:26.057010 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec4f26be-fd2d-4f37-8200-eee788394037" containerName="cleanup" Nov 03 02:15:26.058387 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:26.057022 5605 cpu_manager.go:414] "containerMap: removing container" podUID="9d32fcb0-b25a-4a1c-a480-b32cc423d1bb" containerName="cleanup" Nov 03 02:15:26.058387 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:26.057029 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d32fcb0-b25a-4a1c-a480-b32cc423d1bb" containerName="cleanup" Nov 03 02:15:26.058387 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:26.057042 5605 cpu_manager.go:414] "containerMap: removing container" podUID="9d32fcb0-b25a-4a1c-a480-b32cc423d1bb" containerName="cleanup" Nov 03 02:15:26.058387 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:26.057049 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d32fcb0-b25a-4a1c-a480-b32cc423d1bb" containerName="cleanup" Nov 03 02:15:26.058387 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:26.057065 5605 cpu_manager.go:414] "containerMap: removing container" podUID="ec4f26be-fd2d-4f37-8200-eee788394037" containerName="cleanup" Nov 03 02:15:26.058387 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:26.057073 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec4f26be-fd2d-4f37-8200-eee788394037" containerName="cleanup" Nov 03 02:15:26.058387 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:26.057084 5605 cpu_manager.go:414] "containerMap: removing container" podUID="988f453a-1c54-4410-85a1-aff8a5e66af4" containerName="cleanup" Nov 03 02:15:26.058387 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:26.057089 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="988f453a-1c54-4410-85a1-aff8a5e66af4" containerName="cleanup" Nov 03 02:15:26.058387 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:26.057101 5605 cpu_manager.go:414] "containerMap: removing container" podUID="ec4f26be-fd2d-4f37-8200-eee788394037" containerName="cleanup" Nov 03 02:15:26.058387 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:26.057109 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec4f26be-fd2d-4f37-8200-eee788394037" containerName="cleanup" Nov 03 02:15:26.058387 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:26.057125 5605 cpu_manager.go:414] "containerMap: removing container" podUID="7d38408a-0e73-4d42-b78c-6ce430d3927c" containerName="cleanup" Nov 03 02:15:26.058387 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:26.057133 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d38408a-0e73-4d42-b78c-6ce430d3927c" containerName="cleanup" Nov 03 02:15:26.058387 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:26.057286 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4a0e603-b309-4a5b-861b-f1780b71f6e0" containerName="cleanup" Nov 03 02:15:26.058387 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:26.057304 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d32fcb0-b25a-4a1c-a480-b32cc423d1bb" containerName="cleanup" Nov 03 02:15:26.058387 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:26.057316 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d38408a-0e73-4d42-b78c-6ce430d3927c" containerName="cleanup" Nov 03 02:15:26.058387 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:26.057330 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec4f26be-fd2d-4f37-8200-eee788394037" containerName="cleanup" Nov 03 02:15:26.058387 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:26.057339 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d32fcb0-b25a-4a1c-a480-b32cc423d1bb" containerName="cleanup" Nov 03 02:15:26.058387 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:26.057347 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="988f453a-1c54-4410-85a1-aff8a5e66af4" containerName="cleanup" Nov 03 02:15:26.058387 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:26.057376 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="988f453a-1c54-4410-85a1-aff8a5e66af4" containerName="cleanup" Nov 03 02:15:26.058387 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:26.057387 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d32fcb0-b25a-4a1c-a480-b32cc423d1bb" containerName="cleanup" Nov 03 02:15:26.058387 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:26.057398 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="988f453a-1c54-4410-85a1-aff8a5e66af4" containerName="cleanup" Nov 03 02:15:26.058387 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:26.057407 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4a0e603-b309-4a5b-861b-f1780b71f6e0" containerName="cleanup" Nov 03 02:15:26.058387 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:26.057415 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec4f26be-fd2d-4f37-8200-eee788394037" containerName="cleanup" Nov 03 02:15:26.058387 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:26.057426 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d38408a-0e73-4d42-b78c-6ce430d3927c" containerName="cleanup" Nov 03 02:15:26.063331 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:26.063195 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7241-42734b" Nov 03 02:15:26.127379 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:26.126734 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7241-42734b"] Nov 03 02:15:26.147378 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:26.145803 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-579jc\"" Nov 03 02:15:26.214595 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:26.214557 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w66sn\" (UniqueName: \"kubernetes.io/projected/59299b05-48d7-4507-98e5-46c2681b576b-kube-api-access-w66sn\") pod \"test-curl-7241-42734b\" (UID: \"59299b05-48d7-4507-98e5-46c2681b576b\") " pod="opendatahub/test-curl-7241-42734b" Nov 03 02:15:26.315993 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:26.315897 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w66sn\" (UniqueName: \"kubernetes.io/projected/59299b05-48d7-4507-98e5-46c2681b576b-kube-api-access-w66sn\") pod \"test-curl-7241-42734b\" (UID: \"59299b05-48d7-4507-98e5-46c2681b576b\") " pod="opendatahub/test-curl-7241-42734b" Nov 03 02:15:26.369801 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:26.369765 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w66sn\" (UniqueName: \"kubernetes.io/projected/59299b05-48d7-4507-98e5-46c2681b576b-kube-api-access-w66sn\") pod \"test-curl-7241-42734b\" (UID: \"59299b05-48d7-4507-98e5-46c2681b576b\") " pod="opendatahub/test-curl-7241-42734b" Nov 03 02:15:26.465572 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:26.465536 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7241-42734b" Nov 03 02:15:26.836554 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:26.836519 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7241-42734b"] Nov 03 02:15:26.842333 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:15:26.842304 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod59299b05_48d7_4507_98e5_46c2681b576b.slice/crio-2e2da72ee84f8bdf5d9b7fe1ed17dd17dfeb15e7925b95f8b1ff8ff650e5102d WatchSource:0}: Error finding container 2e2da72ee84f8bdf5d9b7fe1ed17dd17dfeb15e7925b95f8b1ff8ff650e5102d: Status 404 returned error can't find the container with id 2e2da72ee84f8bdf5d9b7fe1ed17dd17dfeb15e7925b95f8b1ff8ff650e5102d Nov 03 02:15:26.845025 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:26.845001 5605 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 02:15:27.851386 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:27.841329 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7241-42734b" event={"ID":"59299b05-48d7-4507-98e5-46c2681b576b","Type":"ContainerStarted","Data":"2e2da72ee84f8bdf5d9b7fe1ed17dd17dfeb15e7925b95f8b1ff8ff650e5102d"} Nov 03 02:15:30.044462 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:30.044398 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29c33734-vqktr"] Nov 03 02:15:30.114380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:30.111907 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29c33734-vqktr"] Nov 03 02:15:30.851217 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:30.851187 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="555f7460-fcf6-4517-8181-157e5bd65ddf" path="/var/lib/kubelet/pods/555f7460-fcf6-4517-8181-157e5bd65ddf/volumes" Nov 03 02:15:30.886342 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:30.886306 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7241-42734b" event={"ID":"59299b05-48d7-4507-98e5-46c2681b576b","Type":"ContainerStarted","Data":"8d7be1f024bb8f859dbe3af338ab2c172ee56ca4d641b836f832510b97b3c465"} Nov 03 02:15:30.994279 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:30.994178 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7241-42734b" podStartSLOduration=1.1263295819999999 podStartE2EDuration="4.994165125s" totalImagesPullingTime="3.867835543s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:15:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:15:30.99200258 +0000 UTC m=+1408.251200748" watchObservedRunningTime="2026-11-03 02:15:30.994165125 +0000 UTC m=+1408.253363278" Nov 03 02:15:32.422227 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:32.422192 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7241-42734b"] Nov 03 02:15:32.425875 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:32.425798 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7241-42734b"] Nov 03 02:15:32.899292 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:32.899255 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7241-42734b" podUID="59299b05-48d7-4507-98e5-46c2681b576b" containerName="test-curl-7241-42734b" containerID="cri-o://8d7be1f024bb8f859dbe3af338ab2c172ee56ca4d641b836f832510b97b3c465" gracePeriod=2 Nov 03 02:15:33.303830 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:33.303798 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-701ba30f-s94vh"] Nov 03 02:15:33.304460 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:33.304443 5605 cpu_manager.go:414] "containerMap: removing container" podUID="988f453a-1c54-4410-85a1-aff8a5e66af4" containerName="cleanup" Nov 03 02:15:33.304522 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:33.304462 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="988f453a-1c54-4410-85a1-aff8a5e66af4" containerName="cleanup" Nov 03 02:15:33.304522 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:33.304474 5605 cpu_manager.go:414] "containerMap: removing container" podUID="59299b05-48d7-4507-98e5-46c2681b576b" containerName="test-curl-7241-42734b" Nov 03 02:15:33.304522 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:33.304479 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59299b05-48d7-4507-98e5-46c2681b576b" containerName="test-curl-7241-42734b" Nov 03 02:15:33.304522 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:33.304499 5605 cpu_manager.go:414] "containerMap: removing container" podUID="7d38408a-0e73-4d42-b78c-6ce430d3927c" containerName="cleanup" Nov 03 02:15:33.304522 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:33.304504 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d38408a-0e73-4d42-b78c-6ce430d3927c" containerName="cleanup" Nov 03 02:15:33.304522 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:33.304516 5605 cpu_manager.go:414] "containerMap: removing container" podUID="b4a0e603-b309-4a5b-861b-f1780b71f6e0" containerName="cleanup" Nov 03 02:15:33.304522 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:33.304521 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4a0e603-b309-4a5b-861b-f1780b71f6e0" containerName="cleanup" Nov 03 02:15:33.304736 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:33.304602 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec4f26be-fd2d-4f37-8200-eee788394037" containerName="cleanup" Nov 03 02:15:33.304736 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:33.304613 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4a0e603-b309-4a5b-861b-f1780b71f6e0" containerName="cleanup" Nov 03 02:15:33.304736 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:33.304621 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="59299b05-48d7-4507-98e5-46c2681b576b" containerName="test-curl-7241-42734b" Nov 03 02:15:33.304736 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:33.304630 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d38408a-0e73-4d42-b78c-6ce430d3927c" containerName="cleanup" Nov 03 02:15:33.309056 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:33.309038 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-701ba30f-s94vh" Nov 03 02:15:33.358453 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:33.358422 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:15:33.405043 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:33.405007 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:15:33.405043 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:33.405034 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:15:33.405260 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:33.405113 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:15:33.408389 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:33.408350 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-701ba30f-s94vh"] Nov 03 02:15:33.441662 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:33.441640 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9a67eff-471a-47e5-bede-7f8ff6ad8958-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-701ba30f-s94vh\" (UID: \"b9a67eff-471a-47e5-bede-7f8ff6ad8958\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-701ba30f-s94vh" Nov 03 02:15:33.442050 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:33.441669 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9a67eff-471a-47e5-bede-7f8ff6ad8958-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-701ba30f-s94vh\" (UID: \"b9a67eff-471a-47e5-bede-7f8ff6ad8958\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-701ba30f-s94vh" Nov 03 02:15:33.442050 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:33.441694 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db5r7\" (UniqueName: \"kubernetes.io/projected/b9a67eff-471a-47e5-bede-7f8ff6ad8958-kube-api-access-db5r7\") pod \"maas-api-revoke-keys-e2e-mt-701ba30f-s94vh\" (UID: \"b9a67eff-471a-47e5-bede-7f8ff6ad8958\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-701ba30f-s94vh" Nov 03 02:15:33.543735 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:33.543658 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9a67eff-471a-47e5-bede-7f8ff6ad8958-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-701ba30f-s94vh\" (UID: \"b9a67eff-471a-47e5-bede-7f8ff6ad8958\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-701ba30f-s94vh" Nov 03 02:15:33.543735 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:33.543712 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9a67eff-471a-47e5-bede-7f8ff6ad8958-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-701ba30f-s94vh\" (UID: \"b9a67eff-471a-47e5-bede-7f8ff6ad8958\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-701ba30f-s94vh" Nov 03 02:15:33.543914 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:33.543748 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db5r7\" (UniqueName: \"kubernetes.io/projected/b9a67eff-471a-47e5-bede-7f8ff6ad8958-kube-api-access-db5r7\") pod \"maas-api-revoke-keys-e2e-mt-701ba30f-s94vh\" (UID: \"b9a67eff-471a-47e5-bede-7f8ff6ad8958\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-701ba30f-s94vh" Nov 03 02:15:33.544439 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:33.544413 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9a67eff-471a-47e5-bede-7f8ff6ad8958-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-701ba30f-s94vh\" (UID: \"b9a67eff-471a-47e5-bede-7f8ff6ad8958\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-701ba30f-s94vh" Nov 03 02:15:33.546069 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:33.546046 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9a67eff-471a-47e5-bede-7f8ff6ad8958-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-701ba30f-s94vh\" (UID: \"b9a67eff-471a-47e5-bede-7f8ff6ad8958\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-701ba30f-s94vh" Nov 03 02:15:33.596996 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:33.596971 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-db5r7\" (UniqueName: \"kubernetes.io/projected/b9a67eff-471a-47e5-bede-7f8ff6ad8958-kube-api-access-db5r7\") pod \"maas-api-revoke-keys-e2e-mt-701ba30f-s94vh\" (UID: \"b9a67eff-471a-47e5-bede-7f8ff6ad8958\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-701ba30f-s94vh" Nov 03 02:15:33.623266 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:33.623241 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-701ba30f-s94vh" Nov 03 02:15:33.997804 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:33.997758 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-701ba30f-s94vh"] Nov 03 02:15:34.004476 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:15:34.004443 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9a67eff_471a_47e5_bede_7f8ff6ad8958.slice/crio-01711438f154ec2f91c5abdd562e3931f02886652bda811f3c01a8bff48270d1 WatchSource:0}: Error finding container 01711438f154ec2f91c5abdd562e3931f02886652bda811f3c01a8bff48270d1: Status 404 returned error can't find the container with id 01711438f154ec2f91c5abdd562e3931f02886652bda811f3c01a8bff48270d1 Nov 03 02:15:34.925441 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:34.925400 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-701ba30f-s94vh" event={"ID":"b9a67eff-471a-47e5-bede-7f8ff6ad8958","Type":"ContainerStarted","Data":"c4a7e3d1b97d5ec5460cb2d96e3ebbd196928b507b83c19a636fdbe7a6fbdb98"} Nov 03 02:15:34.925914 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:34.925478 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-701ba30f-s94vh" event={"ID":"b9a67eff-471a-47e5-bede-7f8ff6ad8958","Type":"ContainerStarted","Data":"01711438f154ec2f91c5abdd562e3931f02886652bda811f3c01a8bff48270d1"} Nov 03 02:15:34.927298 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:34.927266 5605 generic.go:356] "Generic (PLEG): container finished" podID="59299b05-48d7-4507-98e5-46c2681b576b" containerID="8d7be1f024bb8f859dbe3af338ab2c172ee56ca4d641b836f832510b97b3c465" exitCode=137 Nov 03 02:15:35.031919 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:35.031865 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-701ba30f-s94vh" podStartSLOduration=2.03184815 podStartE2EDuration="2.03184815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:15:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:15:35.030611936 +0000 UTC m=+1412.289810091" watchObservedRunningTime="2026-11-03 02:15:35.03184815 +0000 UTC m=+1412.291046325" Nov 03 02:15:35.132888 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:35.132857 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7241-42734b" Nov 03 02:15:35.281044 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:35.280957 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59299b05-48d7-4507-98e5-46c2681b576b-kube-api-access-w66sn\" (UniqueName: \"kubernetes.io/projected/59299b05-48d7-4507-98e5-46c2681b576b-kube-api-access-w66sn\") pod \"59299b05-48d7-4507-98e5-46c2681b576b\" (UID: \"59299b05-48d7-4507-98e5-46c2681b576b\") " Nov 03 02:15:35.283212 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:35.283180 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59299b05-48d7-4507-98e5-46c2681b576b-kube-api-access-w66sn" pod "59299b05-48d7-4507-98e5-46c2681b576b" (UID: "59299b05-48d7-4507-98e5-46c2681b576b"). InnerVolumeSpecName "kube-api-access-w66sn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:15:35.382575 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:35.382534 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w66sn\" (UniqueName: \"kubernetes.io/projected/59299b05-48d7-4507-98e5-46c2681b576b-kube-api-access-w66sn\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:15:35.936223 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:35.936130 5605 generic.go:356] "Generic (PLEG): container finished" podID="b9a67eff-471a-47e5-bede-7f8ff6ad8958" containerID="c4a7e3d1b97d5ec5460cb2d96e3ebbd196928b507b83c19a636fdbe7a6fbdb98" exitCode=0 Nov 03 02:15:35.936223 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:35.936201 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-701ba30f-s94vh" event={"ID":"b9a67eff-471a-47e5-bede-7f8ff6ad8958","Type":"ContainerDied","Data":"c4a7e3d1b97d5ec5460cb2d96e3ebbd196928b507b83c19a636fdbe7a6fbdb98"} Nov 03 02:15:35.937820 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:35.937797 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7241-42734b" Nov 03 02:15:35.937937 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:35.937839 5605 scope.go:122] "RemoveContainer" containerID="8d7be1f024bb8f859dbe3af338ab2c172ee56ca4d641b836f832510b97b3c465" Nov 03 02:15:36.089347 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:15:36.089297 5605 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7241-42734b\" is forbidden: User \"system:node:ip-10-0-140-140.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-140.us-west-2.compute.internal' and this object" podUID="59299b05-48d7-4507-98e5-46c2681b576b" pod="opendatahub/test-curl-7241-42734b" Nov 03 02:15:36.401954 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:36.401901 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf"] Nov 03 02:15:36.407681 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:36.407647 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf" Nov 03 02:15:36.429961 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:36.429924 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b"] Nov 03 02:15:36.434146 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:36.434120 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b" Nov 03 02:15:36.455637 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:15:36.455596 5605 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7241-42734b\" is forbidden: User \"system:node:ip-10-0-140-140.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-140.us-west-2.compute.internal' and this object" podUID="59299b05-48d7-4507-98e5-46c2681b576b" pod="opendatahub/test-curl-7241-42734b" Nov 03 02:15:36.497542 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:36.497503 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjtct\" (UniqueName: \"kubernetes.io/projected/cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d-kube-api-access-zjtct\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf\" (UID: \"cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf" Nov 03 02:15:36.497758 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:36.497566 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d-istio-token\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf\" (UID: \"cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf" Nov 03 02:15:36.497758 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:36.497698 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d-workload-certs\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf\" (UID: \"cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf" Nov 03 02:15:36.497758 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:36.497741 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d-istio-envoy\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf\" (UID: \"cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf" Nov 03 02:15:36.497933 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:36.497768 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d-istio-data\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf\" (UID: \"cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf" Nov 03 02:15:36.497933 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:36.497908 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d-credential-socket\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf\" (UID: \"cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf" Nov 03 02:15:36.498038 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:36.497979 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d-workload-socket\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf\" (UID: \"cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf" Nov 03 02:15:36.498092 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:36.498036 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d-istio-podinfo\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf\" (UID: \"cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf" Nov 03 02:15:36.498187 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:36.498159 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d-istiod-ca-cert\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf\" (UID: \"cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf" Nov 03 02:15:36.502447 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:36.502413 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf"] Nov 03 02:15:36.558531 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:36.556768 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b"] Nov 03 02:15:36.599777 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:36.599733 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d-workload-certs\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf\" (UID: \"cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf" Nov 03 02:15:36.599777 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:36.599778 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d-istio-envoy\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf\" (UID: \"cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf" Nov 03 02:15:36.600049 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:36.599798 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d-istio-data\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf\" (UID: \"cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf" Nov 03 02:15:36.600049 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:36.599821 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17-istio-data\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b\" (UID: \"338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b" Nov 03 02:15:36.600049 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:36.599870 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17-istio-podinfo\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b\" (UID: \"338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b" Nov 03 02:15:36.600049 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:36.599898 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d-credential-socket\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf\" (UID: \"cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf" Nov 03 02:15:36.600262 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:36.600085 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17-istio-envoy\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b\" (UID: \"338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b" Nov 03 02:15:36.600262 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:36.600116 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d-workload-socket\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf\" (UID: \"cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf" Nov 03 02:15:36.600262 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:36.600155 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d-istio-podinfo\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf\" (UID: \"cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf" Nov 03 02:15:36.600262 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:36.600181 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d-workload-certs\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf\" (UID: \"cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf" Nov 03 02:15:36.600262 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:36.600194 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17-credential-socket\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b\" (UID: \"338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b" Nov 03 02:15:36.600262 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:36.600212 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d-istio-data\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf\" (UID: \"cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf" Nov 03 02:15:36.600601 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:36.600303 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17-workload-socket\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b\" (UID: \"338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b" Nov 03 02:15:36.600601 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:36.600327 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d-credential-socket\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf\" (UID: \"cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf" Nov 03 02:15:36.600601 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:36.600345 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d-istiod-ca-cert\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf\" (UID: \"cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf" Nov 03 02:15:36.600601 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:36.600469 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d-workload-socket\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf\" (UID: \"cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf" Nov 03 02:15:36.600601 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:36.600551 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjtct\" (UniqueName: \"kubernetes.io/projected/cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d-kube-api-access-zjtct\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf\" (UID: \"cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf" Nov 03 02:15:36.600601 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:36.600585 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddwdx\" (UniqueName: \"kubernetes.io/projected/338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17-kube-api-access-ddwdx\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b\" (UID: \"338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b" Nov 03 02:15:36.600914 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:36.600678 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d-istio-token\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf\" (UID: \"cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf" Nov 03 02:15:36.600914 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:36.600707 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17-workload-certs\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b\" (UID: \"338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b" Nov 03 02:15:36.600914 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:36.600742 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17-istio-token\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b\" (UID: \"338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b" Nov 03 02:15:36.600914 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:36.600764 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17-istiod-ca-cert\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b\" (UID: \"338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b" Nov 03 02:15:36.600914 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:36.600887 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d-istiod-ca-cert\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf\" (UID: \"cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf" Nov 03 02:15:36.602201 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:36.602180 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d-istio-envoy\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf\" (UID: \"cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf" Nov 03 02:15:36.602939 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:36.602920 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d-istio-podinfo\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf\" (UID: \"cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf" Nov 03 02:15:36.614402 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:15:36.613522 5605 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7241-42734b\" is forbidden: User \"system:node:ip-10-0-140-140.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-140.us-west-2.compute.internal' and this object" podUID="59299b05-48d7-4507-98e5-46c2681b576b" pod="opendatahub/test-curl-7241-42734b" Nov 03 02:15:36.657480 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:36.657392 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjtct\" (UniqueName: \"kubernetes.io/projected/cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d-kube-api-access-zjtct\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf\" (UID: \"cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf" Nov 03 02:15:36.657640 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:36.657602 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d-istio-token\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf\" (UID: \"cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf" Nov 03 02:15:36.732385 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:36.727598 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17-istio-envoy\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b\" (UID: \"338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b" Nov 03 02:15:36.732385 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:36.727693 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17-credential-socket\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b\" (UID: \"338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b" Nov 03 02:15:36.732385 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:36.727740 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17-workload-socket\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b\" (UID: \"338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b" Nov 03 02:15:36.732385 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:36.727844 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddwdx\" (UniqueName: \"kubernetes.io/projected/338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17-kube-api-access-ddwdx\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b\" (UID: \"338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b" Nov 03 02:15:36.732385 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:36.727906 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17-workload-certs\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b\" (UID: \"338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b" Nov 03 02:15:36.732385 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:36.727939 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17-istio-token\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b\" (UID: \"338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b" Nov 03 02:15:36.732385 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:36.727966 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17-istiod-ca-cert\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b\" (UID: \"338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b" Nov 03 02:15:36.732385 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:36.728019 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17-istio-data\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b\" (UID: \"338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b" Nov 03 02:15:36.732385 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:36.728103 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17-istio-podinfo\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b\" (UID: \"338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b" Nov 03 02:15:36.754401 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:36.752629 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf" Nov 03 02:15:36.754401 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:36.753326 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17-credential-socket\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b\" (UID: \"338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b" Nov 03 02:15:36.754401 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:36.753599 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17-workload-socket\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b\" (UID: \"338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b" Nov 03 02:15:36.754694 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:36.754584 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17-workload-certs\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b\" (UID: \"338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b" Nov 03 02:15:36.781994 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:36.755223 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17-istiod-ca-cert\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b\" (UID: \"338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b" Nov 03 02:15:36.781994 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:36.755503 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17-istio-data\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b\" (UID: \"338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b" Nov 03 02:15:36.809254 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:36.795152 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddwdx\" (UniqueName: \"kubernetes.io/projected/338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17-kube-api-access-ddwdx\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b\" (UID: \"338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b" Nov 03 02:15:36.809254 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:36.795493 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17-istio-envoy\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b\" (UID: \"338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b" Nov 03 02:15:36.810016 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:36.809747 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17-istio-podinfo\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b\" (UID: \"338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b" Nov 03 02:15:36.810316 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:36.810267 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17-istio-token\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b\" (UID: \"338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b" Nov 03 02:15:36.984442 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:36.981090 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59299b05-48d7-4507-98e5-46c2681b576b" path="/var/lib/kubelet/pods/59299b05-48d7-4507-98e5-46c2681b576b/volumes" Nov 03 02:15:36.988687 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:36.988650 5605 generic.go:356] "Generic (PLEG): container finished" podID="87ac6502-78a2-41ed-9033-77eecf21086f" containerID="80fa1b08040a5dd9081261734bb32061db14746eb9c9c04e4d5e91f90881f114" exitCode=0 Nov 03 02:15:36.989239 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:36.989209 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8" event={"ID":"87ac6502-78a2-41ed-9033-77eecf21086f","Type":"ContainerDied","Data":"80fa1b08040a5dd9081261734bb32061db14746eb9c9c04e4d5e91f90881f114"} Nov 03 02:15:36.989994 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:36.989974 5605 scope.go:122] "RemoveContainer" containerID="80fa1b08040a5dd9081261734bb32061db14746eb9c9c04e4d5e91f90881f114" Nov 03 02:15:37.047383 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:37.026994 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:15:37.082374 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:37.082308 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:15:37.082685 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:37.082657 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:15:37.083274 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:37.050817 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b" Nov 03 02:15:37.576643 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:37.576612 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-701ba30f-s94vh" Nov 03 02:15:37.673442 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:37.673411 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9a67eff-471a-47e5-bede-7f8ff6ad8958-kube-api-access-db5r7\" (UniqueName: \"kubernetes.io/projected/b9a67eff-471a-47e5-bede-7f8ff6ad8958-kube-api-access-db5r7\") pod \"b9a67eff-471a-47e5-bede-7f8ff6ad8958\" (UID: \"b9a67eff-471a-47e5-bede-7f8ff6ad8958\") " Nov 03 02:15:37.673630 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:37.673524 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9a67eff-471a-47e5-bede-7f8ff6ad8958-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9a67eff-471a-47e5-bede-7f8ff6ad8958-maas-api-openshift-ca\") pod \"b9a67eff-471a-47e5-bede-7f8ff6ad8958\" (UID: \"b9a67eff-471a-47e5-bede-7f8ff6ad8958\") " Nov 03 02:15:37.673630 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:37.673600 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9a67eff-471a-47e5-bede-7f8ff6ad8958-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9a67eff-471a-47e5-bede-7f8ff6ad8958-maas-api-xks-ca\") pod \"b9a67eff-471a-47e5-bede-7f8ff6ad8958\" (UID: \"b9a67eff-471a-47e5-bede-7f8ff6ad8958\") " Nov 03 02:15:37.674021 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:37.673966 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9a67eff-471a-47e5-bede-7f8ff6ad8958-maas-api-openshift-ca" pod "b9a67eff-471a-47e5-bede-7f8ff6ad8958" (UID: "b9a67eff-471a-47e5-bede-7f8ff6ad8958"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:15:37.674267 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:37.674251 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9a67eff-471a-47e5-bede-7f8ff6ad8958-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:15:37.675833 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:37.675809 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9a67eff-471a-47e5-bede-7f8ff6ad8958-kube-api-access-db5r7" pod "b9a67eff-471a-47e5-bede-7f8ff6ad8958" (UID: "b9a67eff-471a-47e5-bede-7f8ff6ad8958"). InnerVolumeSpecName "kube-api-access-db5r7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:15:37.676066 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:37.676051 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9a67eff-471a-47e5-bede-7f8ff6ad8958-maas-api-xks-ca" pod "b9a67eff-471a-47e5-bede-7f8ff6ad8958" (UID: "b9a67eff-471a-47e5-bede-7f8ff6ad8958"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:15:37.776017 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:37.775975 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9a67eff-471a-47e5-bede-7f8ff6ad8958-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:15:37.776017 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:37.776014 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-db5r7\" (UniqueName: \"kubernetes.io/projected/b9a67eff-471a-47e5-bede-7f8ff6ad8958-kube-api-access-db5r7\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:15:37.820091 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:37.819575 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf"] Nov 03 02:15:37.830064 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:37.829960 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b"] Nov 03 02:15:37.831904 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:37.831609 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:15:37.831904 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:37.831692 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:15:37.831904 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:37.831737 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:15:37.832075 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:37.831978 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:15:37.832075 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:37.832052 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:15:37.832150 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:37.832093 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:15:38.005850 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:38.005821 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-701ba30f-s94vh" Nov 03 02:15:38.007422 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:38.005831 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-701ba30f-s94vh" event={"ID":"b9a67eff-471a-47e5-bede-7f8ff6ad8958","Type":"ContainerDied","Data":"01711438f154ec2f91c5abdd562e3931f02886652bda811f3c01a8bff48270d1"} Nov 03 02:15:38.007422 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:38.007097 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01711438f154ec2f91c5abdd562e3931f02886652bda811f3c01a8bff48270d1" Nov 03 02:15:38.009463 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:38.008638 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8" event={"ID":"87ac6502-78a2-41ed-9033-77eecf21086f","Type":"ContainerStarted","Data":"d44aa5007f19f3bd09d319f9ae1d357d1c45730d4d6f5ac24750271d337ac1fa"} Nov 03 02:15:38.011208 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:38.010526 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b" event={"ID":"338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17","Type":"ContainerStarted","Data":"7599139daa8c656f02beccfaf750a431ba0ed088a5ed594b1c33c9eea7a7c8f1"} Nov 03 02:15:38.011208 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:38.010571 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b" event={"ID":"338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17","Type":"ContainerStarted","Data":"0b8d8e60ca95699795154e6dab5debacc39728e3397b20de28595659fedbe846"} Nov 03 02:15:38.013728 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:38.013073 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf" event={"ID":"cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d","Type":"ContainerStarted","Data":"643d31b51ebc8524645f42031921943ab9e4f46c5fec97cae44d153343ab555f"} Nov 03 02:15:38.013728 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:38.013118 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf" event={"ID":"cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d","Type":"ContainerStarted","Data":"9806cd2a0e192b87e5adfcd80422dbb2d382a30375ae3d06ade6cb8bec89282d"} Nov 03 02:15:38.051918 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:38.051694 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b" Nov 03 02:15:38.052655 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:38.052630 5605 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 03 02:15:38.052717 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:38.052692 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b" podUID="338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 03 02:15:38.077443 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:38.073462 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8" Nov 03 02:15:38.077443 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:38.076835 5605 patch_prober.go:29] interesting pod/e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 02:15:38.077443 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:38.076907 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8" podUID="87ac6502-78a2-41ed-9033-77eecf21086f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 02:15:38.303668 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:38.303599 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf" podStartSLOduration=2.303582618 podStartE2EDuration="2.303582618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:15:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:15:38.302986475 +0000 UTC m=+1415.562184634" watchObservedRunningTime="2026-11-03 02:15:38.303582618 +0000 UTC m=+1415.562780774" Nov 03 02:15:38.428333 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:38.424555 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b" podStartSLOduration=2.424532271 podStartE2EDuration="2.424532271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:15:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:15:38.421153676 +0000 UTC m=+1415.680351827" watchObservedRunningTime="2026-11-03 02:15:38.424532271 +0000 UTC m=+1415.683730427" Nov 03 02:15:38.765439 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:38.765315 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf" Nov 03 02:15:38.765639 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:38.765545 5605 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 03 02:15:38.765639 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:38.765595 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf" podUID="cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 03 02:15:39.051419 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:39.051390 5605 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 03 02:15:39.051852 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:39.051826 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b" podUID="338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 03 02:15:39.083384 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:39.076813 5605 patch_prober.go:29] interesting pod/e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 02:15:39.083384 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:39.076880 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8" podUID="87ac6502-78a2-41ed-9033-77eecf21086f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 02:15:39.754876 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:39.754830 5605 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 03 02:15:39.755245 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:39.755209 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf" podUID="cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 03 02:15:40.051743 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:40.051714 5605 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 03 02:15:40.052103 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:40.051768 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b" podUID="338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 03 02:15:40.083384 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:40.075694 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8" Nov 03 02:15:40.083384 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:40.077980 5605 patch_prober.go:29] interesting pod/e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 02:15:40.083384 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:40.078037 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8" podUID="87ac6502-78a2-41ed-9033-77eecf21086f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 02:15:40.762386 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:40.758796 5605 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 03 02:15:40.762386 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:40.758866 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf" podUID="cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 03 02:15:41.051761 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:41.051719 5605 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 03 02:15:41.052144 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:41.051784 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b" podUID="338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 03 02:15:41.074833 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:41.074785 5605 patch_prober.go:29] interesting pod/e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 02:15:41.075128 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:41.075095 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8" podUID="87ac6502-78a2-41ed-9033-77eecf21086f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 02:15:41.758386 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:41.754431 5605 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 03 02:15:41.762384 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:41.758690 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf" podUID="cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 03 02:15:42.051827 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:42.051790 5605 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 03 02:15:42.052338 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:42.051857 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b" podUID="338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 03 02:15:42.092384 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:42.092047 5605 patch_prober.go:29] interesting pod/e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 02:15:42.092384 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:42.092132 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8" podUID="87ac6502-78a2-41ed-9033-77eecf21086f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 02:15:42.762385 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:42.758791 5605 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 03 02:15:42.762385 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:42.758852 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf" podUID="cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 03 02:15:43.051954 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:43.051916 5605 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 03 02:15:43.052373 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:43.051982 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b" podUID="338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 03 02:15:43.073382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:43.072802 5605 patch_prober.go:29] interesting pod/e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 02:15:43.073382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:43.072867 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8" podUID="87ac6502-78a2-41ed-9033-77eecf21086f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 02:15:43.762546 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:43.762500 5605 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 03 02:15:43.762850 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:43.762814 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf" podUID="cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 03 02:15:44.051867 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:44.051828 5605 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 03 02:15:44.052069 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:44.051900 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b" podUID="338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 03 02:15:44.100382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:44.099742 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8" Nov 03 02:15:44.107382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:44.103543 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8" Nov 03 02:15:44.754916 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:44.754866 5605 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 03 02:15:44.755127 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:44.754935 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf" podUID="cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 03 02:15:45.145437 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:45.144830 5605 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 03 02:15:45.145437 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:45.144957 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b" podUID="338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 03 02:15:46.078883 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:46.078848 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf" Nov 03 02:15:46.080269 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:46.080248 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf" Nov 03 02:15:46.081768 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:46.081736 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf" Nov 03 02:15:46.147462 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:46.147414 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b" Nov 03 02:15:46.148430 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:46.148398 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b" Nov 03 02:15:46.149399 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:46.149381 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b" Nov 03 02:15:48.043739 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:48.043703 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fca8eb6-sj67k"] Nov 03 02:15:48.090234 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:48.090195 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fca8eb6-sj67k"] Nov 03 02:15:48.852255 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:48.852220 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff93bdda-9ad2-4fbb-b36b-107aebce350d" path="/var/lib/kubelet/pods/ff93bdda-9ad2-4fbb-b36b-107aebce350d/volumes" Nov 03 02:15:51.869201 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:15:51.869162 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:15:53.915981 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:15:53.915941 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:16:01.049251 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:01.049214 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0e38c2b-jdbqx"] Nov 03 02:16:01.096106 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:01.096070 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0e38c2b-jdbqx"] Nov 03 02:16:02.858627 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:02.858591 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="171615bb-573d-4b45-8566-dd2a44262a38" path="/var/lib/kubelet/pods/171615bb-573d-4b45-8566-dd2a44262a38/volumes" Nov 03 02:16:18.449087 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:18.448873 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f911b8f-jl4ph"] Nov 03 02:16:18.452205 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:18.452173 5605 cpu_manager.go:414] "containerMap: removing container" podUID="b9a67eff-471a-47e5-bede-7f8ff6ad8958" containerName="revoke-keys" Nov 03 02:16:18.452205 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:18.452201 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9a67eff-471a-47e5-bede-7f8ff6ad8958" containerName="revoke-keys" Nov 03 02:16:18.452479 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:18.452304 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9a67eff-471a-47e5-bede-7f8ff6ad8958" containerName="revoke-keys" Nov 03 02:16:18.463929 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:18.459704 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f911b8f-jl4ph" Nov 03 02:16:18.503545 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:18.503504 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f911b8f-jl4ph"] Nov 03 02:16:18.538970 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:18.538922 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c5b8\" (UniqueName: \"kubernetes.io/projected/0c65d37f-f5f0-48a1-a4d8-9cfdeae07e75-kube-api-access-2c5b8\") pod \"maas-api-revoke-sub-5f911b8f-jl4ph\" (UID: \"0c65d37f-f5f0-48a1-a4d8-9cfdeae07e75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f911b8f-jl4ph" Nov 03 02:16:18.538970 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:18.538975 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c65d37f-f5f0-48a1-a4d8-9cfdeae07e75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f911b8f-jl4ph\" (UID: \"0c65d37f-f5f0-48a1-a4d8-9cfdeae07e75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f911b8f-jl4ph" Nov 03 02:16:18.539267 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:18.539205 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c65d37f-f5f0-48a1-a4d8-9cfdeae07e75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f911b8f-jl4ph\" (UID: \"0c65d37f-f5f0-48a1-a4d8-9cfdeae07e75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f911b8f-jl4ph" Nov 03 02:16:18.545716 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:18.545681 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:16:18.545885 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:18.545757 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:16:18.545885 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:18.545770 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:16:18.555766 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:18.555733 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:16:18.640787 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:18.640740 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c5b8\" (UniqueName: \"kubernetes.io/projected/0c65d37f-f5f0-48a1-a4d8-9cfdeae07e75-kube-api-access-2c5b8\") pod \"maas-api-revoke-sub-5f911b8f-jl4ph\" (UID: \"0c65d37f-f5f0-48a1-a4d8-9cfdeae07e75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f911b8f-jl4ph" Nov 03 02:16:18.641012 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:18.640798 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c65d37f-f5f0-48a1-a4d8-9cfdeae07e75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f911b8f-jl4ph\" (UID: \"0c65d37f-f5f0-48a1-a4d8-9cfdeae07e75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f911b8f-jl4ph" Nov 03 02:16:18.641012 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:18.640977 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c65d37f-f5f0-48a1-a4d8-9cfdeae07e75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f911b8f-jl4ph\" (UID: \"0c65d37f-f5f0-48a1-a4d8-9cfdeae07e75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f911b8f-jl4ph" Nov 03 02:16:18.641630 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:18.641605 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c65d37f-f5f0-48a1-a4d8-9cfdeae07e75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f911b8f-jl4ph\" (UID: \"0c65d37f-f5f0-48a1-a4d8-9cfdeae07e75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f911b8f-jl4ph" Nov 03 02:16:18.643214 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:18.643195 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c65d37f-f5f0-48a1-a4d8-9cfdeae07e75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f911b8f-jl4ph\" (UID: \"0c65d37f-f5f0-48a1-a4d8-9cfdeae07e75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f911b8f-jl4ph" Nov 03 02:16:18.694089 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:18.694052 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c5b8\" (UniqueName: \"kubernetes.io/projected/0c65d37f-f5f0-48a1-a4d8-9cfdeae07e75-kube-api-access-2c5b8\") pod \"maas-api-revoke-sub-5f911b8f-jl4ph\" (UID: \"0c65d37f-f5f0-48a1-a4d8-9cfdeae07e75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f911b8f-jl4ph" Nov 03 02:16:18.782637 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:18.782547 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f911b8f-jl4ph" Nov 03 02:16:19.046279 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:19.046193 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e0624c5-fcdxk"] Nov 03 02:16:19.051219 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:19.051184 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e0624c5-fcdxk"] Nov 03 02:16:19.156121 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:19.156074 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f911b8f-jl4ph"] Nov 03 02:16:19.161396 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:16:19.161335 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c65d37f_f5f0_48a1_a4d8_9cfdeae07e75.slice/crio-b116bcaee9a15d4fd071422fbaa8cc3d3f6d1028b6bb53cff42149c3a76da85a WatchSource:0}: Error finding container b116bcaee9a15d4fd071422fbaa8cc3d3f6d1028b6bb53cff42149c3a76da85a: Status 404 returned error can't find the container with id b116bcaee9a15d4fd071422fbaa8cc3d3f6d1028b6bb53cff42149c3a76da85a Nov 03 02:16:19.471551 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:19.471516 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f911b8f-jl4ph" event={"ID":"0c65d37f-f5f0-48a1-a4d8-9cfdeae07e75","Type":"ContainerStarted","Data":"b116bcaee9a15d4fd071422fbaa8cc3d3f6d1028b6bb53cff42149c3a76da85a"} Nov 03 02:16:20.480611 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:20.480511 5605 generic.go:356] "Generic (PLEG): container finished" podID="0c65d37f-f5f0-48a1-a4d8-9cfdeae07e75" containerID="8f056eccafef0869ada939c19dfe4ad5c9a4e4c8ada1fbe470097f9c64a65eed" exitCode=0 Nov 03 02:16:20.480611 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:20.480590 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f911b8f-jl4ph" event={"ID":"0c65d37f-f5f0-48a1-a4d8-9cfdeae07e75","Type":"ContainerDied","Data":"8f056eccafef0869ada939c19dfe4ad5c9a4e4c8ada1fbe470097f9c64a65eed"} Nov 03 02:16:20.850734 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:20.850701 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8772ad29-8e2b-4236-a07a-99878ef696cc" path="/var/lib/kubelet/pods/8772ad29-8e2b-4236-a07a-99878ef696cc/volumes" Nov 03 02:16:21.644969 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:21.644941 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f911b8f-jl4ph" Nov 03 02:16:21.693616 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:21.693584 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c65d37f-f5f0-48a1-a4d8-9cfdeae07e75-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c65d37f-f5f0-48a1-a4d8-9cfdeae07e75-maas-api-openshift-ca\") pod \"0c65d37f-f5f0-48a1-a4d8-9cfdeae07e75\" (UID: \"0c65d37f-f5f0-48a1-a4d8-9cfdeae07e75\") " Nov 03 02:16:21.693799 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:21.693625 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c65d37f-f5f0-48a1-a4d8-9cfdeae07e75-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c65d37f-f5f0-48a1-a4d8-9cfdeae07e75-maas-api-xks-ca\") pod \"0c65d37f-f5f0-48a1-a4d8-9cfdeae07e75\" (UID: \"0c65d37f-f5f0-48a1-a4d8-9cfdeae07e75\") " Nov 03 02:16:21.693799 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:21.693655 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c65d37f-f5f0-48a1-a4d8-9cfdeae07e75-kube-api-access-2c5b8\" (UniqueName: \"kubernetes.io/projected/0c65d37f-f5f0-48a1-a4d8-9cfdeae07e75-kube-api-access-2c5b8\") pod \"0c65d37f-f5f0-48a1-a4d8-9cfdeae07e75\" (UID: \"0c65d37f-f5f0-48a1-a4d8-9cfdeae07e75\") " Nov 03 02:16:21.694050 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:21.694015 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c65d37f-f5f0-48a1-a4d8-9cfdeae07e75-maas-api-openshift-ca" pod "0c65d37f-f5f0-48a1-a4d8-9cfdeae07e75" (UID: "0c65d37f-f5f0-48a1-a4d8-9cfdeae07e75"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:16:21.694169 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:21.694152 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c65d37f-f5f0-48a1-a4d8-9cfdeae07e75-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:21.695797 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:21.695766 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c65d37f-f5f0-48a1-a4d8-9cfdeae07e75-maas-api-xks-ca" pod "0c65d37f-f5f0-48a1-a4d8-9cfdeae07e75" (UID: "0c65d37f-f5f0-48a1-a4d8-9cfdeae07e75"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:16:21.695797 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:21.695787 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c65d37f-f5f0-48a1-a4d8-9cfdeae07e75-kube-api-access-2c5b8" pod "0c65d37f-f5f0-48a1-a4d8-9cfdeae07e75" (UID: "0c65d37f-f5f0-48a1-a4d8-9cfdeae07e75"). InnerVolumeSpecName "kube-api-access-2c5b8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:16:21.795128 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:21.795046 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c65d37f-f5f0-48a1-a4d8-9cfdeae07e75-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:21.795128 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:21.795080 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c5b8\" (UniqueName: \"kubernetes.io/projected/0c65d37f-f5f0-48a1-a4d8-9cfdeae07e75-kube-api-access-2c5b8\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:22.493919 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:22.493887 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f911b8f-jl4ph" Nov 03 02:16:22.494102 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:22.493922 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f911b8f-jl4ph" event={"ID":"0c65d37f-f5f0-48a1-a4d8-9cfdeae07e75","Type":"ContainerDied","Data":"b116bcaee9a15d4fd071422fbaa8cc3d3f6d1028b6bb53cff42149c3a76da85a"} Nov 03 02:16:22.494102 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:22.493979 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b116bcaee9a15d4fd071422fbaa8cc3d3f6d1028b6bb53cff42149c3a76da85a" Nov 03 02:16:22.923224 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:22.923189 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:16:24.112843 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:16:24.112514 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:16:31.044031 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:31.043999 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5911deb-p7d59"] Nov 03 02:16:31.091169 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:31.091144 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5911deb-p7d59"] Nov 03 02:16:32.851221 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:32.851188 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31c54e65-3b9f-4c2e-b8a8-a9a5166fdedf" path="/var/lib/kubelet/pods/31c54e65-3b9f-4c2e-b8a8-a9a5166fdedf/volumes" Nov 03 02:16:49.400718 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:49.400681 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28"] Nov 03 02:16:49.401256 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:49.400991 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28" podUID="5c586c92-59da-421e-8224-45b9a833c2d4" containerName="istio-proxy" containerID="cri-o://788c0b9b519150225b16e0b4159f319099b242ae05216b472cf01f8bef2be731" gracePeriod=30 Nov 03 02:16:53.916961 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:16:53.916925 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:16:54.676046 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:54.675988 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28" Nov 03 02:16:54.733802 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:54.733774 5605 generic.go:356] "Generic (PLEG): container finished" podID="5c586c92-59da-421e-8224-45b9a833c2d4" containerID="788c0b9b519150225b16e0b4159f319099b242ae05216b472cf01f8bef2be731" exitCode=0 Nov 03 02:16:54.733916 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:54.733821 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28" Nov 03 02:16:54.733916 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:54.733824 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28" event={"ID":"5c586c92-59da-421e-8224-45b9a833c2d4","Type":"ContainerDied","Data":"788c0b9b519150225b16e0b4159f319099b242ae05216b472cf01f8bef2be731"} Nov 03 02:16:54.733993 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:54.733956 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28" event={"ID":"5c586c92-59da-421e-8224-45b9a833c2d4","Type":"ContainerDied","Data":"87d7f9ea389ead9c2d5dd6a6443037905478bf7ec2ac0cbcf2a730e387282e00"} Nov 03 02:16:54.733993 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:54.733984 5605 scope.go:122] "RemoveContainer" containerID="788c0b9b519150225b16e0b4159f319099b242ae05216b472cf01f8bef2be731" Nov 03 02:16:54.743344 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:54.743323 5605 scope.go:122] "RemoveContainer" containerID="788c0b9b519150225b16e0b4159f319099b242ae05216b472cf01f8bef2be731" Nov 03 02:16:54.743633 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:16:54.743612 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"788c0b9b519150225b16e0b4159f319099b242ae05216b472cf01f8bef2be731\": container with ID starting with 788c0b9b519150225b16e0b4159f319099b242ae05216b472cf01f8bef2be731 not found: ID does not exist" containerID="788c0b9b519150225b16e0b4159f319099b242ae05216b472cf01f8bef2be731" Nov 03 02:16:54.743706 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:54.743643 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"788c0b9b519150225b16e0b4159f319099b242ae05216b472cf01f8bef2be731"} err="failed to get container status \"788c0b9b519150225b16e0b4159f319099b242ae05216b472cf01f8bef2be731\": rpc error: code = NotFound desc = could not find container \"788c0b9b519150225b16e0b4159f319099b242ae05216b472cf01f8bef2be731\": container with ID starting with 788c0b9b519150225b16e0b4159f319099b242ae05216b472cf01f8bef2be731 not found: ID does not exist" Nov 03 02:16:54.771382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:54.769791 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c586c92-59da-421e-8224-45b9a833c2d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c586c92-59da-421e-8224-45b9a833c2d4-workload-certs\") pod \"5c586c92-59da-421e-8224-45b9a833c2d4\" (UID: \"5c586c92-59da-421e-8224-45b9a833c2d4\") " Nov 03 02:16:54.771382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:54.769960 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c586c92-59da-421e-8224-45b9a833c2d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c586c92-59da-421e-8224-45b9a833c2d4-istiod-ca-cert\") pod \"5c586c92-59da-421e-8224-45b9a833c2d4\" (UID: \"5c586c92-59da-421e-8224-45b9a833c2d4\") " Nov 03 02:16:54.771382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:54.770043 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c586c92-59da-421e-8224-45b9a833c2d4-kube-api-access-r4w5n\" (UniqueName: \"kubernetes.io/projected/5c586c92-59da-421e-8224-45b9a833c2d4-kube-api-access-r4w5n\") pod \"5c586c92-59da-421e-8224-45b9a833c2d4\" (UID: \"5c586c92-59da-421e-8224-45b9a833c2d4\") " Nov 03 02:16:54.771382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:54.770087 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c586c92-59da-421e-8224-45b9a833c2d4-istio-token\" (UniqueName: \"kubernetes.io/projected/5c586c92-59da-421e-8224-45b9a833c2d4-istio-token\") pod \"5c586c92-59da-421e-8224-45b9a833c2d4\" (UID: \"5c586c92-59da-421e-8224-45b9a833c2d4\") " Nov 03 02:16:54.771382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:54.770187 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c586c92-59da-421e-8224-45b9a833c2d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c586c92-59da-421e-8224-45b9a833c2d4-istio-podinfo\") pod \"5c586c92-59da-421e-8224-45b9a833c2d4\" (UID: \"5c586c92-59da-421e-8224-45b9a833c2d4\") " Nov 03 02:16:54.771382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:54.770219 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c586c92-59da-421e-8224-45b9a833c2d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c586c92-59da-421e-8224-45b9a833c2d4-credential-socket\") pod \"5c586c92-59da-421e-8224-45b9a833c2d4\" (UID: \"5c586c92-59da-421e-8224-45b9a833c2d4\") " Nov 03 02:16:54.771382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:54.770255 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c586c92-59da-421e-8224-45b9a833c2d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c586c92-59da-421e-8224-45b9a833c2d4-istio-data\") pod \"5c586c92-59da-421e-8224-45b9a833c2d4\" (UID: \"5c586c92-59da-421e-8224-45b9a833c2d4\") " Nov 03 02:16:54.771382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:54.770414 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c586c92-59da-421e-8224-45b9a833c2d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c586c92-59da-421e-8224-45b9a833c2d4-workload-socket\") pod \"5c586c92-59da-421e-8224-45b9a833c2d4\" (UID: \"5c586c92-59da-421e-8224-45b9a833c2d4\") " Nov 03 02:16:54.771382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:54.770489 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c586c92-59da-421e-8224-45b9a833c2d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c586c92-59da-421e-8224-45b9a833c2d4-istio-envoy\") pod \"5c586c92-59da-421e-8224-45b9a833c2d4\" (UID: \"5c586c92-59da-421e-8224-45b9a833c2d4\") " Nov 03 02:16:54.772606 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:54.772252 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c586c92-59da-421e-8224-45b9a833c2d4-credential-socket" pod "5c586c92-59da-421e-8224-45b9a833c2d4" (UID: "5c586c92-59da-421e-8224-45b9a833c2d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:16:54.772606 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:54.772482 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c586c92-59da-421e-8224-45b9a833c2d4-istiod-ca-cert" pod "5c586c92-59da-421e-8224-45b9a833c2d4" (UID: "5c586c92-59da-421e-8224-45b9a833c2d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:16:54.773440 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:54.773347 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c586c92-59da-421e-8224-45b9a833c2d4-istio-data" pod "5c586c92-59da-421e-8224-45b9a833c2d4" (UID: "5c586c92-59da-421e-8224-45b9a833c2d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:16:54.773440 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:54.773433 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c586c92-59da-421e-8224-45b9a833c2d4-workload-socket" pod "5c586c92-59da-421e-8224-45b9a833c2d4" (UID: "5c586c92-59da-421e-8224-45b9a833c2d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:16:54.773649 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:54.773625 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c586c92-59da-421e-8224-45b9a833c2d4-workload-certs" pod "5c586c92-59da-421e-8224-45b9a833c2d4" (UID: "5c586c92-59da-421e-8224-45b9a833c2d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:16:54.775846 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:54.775819 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c586c92-59da-421e-8224-45b9a833c2d4-istio-envoy" pod "5c586c92-59da-421e-8224-45b9a833c2d4" (UID: "5c586c92-59da-421e-8224-45b9a833c2d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:16:54.775846 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:54.775839 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c586c92-59da-421e-8224-45b9a833c2d4-kube-api-access-r4w5n" pod "5c586c92-59da-421e-8224-45b9a833c2d4" (UID: "5c586c92-59da-421e-8224-45b9a833c2d4"). InnerVolumeSpecName "kube-api-access-r4w5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:16:54.776005 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:54.775865 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c586c92-59da-421e-8224-45b9a833c2d4-istio-token" pod "5c586c92-59da-421e-8224-45b9a833c2d4" (UID: "5c586c92-59da-421e-8224-45b9a833c2d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:16:54.776225 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:54.776196 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c586c92-59da-421e-8224-45b9a833c2d4-istio-podinfo" pod "5c586c92-59da-421e-8224-45b9a833c2d4" (UID: "5c586c92-59da-421e-8224-45b9a833c2d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:16:54.872913 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:54.872883 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4w5n\" (UniqueName: \"kubernetes.io/projected/5c586c92-59da-421e-8224-45b9a833c2d4-kube-api-access-r4w5n\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:54.872913 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:54.872912 5605 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c586c92-59da-421e-8224-45b9a833c2d4-istio-token\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:54.873085 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:54.872927 5605 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c586c92-59da-421e-8224-45b9a833c2d4-istio-podinfo\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:54.873085 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:54.872941 5605 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c586c92-59da-421e-8224-45b9a833c2d4-credential-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:54.873085 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:54.872955 5605 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c586c92-59da-421e-8224-45b9a833c2d4-istio-data\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:54.873085 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:54.872968 5605 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c586c92-59da-421e-8224-45b9a833c2d4-workload-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:54.873085 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:54.872980 5605 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c586c92-59da-421e-8224-45b9a833c2d4-istio-envoy\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:54.873085 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:54.872994 5605 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c586c92-59da-421e-8224-45b9a833c2d4-workload-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:54.873085 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:54.873006 5605 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c586c92-59da-421e-8224-45b9a833c2d4-istiod-ca-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:55.142379 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:55.142338 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28"] Nov 03 02:16:55.147140 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:55.147113 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-txq28"] Nov 03 02:16:56.144589 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:56.144554 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:16:56.850569 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:16:56.850539 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c586c92-59da-421e-8224-45b9a833c2d4" path="/var/lib/kubelet/pods/5c586c92-59da-421e-8224-45b9a833c2d4/volumes" Nov 03 02:17:03.194855 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:03.194825 5605 scope.go:122] "RemoveContainer" containerID="1a2420b64178a15abce2299dc1f7c9bfafb7fd0e4f3e100890acdb0729157bd6" Nov 03 02:17:03.204065 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:03.204034 5605 scope.go:122] "RemoveContainer" containerID="3166cd7d006757d5590cf65dbcbf161b28fa36e9bb60b3c6bb68da5f7531f81f" Nov 03 02:17:03.212557 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:03.212537 5605 scope.go:122] "RemoveContainer" containerID="75bbffff08316585c934d7f4467f64d709e2fa387d6343000ed29f280529d4d9" Nov 03 02:17:03.220558 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:03.220537 5605 scope.go:122] "RemoveContainer" containerID="7739f3acfd1d29155b570fe5cfab0aa8f0d68ba8741c381f020ef8cea05afa88" Nov 03 02:17:08.918088 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:08.918048 5605 generic.go:356] "Generic (PLEG): container finished" podID="2225e2b8-fca7-4a8a-8c47-a687a0d9b214" containerID="8938b6aa86b546ebb49211ec1e640193f4f45bf3962d90615893c4f209f900aa" exitCode=0 Nov 03 02:17:08.918678 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:08.918116 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d" event={"ID":"2225e2b8-fca7-4a8a-8c47-a687a0d9b214","Type":"ContainerDied","Data":"8938b6aa86b546ebb49211ec1e640193f4f45bf3962d90615893c4f209f900aa"} Nov 03 02:17:08.918678 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:08.918619 5605 scope.go:122] "RemoveContainer" containerID="8938b6aa86b546ebb49211ec1e640193f4f45bf3962d90615893c4f209f900aa" Nov 03 02:17:08.920680 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:08.920646 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:17:08.920793 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:08.920726 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:17:08.920793 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:08.920762 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:17:09.945653 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:09.945616 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d" event={"ID":"2225e2b8-fca7-4a8a-8c47-a687a0d9b214","Type":"ContainerStarted","Data":"94cab9770b825e5812b835012f8f16b0b99c81f425d477eaf4ce513cf9960c6d"} Nov 03 02:17:10.542050 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:10.542016 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a772c3-6cc589fd8c-pf6hh"] Nov 03 02:17:10.554915 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:10.554884 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d" Nov 03 02:17:10.576608 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:10.576567 5605 patch_prober.go:29] interesting pod/e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 03 02:17:10.576932 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:10.576882 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d" podUID="2225e2b8-fca7-4a8a-8c47-a687a0d9b214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 03 02:17:11.331118 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:11.330679 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-701ba30f-s94vh"] Nov 03 02:17:11.334248 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:11.334222 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-701ba30f-s94vh"] Nov 03 02:17:11.551625 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:11.551589 5605 patch_prober.go:29] interesting pod/e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 03 02:17:11.551812 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:11.551662 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d" podUID="2225e2b8-fca7-4a8a-8c47-a687a0d9b214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 03 02:17:12.551561 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:12.551529 5605 patch_prober.go:29] interesting pod/e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 03 02:17:12.551987 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:12.551586 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d" podUID="2225e2b8-fca7-4a8a-8c47-a687a0d9b214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 03 02:17:12.850555 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:12.850480 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9a67eff-471a-47e5-bede-7f8ff6ad8958" path="/var/lib/kubelet/pods/b9a67eff-471a-47e5-bede-7f8ff6ad8958/volumes" Nov 03 02:17:13.551975 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:13.551942 5605 patch_prober.go:29] interesting pod/e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 03 02:17:13.552463 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:13.552000 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d" podUID="2225e2b8-fca7-4a8a-8c47-a687a0d9b214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 03 02:17:14.515138 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:14.515096 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d" Nov 03 02:17:14.551985 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:14.551955 5605 patch_prober.go:29] interesting pod/e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 03 02:17:14.552372 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:14.552008 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d" podUID="2225e2b8-fca7-4a8a-8c47-a687a0d9b214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 03 02:17:15.542809 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:15.542763 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a772c3-6cc589fd8c-pf6hh" podUID="3e782231-a2f8-45d9-8507-fcc038f1a368" containerName="payload-pre-processing" containerID="cri-o://87835394c46ddbe897479e341429a5b51ea93bdb5b2cf2311c2b0e052ed5dad9" gracePeriod=25 Nov 03 02:17:15.551646 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:15.551619 5605 patch_prober.go:29] interesting pod/e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 03 02:17:15.551743 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:15.551670 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d" podUID="2225e2b8-fca7-4a8a-8c47-a687a0d9b214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 03 02:17:15.952724 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:15.952698 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a772c3-6cc589fd8c-pf6hh" Nov 03 02:17:16.003713 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:16.003681 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e782231-a2f8-45d9-8507-fcc038f1a368-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e782231-a2f8-45d9-8507-fcc038f1a368-plugins-config-volume\") pod \"3e782231-a2f8-45d9-8507-fcc038f1a368\" (UID: \"3e782231-a2f8-45d9-8507-fcc038f1a368\") " Nov 03 02:17:16.003868 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:16.003811 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e782231-a2f8-45d9-8507-fcc038f1a368-kube-api-access-qqnrs\" (UniqueName: \"kubernetes.io/projected/3e782231-a2f8-45d9-8507-fcc038f1a368-kube-api-access-qqnrs\") pod \"3e782231-a2f8-45d9-8507-fcc038f1a368\" (UID: \"3e782231-a2f8-45d9-8507-fcc038f1a368\") " Nov 03 02:17:16.004100 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:16.004072 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e782231-a2f8-45d9-8507-fcc038f1a368-plugins-config-volume" pod "3e782231-a2f8-45d9-8507-fcc038f1a368" (UID: "3e782231-a2f8-45d9-8507-fcc038f1a368"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:17:16.004401 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:16.004350 5605 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e782231-a2f8-45d9-8507-fcc038f1a368-plugins-config-volume\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:16.005866 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:16.005841 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e782231-a2f8-45d9-8507-fcc038f1a368-kube-api-access-qqnrs" pod "3e782231-a2f8-45d9-8507-fcc038f1a368" (UID: "3e782231-a2f8-45d9-8507-fcc038f1a368"). InnerVolumeSpecName "kube-api-access-qqnrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:17:16.119383 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:16.108235 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqnrs\" (UniqueName: \"kubernetes.io/projected/3e782231-a2f8-45d9-8507-fcc038f1a368-kube-api-access-qqnrs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:16.285090 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:16.284979 5605 generic.go:356] "Generic (PLEG): container finished" podID="3e782231-a2f8-45d9-8507-fcc038f1a368" containerID="87835394c46ddbe897479e341429a5b51ea93bdb5b2cf2311c2b0e052ed5dad9" exitCode=0 Nov 03 02:17:16.285090 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:16.285030 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a772c3-6cc589fd8c-pf6hh" Nov 03 02:17:16.285343 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:16.285135 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a772c3-6cc589fd8c-pf6hh" event={"ID":"3e782231-a2f8-45d9-8507-fcc038f1a368","Type":"ContainerDied","Data":"87835394c46ddbe897479e341429a5b51ea93bdb5b2cf2311c2b0e052ed5dad9"} Nov 03 02:17:16.285343 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:16.285178 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a772c3-6cc589fd8c-pf6hh" event={"ID":"3e782231-a2f8-45d9-8507-fcc038f1a368","Type":"ContainerDied","Data":"68e9f608b5472623cf64ce8fe88dedd3018a6d4aeb2f0ee99c0627ba54d8480a"} Nov 03 02:17:16.285343 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:16.285207 5605 scope.go:122] "RemoveContainer" containerID="87835394c46ddbe897479e341429a5b51ea93bdb5b2cf2311c2b0e052ed5dad9" Nov 03 02:17:16.302573 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:16.302509 5605 scope.go:122] "RemoveContainer" containerID="87835394c46ddbe897479e341429a5b51ea93bdb5b2cf2311c2b0e052ed5dad9" Nov 03 02:17:16.302886 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:17:16.302868 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87835394c46ddbe897479e341429a5b51ea93bdb5b2cf2311c2b0e052ed5dad9\": container with ID starting with 87835394c46ddbe897479e341429a5b51ea93bdb5b2cf2311c2b0e052ed5dad9 not found: ID does not exist" containerID="87835394c46ddbe897479e341429a5b51ea93bdb5b2cf2311c2b0e052ed5dad9" Nov 03 02:17:16.302928 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:16.302895 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87835394c46ddbe897479e341429a5b51ea93bdb5b2cf2311c2b0e052ed5dad9"} err="failed to get container status \"87835394c46ddbe897479e341429a5b51ea93bdb5b2cf2311c2b0e052ed5dad9\": rpc error: code = NotFound desc = could not find container \"87835394c46ddbe897479e341429a5b51ea93bdb5b2cf2311c2b0e052ed5dad9\": container with ID starting with 87835394c46ddbe897479e341429a5b51ea93bdb5b2cf2311c2b0e052ed5dad9 not found: ID does not exist" Nov 03 02:17:16.438807 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:16.438774 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a772c3-6cc589fd8c-pf6hh"] Nov 03 02:17:16.448669 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:16.448634 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a772c3-6cc589fd8c-pf6hh"] Nov 03 02:17:16.555615 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:16.555585 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d" Nov 03 02:17:16.556823 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:16.556800 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d" Nov 03 02:17:16.851400 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:16.851311 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e782231-a2f8-45d9-8507-fcc038f1a368" path="/var/lib/kubelet/pods/3e782231-a2f8-45d9-8507-fcc038f1a368/volumes" Nov 03 02:17:23.914616 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:17:23.914578 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:17:35.347452 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:35.347421 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c"] Nov 03 02:17:35.347945 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:35.347621 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c" podUID="69d57c7a-8495-4d67-9547-dfe082939941" containerName="istio-proxy" containerID="cri-o://25d644dfb50b5d35b22b71078dfa66be4afcc07b728aa0db3ca3de6bc774b4e3" gracePeriod=30 Nov 03 02:17:40.552994 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:40.552961 5605 generic.go:356] "Generic (PLEG): container finished" podID="69d57c7a-8495-4d67-9547-dfe082939941" containerID="25d644dfb50b5d35b22b71078dfa66be4afcc07b728aa0db3ca3de6bc774b4e3" exitCode=0 Nov 03 02:17:40.553537 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:40.553029 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c" event={"ID":"69d57c7a-8495-4d67-9547-dfe082939941","Type":"ContainerDied","Data":"25d644dfb50b5d35b22b71078dfa66be4afcc07b728aa0db3ca3de6bc774b4e3"} Nov 03 02:17:40.635771 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:40.635744 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c" Nov 03 02:17:40.654622 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:40.654595 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69d57c7a-8495-4d67-9547-dfe082939941-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69d57c7a-8495-4d67-9547-dfe082939941-istio-envoy\") pod \"69d57c7a-8495-4d67-9547-dfe082939941\" (UID: \"69d57c7a-8495-4d67-9547-dfe082939941\") " Nov 03 02:17:40.654734 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:40.654633 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69d57c7a-8495-4d67-9547-dfe082939941-istio-token\" (UniqueName: \"kubernetes.io/projected/69d57c7a-8495-4d67-9547-dfe082939941-istio-token\") pod \"69d57c7a-8495-4d67-9547-dfe082939941\" (UID: \"69d57c7a-8495-4d67-9547-dfe082939941\") " Nov 03 02:17:40.654734 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:40.654663 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69d57c7a-8495-4d67-9547-dfe082939941-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69d57c7a-8495-4d67-9547-dfe082939941-workload-socket\") pod \"69d57c7a-8495-4d67-9547-dfe082939941\" (UID: \"69d57c7a-8495-4d67-9547-dfe082939941\") " Nov 03 02:17:40.654734 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:40.654711 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69d57c7a-8495-4d67-9547-dfe082939941-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69d57c7a-8495-4d67-9547-dfe082939941-credential-socket\") pod \"69d57c7a-8495-4d67-9547-dfe082939941\" (UID: \"69d57c7a-8495-4d67-9547-dfe082939941\") " Nov 03 02:17:40.654913 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:40.654838 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69d57c7a-8495-4d67-9547-dfe082939941-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69d57c7a-8495-4d67-9547-dfe082939941-istio-data\") pod \"69d57c7a-8495-4d67-9547-dfe082939941\" (UID: \"69d57c7a-8495-4d67-9547-dfe082939941\") " Nov 03 02:17:40.654974 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:40.654954 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69d57c7a-8495-4d67-9547-dfe082939941-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69d57c7a-8495-4d67-9547-dfe082939941-workload-certs\") pod \"69d57c7a-8495-4d67-9547-dfe082939941\" (UID: \"69d57c7a-8495-4d67-9547-dfe082939941\") " Nov 03 02:17:40.655034 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:40.654964 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69d57c7a-8495-4d67-9547-dfe082939941-credential-socket" pod "69d57c7a-8495-4d67-9547-dfe082939941" (UID: "69d57c7a-8495-4d67-9547-dfe082939941"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:17:40.655034 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:40.654975 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69d57c7a-8495-4d67-9547-dfe082939941-workload-socket" pod "69d57c7a-8495-4d67-9547-dfe082939941" (UID: "69d57c7a-8495-4d67-9547-dfe082939941"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:17:40.655034 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:40.655015 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69d57c7a-8495-4d67-9547-dfe082939941-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69d57c7a-8495-4d67-9547-dfe082939941-istio-podinfo\") pod \"69d57c7a-8495-4d67-9547-dfe082939941\" (UID: \"69d57c7a-8495-4d67-9547-dfe082939941\") " Nov 03 02:17:40.655217 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:40.655079 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69d57c7a-8495-4d67-9547-dfe082939941-istio-data" pod "69d57c7a-8495-4d67-9547-dfe082939941" (UID: "69d57c7a-8495-4d67-9547-dfe082939941"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:17:40.655217 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:40.655105 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69d57c7a-8495-4d67-9547-dfe082939941-kube-api-access-q7tcj\" (UniqueName: \"kubernetes.io/projected/69d57c7a-8495-4d67-9547-dfe082939941-kube-api-access-q7tcj\") pod \"69d57c7a-8495-4d67-9547-dfe082939941\" (UID: \"69d57c7a-8495-4d67-9547-dfe082939941\") " Nov 03 02:17:40.655217 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:40.655185 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69d57c7a-8495-4d67-9547-dfe082939941-workload-certs" pod "69d57c7a-8495-4d67-9547-dfe082939941" (UID: "69d57c7a-8495-4d67-9547-dfe082939941"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:17:40.655217 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:40.655201 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69d57c7a-8495-4d67-9547-dfe082939941-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69d57c7a-8495-4d67-9547-dfe082939941-istiod-ca-cert\") pod \"69d57c7a-8495-4d67-9547-dfe082939941\" (UID: \"69d57c7a-8495-4d67-9547-dfe082939941\") " Nov 03 02:17:40.655586 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:40.655562 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69d57c7a-8495-4d67-9547-dfe082939941-istiod-ca-cert" pod "69d57c7a-8495-4d67-9547-dfe082939941" (UID: "69d57c7a-8495-4d67-9547-dfe082939941"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:17:40.655859 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:40.655837 5605 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69d57c7a-8495-4d67-9547-dfe082939941-workload-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:40.655936 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:40.655865 5605 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69d57c7a-8495-4d67-9547-dfe082939941-credential-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:40.655936 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:40.655883 5605 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69d57c7a-8495-4d67-9547-dfe082939941-istio-data\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:40.655936 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:40.655898 5605 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69d57c7a-8495-4d67-9547-dfe082939941-workload-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:40.655936 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:40.655913 5605 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69d57c7a-8495-4d67-9547-dfe082939941-istiod-ca-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:40.656894 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:40.656872 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69d57c7a-8495-4d67-9547-dfe082939941-istio-envoy" pod "69d57c7a-8495-4d67-9547-dfe082939941" (UID: "69d57c7a-8495-4d67-9547-dfe082939941"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:17:40.657084 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:40.657058 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69d57c7a-8495-4d67-9547-dfe082939941-istio-token" pod "69d57c7a-8495-4d67-9547-dfe082939941" (UID: "69d57c7a-8495-4d67-9547-dfe082939941"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:17:40.657182 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:40.657107 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69d57c7a-8495-4d67-9547-dfe082939941-istio-podinfo" pod "69d57c7a-8495-4d67-9547-dfe082939941" (UID: "69d57c7a-8495-4d67-9547-dfe082939941"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:17:40.657182 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:40.657162 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69d57c7a-8495-4d67-9547-dfe082939941-kube-api-access-q7tcj" pod "69d57c7a-8495-4d67-9547-dfe082939941" (UID: "69d57c7a-8495-4d67-9547-dfe082939941"). InnerVolumeSpecName "kube-api-access-q7tcj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:17:40.757539 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:40.757504 5605 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69d57c7a-8495-4d67-9547-dfe082939941-istio-podinfo\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:40.757539 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:40.757535 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7tcj\" (UniqueName: \"kubernetes.io/projected/69d57c7a-8495-4d67-9547-dfe082939941-kube-api-access-q7tcj\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:40.757713 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:40.757549 5605 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69d57c7a-8495-4d67-9547-dfe082939941-istio-envoy\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:40.757713 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:40.757563 5605 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69d57c7a-8495-4d67-9547-dfe082939941-istio-token\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:41.560511 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:41.560480 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c" Nov 03 02:17:41.560960 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:41.560469 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c" event={"ID":"69d57c7a-8495-4d67-9547-dfe082939941","Type":"ContainerDied","Data":"60663abcbc3c844e4d2b9238f153c68783706a6152bce58a0c40b834f95995fd"} Nov 03 02:17:41.560960 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:41.560613 5605 scope.go:122] "RemoveContainer" containerID="25d644dfb50b5d35b22b71078dfa66be4afcc07b728aa0db3ca3de6bc774b4e3" Nov 03 02:17:41.775460 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:41.775376 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c"] Nov 03 02:17:41.783507 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:41.783417 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-cft4c"] Nov 03 02:17:42.850889 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:42.850855 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69d57c7a-8495-4d67-9547-dfe082939941" path="/var/lib/kubelet/pods/69d57c7a-8495-4d67-9547-dfe082939941/volumes" Nov 03 02:17:43.240628 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:43.240547 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:17:46.051427 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:46.051386 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3632597-bbsbx"] Nov 03 02:17:46.096881 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:46.096849 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3632597-bbsbx"] Nov 03 02:17:46.307346 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:46.307264 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8"] Nov 03 02:17:46.307844 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:46.307803 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8" podUID="87ac6502-78a2-41ed-9033-77eecf21086f" containerName="istio-proxy" containerID="cri-o://d44aa5007f19f3bd09d319f9ae1d357d1c45730d4d6f5ac24750271d337ac1fa" gracePeriod=30 Nov 03 02:17:46.849972 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:46.849940 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="842dabe0-fc9a-4a1f-a923-ad77169f3ae9" path="/var/lib/kubelet/pods/842dabe0-fc9a-4a1f-a923-ad77169f3ae9/volumes" Nov 03 02:17:51.577529 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:51.577468 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8" Nov 03 02:17:51.636191 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:51.635113 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87ac6502-78a2-41ed-9033-77eecf21086f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87ac6502-78a2-41ed-9033-77eecf21086f-credential-socket\") pod \"87ac6502-78a2-41ed-9033-77eecf21086f\" (UID: \"87ac6502-78a2-41ed-9033-77eecf21086f\") " Nov 03 02:17:51.636191 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:51.635160 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87ac6502-78a2-41ed-9033-77eecf21086f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87ac6502-78a2-41ed-9033-77eecf21086f-istiod-ca-cert\") pod \"87ac6502-78a2-41ed-9033-77eecf21086f\" (UID: \"87ac6502-78a2-41ed-9033-77eecf21086f\") " Nov 03 02:17:51.636191 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:51.635207 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87ac6502-78a2-41ed-9033-77eecf21086f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87ac6502-78a2-41ed-9033-77eecf21086f-istio-data\") pod \"87ac6502-78a2-41ed-9033-77eecf21086f\" (UID: \"87ac6502-78a2-41ed-9033-77eecf21086f\") " Nov 03 02:17:51.636191 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:51.635261 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87ac6502-78a2-41ed-9033-77eecf21086f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87ac6502-78a2-41ed-9033-77eecf21086f-istio-envoy\") pod \"87ac6502-78a2-41ed-9033-77eecf21086f\" (UID: \"87ac6502-78a2-41ed-9033-77eecf21086f\") " Nov 03 02:17:51.636191 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:51.635414 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87ac6502-78a2-41ed-9033-77eecf21086f-istio-token\" (UniqueName: \"kubernetes.io/projected/87ac6502-78a2-41ed-9033-77eecf21086f-istio-token\") pod \"87ac6502-78a2-41ed-9033-77eecf21086f\" (UID: \"87ac6502-78a2-41ed-9033-77eecf21086f\") " Nov 03 02:17:51.636191 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:51.635443 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87ac6502-78a2-41ed-9033-77eecf21086f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87ac6502-78a2-41ed-9033-77eecf21086f-workload-socket\") pod \"87ac6502-78a2-41ed-9033-77eecf21086f\" (UID: \"87ac6502-78a2-41ed-9033-77eecf21086f\") " Nov 03 02:17:51.636191 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:51.635510 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/87ac6502-78a2-41ed-9033-77eecf21086f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87ac6502-78a2-41ed-9033-77eecf21086f-istio-podinfo\") pod \"87ac6502-78a2-41ed-9033-77eecf21086f\" (UID: \"87ac6502-78a2-41ed-9033-77eecf21086f\") " Nov 03 02:17:51.636191 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:51.635573 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87ac6502-78a2-41ed-9033-77eecf21086f-kube-api-access-l4559\" (UniqueName: \"kubernetes.io/projected/87ac6502-78a2-41ed-9033-77eecf21086f-kube-api-access-l4559\") pod \"87ac6502-78a2-41ed-9033-77eecf21086f\" (UID: \"87ac6502-78a2-41ed-9033-77eecf21086f\") " Nov 03 02:17:51.636191 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:51.635630 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87ac6502-78a2-41ed-9033-77eecf21086f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87ac6502-78a2-41ed-9033-77eecf21086f-workload-certs\") pod \"87ac6502-78a2-41ed-9033-77eecf21086f\" (UID: \"87ac6502-78a2-41ed-9033-77eecf21086f\") " Nov 03 02:17:51.636575 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:51.636527 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87ac6502-78a2-41ed-9033-77eecf21086f-credential-socket" pod "87ac6502-78a2-41ed-9033-77eecf21086f" (UID: "87ac6502-78a2-41ed-9033-77eecf21086f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:17:51.636676 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:51.636650 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87ac6502-78a2-41ed-9033-77eecf21086f-workload-certs" pod "87ac6502-78a2-41ed-9033-77eecf21086f" (UID: "87ac6502-78a2-41ed-9033-77eecf21086f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:17:51.636768 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:51.636751 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87ac6502-78a2-41ed-9033-77eecf21086f-istiod-ca-cert" pod "87ac6502-78a2-41ed-9033-77eecf21086f" (UID: "87ac6502-78a2-41ed-9033-77eecf21086f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:17:51.636958 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:51.636940 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87ac6502-78a2-41ed-9033-77eecf21086f-workload-socket" pod "87ac6502-78a2-41ed-9033-77eecf21086f" (UID: "87ac6502-78a2-41ed-9033-77eecf21086f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:17:51.637260 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:51.637225 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87ac6502-78a2-41ed-9033-77eecf21086f-istio-data" pod "87ac6502-78a2-41ed-9033-77eecf21086f" (UID: "87ac6502-78a2-41ed-9033-77eecf21086f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:17:51.639312 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:51.639288 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/87ac6502-78a2-41ed-9033-77eecf21086f-istio-podinfo" pod "87ac6502-78a2-41ed-9033-77eecf21086f" (UID: "87ac6502-78a2-41ed-9033-77eecf21086f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:17:51.639464 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:51.639306 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87ac6502-78a2-41ed-9033-77eecf21086f-kube-api-access-l4559" pod "87ac6502-78a2-41ed-9033-77eecf21086f" (UID: "87ac6502-78a2-41ed-9033-77eecf21086f"). InnerVolumeSpecName "kube-api-access-l4559". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:17:51.639464 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:51.639323 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87ac6502-78a2-41ed-9033-77eecf21086f-istio-token" pod "87ac6502-78a2-41ed-9033-77eecf21086f" (UID: "87ac6502-78a2-41ed-9033-77eecf21086f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:17:51.640476 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:51.640454 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87ac6502-78a2-41ed-9033-77eecf21086f-istio-envoy" pod "87ac6502-78a2-41ed-9033-77eecf21086f" (UID: "87ac6502-78a2-41ed-9033-77eecf21086f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:17:51.652647 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:51.652623 5605 generic.go:356] "Generic (PLEG): container finished" podID="87ac6502-78a2-41ed-9033-77eecf21086f" containerID="d44aa5007f19f3bd09d319f9ae1d357d1c45730d4d6f5ac24750271d337ac1fa" exitCode=0 Nov 03 02:17:51.652742 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:51.652666 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8" event={"ID":"87ac6502-78a2-41ed-9033-77eecf21086f","Type":"ContainerDied","Data":"d44aa5007f19f3bd09d319f9ae1d357d1c45730d4d6f5ac24750271d337ac1fa"} Nov 03 02:17:51.652742 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:51.652725 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8" event={"ID":"87ac6502-78a2-41ed-9033-77eecf21086f","Type":"ContainerDied","Data":"8e2b810cfdc89c37625989c2e6aa943702902b51a648bbff6f21b66694017edf"} Nov 03 02:17:51.652811 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:51.652746 5605 scope.go:122] "RemoveContainer" containerID="d44aa5007f19f3bd09d319f9ae1d357d1c45730d4d6f5ac24750271d337ac1fa" Nov 03 02:17:51.652870 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:51.652860 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8" Nov 03 02:17:51.665798 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:51.665781 5605 scope.go:122] "RemoveContainer" containerID="80fa1b08040a5dd9081261734bb32061db14746eb9c9c04e4d5e91f90881f114" Nov 03 02:17:51.674090 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:51.674072 5605 scope.go:122] "RemoveContainer" containerID="d44aa5007f19f3bd09d319f9ae1d357d1c45730d4d6f5ac24750271d337ac1fa" Nov 03 02:17:51.674334 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:17:51.674317 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d44aa5007f19f3bd09d319f9ae1d357d1c45730d4d6f5ac24750271d337ac1fa\": container with ID starting with d44aa5007f19f3bd09d319f9ae1d357d1c45730d4d6f5ac24750271d337ac1fa not found: ID does not exist" containerID="d44aa5007f19f3bd09d319f9ae1d357d1c45730d4d6f5ac24750271d337ac1fa" Nov 03 02:17:51.674408 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:51.674340 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d44aa5007f19f3bd09d319f9ae1d357d1c45730d4d6f5ac24750271d337ac1fa"} err="failed to get container status \"d44aa5007f19f3bd09d319f9ae1d357d1c45730d4d6f5ac24750271d337ac1fa\": rpc error: code = NotFound desc = could not find container \"d44aa5007f19f3bd09d319f9ae1d357d1c45730d4d6f5ac24750271d337ac1fa\": container with ID starting with d44aa5007f19f3bd09d319f9ae1d357d1c45730d4d6f5ac24750271d337ac1fa not found: ID does not exist" Nov 03 02:17:51.674408 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:51.674370 5605 scope.go:122] "RemoveContainer" containerID="80fa1b08040a5dd9081261734bb32061db14746eb9c9c04e4d5e91f90881f114" Nov 03 02:17:51.674616 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:17:51.674598 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80fa1b08040a5dd9081261734bb32061db14746eb9c9c04e4d5e91f90881f114\": container with ID starting with 80fa1b08040a5dd9081261734bb32061db14746eb9c9c04e4d5e91f90881f114 not found: ID does not exist" containerID="80fa1b08040a5dd9081261734bb32061db14746eb9c9c04e4d5e91f90881f114" Nov 03 02:17:51.674660 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:51.674621 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80fa1b08040a5dd9081261734bb32061db14746eb9c9c04e4d5e91f90881f114"} err="failed to get container status \"80fa1b08040a5dd9081261734bb32061db14746eb9c9c04e4d5e91f90881f114\": rpc error: code = NotFound desc = could not find container \"80fa1b08040a5dd9081261734bb32061db14746eb9c9c04e4d5e91f90881f114\": container with ID starting with 80fa1b08040a5dd9081261734bb32061db14746eb9c9c04e4d5e91f90881f114 not found: ID does not exist" Nov 03 02:17:51.736782 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:51.736758 5605 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87ac6502-78a2-41ed-9033-77eecf21086f-workload-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:51.736782 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:51.736780 5605 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87ac6502-78a2-41ed-9033-77eecf21086f-istio-podinfo\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:51.736900 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:51.736790 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4559\" (UniqueName: \"kubernetes.io/projected/87ac6502-78a2-41ed-9033-77eecf21086f-kube-api-access-l4559\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:51.736900 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:51.736799 5605 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87ac6502-78a2-41ed-9033-77eecf21086f-workload-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:51.736900 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:51.736807 5605 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87ac6502-78a2-41ed-9033-77eecf21086f-credential-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:51.736900 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:51.736815 5605 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87ac6502-78a2-41ed-9033-77eecf21086f-istiod-ca-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:51.736900 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:51.736823 5605 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87ac6502-78a2-41ed-9033-77eecf21086f-istio-data\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:51.736900 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:51.736831 5605 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87ac6502-78a2-41ed-9033-77eecf21086f-istio-envoy\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:51.736900 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:51.736838 5605 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87ac6502-78a2-41ed-9033-77eecf21086f-istio-token\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:51.839158 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:51.839079 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8"] Nov 03 02:17:51.843742 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:51.843708 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-c617de-openshift-default-6cb94bc959-gsww8"] Nov 03 02:17:52.851613 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:17:52.851580 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87ac6502-78a2-41ed-9033-77eecf21086f" path="/var/lib/kubelet/pods/87ac6502-78a2-41ed-9033-77eecf21086f/volumes" Nov 03 02:17:53.914886 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:17:53.914851 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:18:03.283674 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:03.283641 5605 scope.go:122] "RemoveContainer" containerID="f0ecdde3cf90200cbdd8c92f35242e20f4c2ea8210e94fbf46ca024b5d2c91ff" Nov 03 02:18:03.293433 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:03.293409 5605 scope.go:122] "RemoveContainer" containerID="e7bde04ea44ea5f394d0662d3e1d5e4501e0672bb2c74557b5937265caa6642c" Nov 03 02:18:04.044582 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:04.044550 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbceb3f9-kvzfx"] Nov 03 02:18:04.090942 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:04.090916 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbceb3f9-kvzfx"] Nov 03 02:18:04.851340 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:04.851305 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d3aea59-b051-4377-8108-73a7d958922f" path="/var/lib/kubelet/pods/0d3aea59-b051-4377-8108-73a7d958922f/volumes" Nov 03 02:18:22.369098 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:22.369063 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:18:23.943288 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:18:23.943241 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:18:26.044445 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:26.044398 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54abb9d6-5tzw2"] Nov 03 02:18:26.090818 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:26.090781 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54abb9d6-5tzw2"] Nov 03 02:18:26.850699 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:26.850661 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a60fd0a2-31e0-4549-98d9-74c013efb7cf" path="/var/lib/kubelet/pods/a60fd0a2-31e0-4549-98d9-74c013efb7cf/volumes" Nov 03 02:18:30.997475 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:30.997438 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n"] Nov 03 02:18:30.997897 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:30.997618 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n" podUID="58ecd8f7-099d-401b-a63a-6c4e79e56077" containerName="istio-proxy" containerID="cri-o://d37cf73f92722c2c7fc104b8c1c6285b5c68634bc03028bfbe49f62143b4694a" gracePeriod=30 Nov 03 02:18:36.265550 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:36.265488 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n" Nov 03 02:18:36.346722 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:36.346692 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/58ecd8f7-099d-401b-a63a-6c4e79e56077-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58ecd8f7-099d-401b-a63a-6c4e79e56077-istio-podinfo\") pod \"58ecd8f7-099d-401b-a63a-6c4e79e56077\" (UID: \"58ecd8f7-099d-401b-a63a-6c4e79e56077\") " Nov 03 02:18:36.346903 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:36.346742 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58ecd8f7-099d-401b-a63a-6c4e79e56077-istio-token\" (UniqueName: \"kubernetes.io/projected/58ecd8f7-099d-401b-a63a-6c4e79e56077-istio-token\") pod \"58ecd8f7-099d-401b-a63a-6c4e79e56077\" (UID: \"58ecd8f7-099d-401b-a63a-6c4e79e56077\") " Nov 03 02:18:36.346903 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:36.346767 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58ecd8f7-099d-401b-a63a-6c4e79e56077-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58ecd8f7-099d-401b-a63a-6c4e79e56077-istio-data\") pod \"58ecd8f7-099d-401b-a63a-6c4e79e56077\" (UID: \"58ecd8f7-099d-401b-a63a-6c4e79e56077\") " Nov 03 02:18:36.346903 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:36.346876 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58ecd8f7-099d-401b-a63a-6c4e79e56077-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58ecd8f7-099d-401b-a63a-6c4e79e56077-istio-envoy\") pod \"58ecd8f7-099d-401b-a63a-6c4e79e56077\" (UID: \"58ecd8f7-099d-401b-a63a-6c4e79e56077\") " Nov 03 02:18:36.347053 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:36.346912 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58ecd8f7-099d-401b-a63a-6c4e79e56077-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58ecd8f7-099d-401b-a63a-6c4e79e56077-workload-certs\") pod \"58ecd8f7-099d-401b-a63a-6c4e79e56077\" (UID: \"58ecd8f7-099d-401b-a63a-6c4e79e56077\") " Nov 03 02:18:36.347053 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:36.346943 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58ecd8f7-099d-401b-a63a-6c4e79e56077-kube-api-access-kjzg5\" (UniqueName: \"kubernetes.io/projected/58ecd8f7-099d-401b-a63a-6c4e79e56077-kube-api-access-kjzg5\") pod \"58ecd8f7-099d-401b-a63a-6c4e79e56077\" (UID: \"58ecd8f7-099d-401b-a63a-6c4e79e56077\") " Nov 03 02:18:36.347053 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:36.347036 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58ecd8f7-099d-401b-a63a-6c4e79e56077-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ecd8f7-099d-401b-a63a-6c4e79e56077-credential-socket\") pod \"58ecd8f7-099d-401b-a63a-6c4e79e56077\" (UID: \"58ecd8f7-099d-401b-a63a-6c4e79e56077\") " Nov 03 02:18:36.347217 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:36.347093 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58ecd8f7-099d-401b-a63a-6c4e79e56077-istio-data" pod "58ecd8f7-099d-401b-a63a-6c4e79e56077" (UID: "58ecd8f7-099d-401b-a63a-6c4e79e56077"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:18:36.347217 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:36.347122 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58ecd8f7-099d-401b-a63a-6c4e79e56077-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58ecd8f7-099d-401b-a63a-6c4e79e56077-istiod-ca-cert\") pod \"58ecd8f7-099d-401b-a63a-6c4e79e56077\" (UID: \"58ecd8f7-099d-401b-a63a-6c4e79e56077\") " Nov 03 02:18:36.347217 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:36.347175 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58ecd8f7-099d-401b-a63a-6c4e79e56077-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ecd8f7-099d-401b-a63a-6c4e79e56077-workload-socket\") pod \"58ecd8f7-099d-401b-a63a-6c4e79e56077\" (UID: \"58ecd8f7-099d-401b-a63a-6c4e79e56077\") " Nov 03 02:18:36.347387 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:36.347221 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58ecd8f7-099d-401b-a63a-6c4e79e56077-workload-certs" pod "58ecd8f7-099d-401b-a63a-6c4e79e56077" (UID: "58ecd8f7-099d-401b-a63a-6c4e79e56077"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:18:36.347387 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:36.347304 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58ecd8f7-099d-401b-a63a-6c4e79e56077-credential-socket" pod "58ecd8f7-099d-401b-a63a-6c4e79e56077" (UID: "58ecd8f7-099d-401b-a63a-6c4e79e56077"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:18:36.347588 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:36.347550 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58ecd8f7-099d-401b-a63a-6c4e79e56077-workload-socket" pod "58ecd8f7-099d-401b-a63a-6c4e79e56077" (UID: "58ecd8f7-099d-401b-a63a-6c4e79e56077"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:18:36.347668 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:36.347599 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58ecd8f7-099d-401b-a63a-6c4e79e56077-istiod-ca-cert" pod "58ecd8f7-099d-401b-a63a-6c4e79e56077" (UID: "58ecd8f7-099d-401b-a63a-6c4e79e56077"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:18:36.347822 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:36.347802 5605 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ecd8f7-099d-401b-a63a-6c4e79e56077-credential-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:18:36.347880 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:36.347828 5605 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58ecd8f7-099d-401b-a63a-6c4e79e56077-istiod-ca-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:18:36.347880 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:36.347843 5605 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ecd8f7-099d-401b-a63a-6c4e79e56077-workload-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:18:36.347880 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:36.347857 5605 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58ecd8f7-099d-401b-a63a-6c4e79e56077-istio-data\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:18:36.347880 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:36.347871 5605 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58ecd8f7-099d-401b-a63a-6c4e79e56077-workload-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:18:36.349160 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:36.349132 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58ecd8f7-099d-401b-a63a-6c4e79e56077-istio-token" pod "58ecd8f7-099d-401b-a63a-6c4e79e56077" (UID: "58ecd8f7-099d-401b-a63a-6c4e79e56077"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:18:36.349477 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:36.349453 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58ecd8f7-099d-401b-a63a-6c4e79e56077-istio-envoy" pod "58ecd8f7-099d-401b-a63a-6c4e79e56077" (UID: "58ecd8f7-099d-401b-a63a-6c4e79e56077"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:18:36.349477 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:36.349459 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58ecd8f7-099d-401b-a63a-6c4e79e56077-kube-api-access-kjzg5" pod "58ecd8f7-099d-401b-a63a-6c4e79e56077" (UID: "58ecd8f7-099d-401b-a63a-6c4e79e56077"). InnerVolumeSpecName "kube-api-access-kjzg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:18:36.349618 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:36.349496 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58ecd8f7-099d-401b-a63a-6c4e79e56077-istio-podinfo" pod "58ecd8f7-099d-401b-a63a-6c4e79e56077" (UID: "58ecd8f7-099d-401b-a63a-6c4e79e56077"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:18:36.449398 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:36.449347 5605 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58ecd8f7-099d-401b-a63a-6c4e79e56077-istio-envoy\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:18:36.449594 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:36.449402 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjzg5\" (UniqueName: \"kubernetes.io/projected/58ecd8f7-099d-401b-a63a-6c4e79e56077-kube-api-access-kjzg5\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:18:36.449594 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:36.449420 5605 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58ecd8f7-099d-401b-a63a-6c4e79e56077-istio-podinfo\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:18:36.449594 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:36.449437 5605 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58ecd8f7-099d-401b-a63a-6c4e79e56077-istio-token\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:18:37.030805 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:37.030762 5605 generic.go:356] "Generic (PLEG): container finished" podID="58ecd8f7-099d-401b-a63a-6c4e79e56077" containerID="d37cf73f92722c2c7fc104b8c1c6285b5c68634bc03028bfbe49f62143b4694a" exitCode=0 Nov 03 02:18:37.030966 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:37.030814 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n" Nov 03 02:18:37.030966 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:37.030831 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n" event={"ID":"58ecd8f7-099d-401b-a63a-6c4e79e56077","Type":"ContainerDied","Data":"d37cf73f92722c2c7fc104b8c1c6285b5c68634bc03028bfbe49f62143b4694a"} Nov 03 02:18:37.030966 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:37.030900 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n" event={"ID":"58ecd8f7-099d-401b-a63a-6c4e79e56077","Type":"ContainerDied","Data":"13c28f50395a97493176a5f1d431b2331336a73b74c262cabad0f841218c3bf8"} Nov 03 02:18:37.030966 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:37.030924 5605 scope.go:122] "RemoveContainer" containerID="d37cf73f92722c2c7fc104b8c1c6285b5c68634bc03028bfbe49f62143b4694a" Nov 03 02:18:37.040613 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:37.040596 5605 scope.go:122] "RemoveContainer" containerID="d37cf73f92722c2c7fc104b8c1c6285b5c68634bc03028bfbe49f62143b4694a" Nov 03 02:18:37.040884 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:18:37.040866 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d37cf73f92722c2c7fc104b8c1c6285b5c68634bc03028bfbe49f62143b4694a\": container with ID starting with d37cf73f92722c2c7fc104b8c1c6285b5c68634bc03028bfbe49f62143b4694a not found: ID does not exist" containerID="d37cf73f92722c2c7fc104b8c1c6285b5c68634bc03028bfbe49f62143b4694a" Nov 03 02:18:37.040956 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:37.040897 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d37cf73f92722c2c7fc104b8c1c6285b5c68634bc03028bfbe49f62143b4694a"} err="failed to get container status \"d37cf73f92722c2c7fc104b8c1c6285b5c68634bc03028bfbe49f62143b4694a\": rpc error: code = NotFound desc = could not find container \"d37cf73f92722c2c7fc104b8c1c6285b5c68634bc03028bfbe49f62143b4694a\": container with ID starting with d37cf73f92722c2c7fc104b8c1c6285b5c68634bc03028bfbe49f62143b4694a not found: ID does not exist" Nov 03 02:18:37.231747 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:37.231672 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n"] Nov 03 02:18:37.235125 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:37.235102 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-jft7n"] Nov 03 02:18:37.359511 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:37.358979 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d"] Nov 03 02:18:37.360303 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:37.360252 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d" podUID="2225e2b8-fca7-4a8a-8c47-a687a0d9b214" containerName="istio-proxy" containerID="cri-o://94cab9770b825e5812b835012f8f16b0b99c81f425d477eaf4ce513cf9960c6d" gracePeriod=30 Nov 03 02:18:38.854266 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:38.854226 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58ecd8f7-099d-401b-a63a-6c4e79e56077" path="/var/lib/kubelet/pods/58ecd8f7-099d-401b-a63a-6c4e79e56077/volumes" Nov 03 02:18:42.735380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:42.735341 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d" Nov 03 02:18:42.872388 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:42.866392 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2225e2b8-fca7-4a8a-8c47-a687a0d9b214-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2225e2b8-fca7-4a8a-8c47-a687a0d9b214-credential-socket\") pod \"2225e2b8-fca7-4a8a-8c47-a687a0d9b214\" (UID: \"2225e2b8-fca7-4a8a-8c47-a687a0d9b214\") " Nov 03 02:18:42.872388 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:42.866532 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2225e2b8-fca7-4a8a-8c47-a687a0d9b214-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2225e2b8-fca7-4a8a-8c47-a687a0d9b214-istiod-ca-cert\") pod \"2225e2b8-fca7-4a8a-8c47-a687a0d9b214\" (UID: \"2225e2b8-fca7-4a8a-8c47-a687a0d9b214\") " Nov 03 02:18:42.872388 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:42.866626 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2225e2b8-fca7-4a8a-8c47-a687a0d9b214-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2225e2b8-fca7-4a8a-8c47-a687a0d9b214-workload-socket\") pod \"2225e2b8-fca7-4a8a-8c47-a687a0d9b214\" (UID: \"2225e2b8-fca7-4a8a-8c47-a687a0d9b214\") " Nov 03 02:18:42.872388 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:42.866693 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2225e2b8-fca7-4a8a-8c47-a687a0d9b214-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2225e2b8-fca7-4a8a-8c47-a687a0d9b214-istio-data\") pod \"2225e2b8-fca7-4a8a-8c47-a687a0d9b214\" (UID: \"2225e2b8-fca7-4a8a-8c47-a687a0d9b214\") " Nov 03 02:18:42.872388 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:42.866726 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2225e2b8-fca7-4a8a-8c47-a687a0d9b214-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2225e2b8-fca7-4a8a-8c47-a687a0d9b214-istio-podinfo\") pod \"2225e2b8-fca7-4a8a-8c47-a687a0d9b214\" (UID: \"2225e2b8-fca7-4a8a-8c47-a687a0d9b214\") " Nov 03 02:18:42.872388 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:42.866829 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2225e2b8-fca7-4a8a-8c47-a687a0d9b214-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2225e2b8-fca7-4a8a-8c47-a687a0d9b214-workload-certs\") pod \"2225e2b8-fca7-4a8a-8c47-a687a0d9b214\" (UID: \"2225e2b8-fca7-4a8a-8c47-a687a0d9b214\") " Nov 03 02:18:42.872388 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:42.866952 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2225e2b8-fca7-4a8a-8c47-a687a0d9b214-istio-token\" (UniqueName: \"kubernetes.io/projected/2225e2b8-fca7-4a8a-8c47-a687a0d9b214-istio-token\") pod \"2225e2b8-fca7-4a8a-8c47-a687a0d9b214\" (UID: \"2225e2b8-fca7-4a8a-8c47-a687a0d9b214\") " Nov 03 02:18:42.872388 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:42.867020 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2225e2b8-fca7-4a8a-8c47-a687a0d9b214-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2225e2b8-fca7-4a8a-8c47-a687a0d9b214-istio-envoy\") pod \"2225e2b8-fca7-4a8a-8c47-a687a0d9b214\" (UID: \"2225e2b8-fca7-4a8a-8c47-a687a0d9b214\") " Nov 03 02:18:42.872388 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:42.867081 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2225e2b8-fca7-4a8a-8c47-a687a0d9b214-kube-api-access-sc2k7\" (UniqueName: \"kubernetes.io/projected/2225e2b8-fca7-4a8a-8c47-a687a0d9b214-kube-api-access-sc2k7\") pod \"2225e2b8-fca7-4a8a-8c47-a687a0d9b214\" (UID: \"2225e2b8-fca7-4a8a-8c47-a687a0d9b214\") " Nov 03 02:18:42.921853 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:42.908686 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2225e2b8-fca7-4a8a-8c47-a687a0d9b214-workload-certs" pod "2225e2b8-fca7-4a8a-8c47-a687a0d9b214" (UID: "2225e2b8-fca7-4a8a-8c47-a687a0d9b214"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:18:42.921853 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:42.908769 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2225e2b8-fca7-4a8a-8c47-a687a0d9b214-istio-data" pod "2225e2b8-fca7-4a8a-8c47-a687a0d9b214" (UID: "2225e2b8-fca7-4a8a-8c47-a687a0d9b214"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:18:42.921853 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:42.908911 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2225e2b8-fca7-4a8a-8c47-a687a0d9b214-istio-podinfo" pod "2225e2b8-fca7-4a8a-8c47-a687a0d9b214" (UID: "2225e2b8-fca7-4a8a-8c47-a687a0d9b214"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:18:42.921853 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:42.918618 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2225e2b8-fca7-4a8a-8c47-a687a0d9b214-credential-socket" pod "2225e2b8-fca7-4a8a-8c47-a687a0d9b214" (UID: "2225e2b8-fca7-4a8a-8c47-a687a0d9b214"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:18:42.921853 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:42.918925 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2225e2b8-fca7-4a8a-8c47-a687a0d9b214-istiod-ca-cert" pod "2225e2b8-fca7-4a8a-8c47-a687a0d9b214" (UID: "2225e2b8-fca7-4a8a-8c47-a687a0d9b214"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:18:42.921853 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:42.919202 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2225e2b8-fca7-4a8a-8c47-a687a0d9b214-workload-socket" pod "2225e2b8-fca7-4a8a-8c47-a687a0d9b214" (UID: "2225e2b8-fca7-4a8a-8c47-a687a0d9b214"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:18:42.921853 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:42.921576 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2225e2b8-fca7-4a8a-8c47-a687a0d9b214-kube-api-access-sc2k7" pod "2225e2b8-fca7-4a8a-8c47-a687a0d9b214" (UID: "2225e2b8-fca7-4a8a-8c47-a687a0d9b214"). InnerVolumeSpecName "kube-api-access-sc2k7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:18:42.953894 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:42.953849 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2225e2b8-fca7-4a8a-8c47-a687a0d9b214-istio-token" pod "2225e2b8-fca7-4a8a-8c47-a687a0d9b214" (UID: "2225e2b8-fca7-4a8a-8c47-a687a0d9b214"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:18:42.955208 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:42.955169 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2225e2b8-fca7-4a8a-8c47-a687a0d9b214-istio-envoy" pod "2225e2b8-fca7-4a8a-8c47-a687a0d9b214" (UID: "2225e2b8-fca7-4a8a-8c47-a687a0d9b214"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:18:42.969154 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:42.969092 5605 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2225e2b8-fca7-4a8a-8c47-a687a0d9b214-credential-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:18:42.969154 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:42.969119 5605 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2225e2b8-fca7-4a8a-8c47-a687a0d9b214-istiod-ca-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:18:42.969154 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:42.969129 5605 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2225e2b8-fca7-4a8a-8c47-a687a0d9b214-workload-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:18:42.969154 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:42.969138 5605 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2225e2b8-fca7-4a8a-8c47-a687a0d9b214-istio-data\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:18:42.969154 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:42.969148 5605 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2225e2b8-fca7-4a8a-8c47-a687a0d9b214-istio-podinfo\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:18:42.969154 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:42.969157 5605 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2225e2b8-fca7-4a8a-8c47-a687a0d9b214-workload-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:18:42.969435 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:42.969165 5605 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2225e2b8-fca7-4a8a-8c47-a687a0d9b214-istio-token\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:18:42.969435 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:42.969173 5605 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2225e2b8-fca7-4a8a-8c47-a687a0d9b214-istio-envoy\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:18:42.969435 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:42.969181 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sc2k7\" (UniqueName: \"kubernetes.io/projected/2225e2b8-fca7-4a8a-8c47-a687a0d9b214-kube-api-access-sc2k7\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:18:43.067484 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:43.067448 5605 generic.go:356] "Generic (PLEG): container finished" podID="2225e2b8-fca7-4a8a-8c47-a687a0d9b214" containerID="94cab9770b825e5812b835012f8f16b0b99c81f425d477eaf4ce513cf9960c6d" exitCode=0 Nov 03 02:18:43.067631 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:43.067518 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d" Nov 03 02:18:43.067631 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:43.067534 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d" event={"ID":"2225e2b8-fca7-4a8a-8c47-a687a0d9b214","Type":"ContainerDied","Data":"94cab9770b825e5812b835012f8f16b0b99c81f425d477eaf4ce513cf9960c6d"} Nov 03 02:18:43.067631 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:43.067613 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d" event={"ID":"2225e2b8-fca7-4a8a-8c47-a687a0d9b214","Type":"ContainerDied","Data":"b66a681f5848a92322af904862fdbd1d4cb98049bb4a72c35a5b1ccde2dd43d9"} Nov 03 02:18:43.067753 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:43.067635 5605 scope.go:122] "RemoveContainer" containerID="94cab9770b825e5812b835012f8f16b0b99c81f425d477eaf4ce513cf9960c6d" Nov 03 02:18:43.077630 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:43.077614 5605 scope.go:122] "RemoveContainer" containerID="8938b6aa86b546ebb49211ec1e640193f4f45bf3962d90615893c4f209f900aa" Nov 03 02:18:43.087703 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:43.087685 5605 scope.go:122] "RemoveContainer" containerID="94cab9770b825e5812b835012f8f16b0b99c81f425d477eaf4ce513cf9960c6d" Nov 03 02:18:43.087955 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:18:43.087935 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94cab9770b825e5812b835012f8f16b0b99c81f425d477eaf4ce513cf9960c6d\": container with ID starting with 94cab9770b825e5812b835012f8f16b0b99c81f425d477eaf4ce513cf9960c6d not found: ID does not exist" containerID="94cab9770b825e5812b835012f8f16b0b99c81f425d477eaf4ce513cf9960c6d" Nov 03 02:18:43.088014 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:43.087964 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94cab9770b825e5812b835012f8f16b0b99c81f425d477eaf4ce513cf9960c6d"} err="failed to get container status \"94cab9770b825e5812b835012f8f16b0b99c81f425d477eaf4ce513cf9960c6d\": rpc error: code = NotFound desc = could not find container \"94cab9770b825e5812b835012f8f16b0b99c81f425d477eaf4ce513cf9960c6d\": container with ID starting with 94cab9770b825e5812b835012f8f16b0b99c81f425d477eaf4ce513cf9960c6d not found: ID does not exist" Nov 03 02:18:43.088014 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:43.087982 5605 scope.go:122] "RemoveContainer" containerID="8938b6aa86b546ebb49211ec1e640193f4f45bf3962d90615893c4f209f900aa" Nov 03 02:18:43.088217 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:18:43.088201 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8938b6aa86b546ebb49211ec1e640193f4f45bf3962d90615893c4f209f900aa\": container with ID starting with 8938b6aa86b546ebb49211ec1e640193f4f45bf3962d90615893c4f209f900aa not found: ID does not exist" containerID="8938b6aa86b546ebb49211ec1e640193f4f45bf3962d90615893c4f209f900aa" Nov 03 02:18:43.088269 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:43.088223 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8938b6aa86b546ebb49211ec1e640193f4f45bf3962d90615893c4f209f900aa"} err="failed to get container status \"8938b6aa86b546ebb49211ec1e640193f4f45bf3962d90615893c4f209f900aa\": rpc error: code = NotFound desc = could not find container \"8938b6aa86b546ebb49211ec1e640193f4f45bf3962d90615893c4f209f900aa\": container with ID starting with 8938b6aa86b546ebb49211ec1e640193f4f45bf3962d90615893c4f209f900aa not found: ID does not exist" Nov 03 02:18:43.222467 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:43.222397 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d"] Nov 03 02:18:43.226401 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:43.226372 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-aa82d2-openshift-default-78bf4ccd4c-gm99d"] Nov 03 02:18:44.850141 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:18:44.850108 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2225e2b8-fca7-4a8a-8c47-a687a0d9b214" path="/var/lib/kubelet/pods/2225e2b8-fca7-4a8a-8c47-a687a0d9b214/volumes" Nov 03 02:18:53.916638 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:18:53.916602 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:19:03.331202 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:19:03.331172 5605 scope.go:122] "RemoveContainer" containerID="d791d936e02ed1fda40b7fbfa99f5f55a66ed1de8573878bda394a3846a9b3c0" Nov 03 02:19:03.339643 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:19:03.339624 5605 scope.go:122] "RemoveContainer" containerID="57803f4c0d2e7b43ace6ac24cd0077922f6862d4c116c3e3bd131d7f27043eaf" Nov 03 02:19:04.695568 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:19:04.695539 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-701ba30f/e2e-lifecycle-model-701ba30f-kserve-64469b6665-bmqzq"] Nov 03 02:19:05.950727 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:19:05.950693 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:19:19.715749 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:19:19.715650 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-701ba30f/e2e-lifecycle-model-701ba30f-kserve-64469b6665-bmqzq" podUID="451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce" containerName="main" containerID="cri-o://df9ddefd0f1c905c028fd7143bb5cca734774745782ee46a0d3a6017c3d2cd21" gracePeriod=45 Nov 03 02:19:19.979630 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:19:19.979576 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-701ba30f/e2e-lifecycle-model-701ba30f-kserve-64469b6665-bmqzq" Nov 03 02:19:20.050490 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:19:20.050453 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce-dshm\" (UniqueName: \"kubernetes.io/empty-dir/451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce-dshm\") pod \"451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce\" (UID: \"451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce\") " Nov 03 02:19:20.050611 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:19:20.050502 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce-model-cache\") pod \"451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce\" (UID: \"451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce\") " Nov 03 02:19:20.050611 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:19:20.050542 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce-home\" (UniqueName: \"kubernetes.io/empty-dir/451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce-home\") pod \"451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce\" (UID: \"451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce\") " Nov 03 02:19:20.050611 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:19:20.050567 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce-kube-api-access-hgpdk\" (UniqueName: \"kubernetes.io/projected/451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce-kube-api-access-hgpdk\") pod \"451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce\" (UID: \"451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce\") " Nov 03 02:19:20.050789 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:19:20.050669 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce-tls-certs\" (UniqueName: \"kubernetes.io/secret/451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce-tls-certs\") pod \"451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce\" (UID: \"451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce\") " Nov 03 02:19:20.050789 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:19:20.050708 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce-tmp-dir\") pod \"451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce\" (UID: \"451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce\") " Nov 03 02:19:20.050789 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:19:20.050762 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce-model-cache" pod "451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce" (UID: "451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:19:20.050922 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:19:20.050795 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce-home" pod "451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce" (UID: "451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:19:20.051071 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:19:20.051041 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce-tmp-dir" pod "451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce" (UID: "451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:19:20.051221 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:19:20.051204 5605 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce-tmp-dir\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:19:20.051279 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:19:20.051228 5605 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce-model-cache\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:19:20.051279 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:19:20.051244 5605 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce-home\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:19:20.052609 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:19:20.052585 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce-dshm" pod "451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce" (UID: "451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:19:20.052760 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:19:20.052736 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce-kube-api-access-hgpdk" pod "451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce" (UID: "451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce"). InnerVolumeSpecName "kube-api-access-hgpdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:19:20.052807 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:19:20.052743 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce-tls-certs" pod "451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce" (UID: "451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:19:20.152731 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:19:20.152701 5605 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce-dshm\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:19:20.152731 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:19:20.152728 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgpdk\" (UniqueName: \"kubernetes.io/projected/451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce-kube-api-access-hgpdk\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:19:20.152901 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:19:20.152738 5605 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce-tls-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:19:20.334482 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:19:20.334447 5605 generic.go:356] "Generic (PLEG): container finished" podID="451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce" containerID="df9ddefd0f1c905c028fd7143bb5cca734774745782ee46a0d3a6017c3d2cd21" exitCode=0 Nov 03 02:19:20.334649 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:19:20.334489 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-701ba30f/e2e-lifecycle-model-701ba30f-kserve-64469b6665-bmqzq" Nov 03 02:19:20.334649 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:19:20.334536 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-701ba30f/e2e-lifecycle-model-701ba30f-kserve-64469b6665-bmqzq" event={"ID":"451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce","Type":"ContainerDied","Data":"df9ddefd0f1c905c028fd7143bb5cca734774745782ee46a0d3a6017c3d2cd21"} Nov 03 02:19:20.334649 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:19:20.334574 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-701ba30f/e2e-lifecycle-model-701ba30f-kserve-64469b6665-bmqzq" event={"ID":"451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce","Type":"ContainerDied","Data":"5bfc1fd4e55296e8caf37ce9e38bbee008d69353a0024183d28bd9686b9d043f"} Nov 03 02:19:20.334649 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:19:20.334590 5605 scope.go:122] "RemoveContainer" containerID="df9ddefd0f1c905c028fd7143bb5cca734774745782ee46a0d3a6017c3d2cd21" Nov 03 02:19:20.344616 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:19:20.344599 5605 scope.go:122] "RemoveContainer" containerID="df9ddefd0f1c905c028fd7143bb5cca734774745782ee46a0d3a6017c3d2cd21" Nov 03 02:19:20.344863 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:19:20.344845 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df9ddefd0f1c905c028fd7143bb5cca734774745782ee46a0d3a6017c3d2cd21\": container with ID starting with df9ddefd0f1c905c028fd7143bb5cca734774745782ee46a0d3a6017c3d2cd21 not found: ID does not exist" containerID="df9ddefd0f1c905c028fd7143bb5cca734774745782ee46a0d3a6017c3d2cd21" Nov 03 02:19:20.344899 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:19:20.344873 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df9ddefd0f1c905c028fd7143bb5cca734774745782ee46a0d3a6017c3d2cd21"} err="failed to get container status \"df9ddefd0f1c905c028fd7143bb5cca734774745782ee46a0d3a6017c3d2cd21\": rpc error: code = NotFound desc = could not find container \"df9ddefd0f1c905c028fd7143bb5cca734774745782ee46a0d3a6017c3d2cd21\": container with ID starting with df9ddefd0f1c905c028fd7143bb5cca734774745782ee46a0d3a6017c3d2cd21 not found: ID does not exist" Nov 03 02:19:20.486840 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:19:20.486817 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-701ba30f/e2e-lifecycle-model-701ba30f-kserve-64469b6665-bmqzq"] Nov 03 02:19:20.533281 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:19:20.533259 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-701ba30f/e2e-lifecycle-model-701ba30f-kserve-64469b6665-bmqzq"] Nov 03 02:19:20.849453 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:19:20.849429 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce" path="/var/lib/kubelet/pods/451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce/volumes" Nov 03 02:19:23.915640 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:19:23.915605 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:19:41.514684 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:19:41.514652 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:19:52.049838 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:19:52.049802 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d705e3c-z46sq"] Nov 03 02:19:52.082822 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:19:52.082797 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d705e3c-z46sq"] Nov 03 02:19:52.851773 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:19:52.851724 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba867d73-335f-4cdd-9dea-beec4c925dd3" path="/var/lib/kubelet/pods/ba867d73-335f-4cdd-9dea-beec4c925dd3/volumes" Nov 03 02:19:53.920623 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:19:53.920588 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:19:55.556335 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:19:55.556305 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf"] Nov 03 02:19:55.556818 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:19:55.556608 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf" podUID="3aa07ac7-fa23-451b-a6a6-0bea3103c9d1" containerName="istio-proxy" containerID="cri-o://c6a47c13c3ccc2ec6b7381ceaaff71c22e6c72edbb545caace97a64e144c4a06" gracePeriod=30 Nov 03 02:20:00.744762 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:00.744722 5605 generic.go:356] "Generic (PLEG): container finished" podID="3aa07ac7-fa23-451b-a6a6-0bea3103c9d1" containerID="c6a47c13c3ccc2ec6b7381ceaaff71c22e6c72edbb545caace97a64e144c4a06" exitCode=0 Nov 03 02:20:00.745211 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:00.744778 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf" event={"ID":"3aa07ac7-fa23-451b-a6a6-0bea3103c9d1","Type":"ContainerDied","Data":"c6a47c13c3ccc2ec6b7381ceaaff71c22e6c72edbb545caace97a64e144c4a06"} Nov 03 02:20:00.745211 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:00.744875 5605 scope.go:122] "RemoveContainer" containerID="15d7220b13fb52a80c5cf245b3f6ee02c489b270610086b5e918c017e3b411d6" Nov 03 02:20:00.826440 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:00.826381 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf" Nov 03 02:20:00.909099 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:00.909071 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aa07ac7-fa23-451b-a6a6-0bea3103c9d1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa07ac7-fa23-451b-a6a6-0bea3103c9d1-credential-socket\") pod \"3aa07ac7-fa23-451b-a6a6-0bea3103c9d1\" (UID: \"3aa07ac7-fa23-451b-a6a6-0bea3103c9d1\") " Nov 03 02:20:00.909208 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:00.909122 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aa07ac7-fa23-451b-a6a6-0bea3103c9d1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aa07ac7-fa23-451b-a6a6-0bea3103c9d1-workload-certs\") pod \"3aa07ac7-fa23-451b-a6a6-0bea3103c9d1\" (UID: \"3aa07ac7-fa23-451b-a6a6-0bea3103c9d1\") " Nov 03 02:20:00.909208 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:00.909156 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aa07ac7-fa23-451b-a6a6-0bea3103c9d1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa07ac7-fa23-451b-a6a6-0bea3103c9d1-workload-socket\") pod \"3aa07ac7-fa23-451b-a6a6-0bea3103c9d1\" (UID: \"3aa07ac7-fa23-451b-a6a6-0bea3103c9d1\") " Nov 03 02:20:00.909288 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:00.909205 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aa07ac7-fa23-451b-a6a6-0bea3103c9d1-kube-api-access-z74sx\" (UniqueName: \"kubernetes.io/projected/3aa07ac7-fa23-451b-a6a6-0bea3103c9d1-kube-api-access-z74sx\") pod \"3aa07ac7-fa23-451b-a6a6-0bea3103c9d1\" (UID: \"3aa07ac7-fa23-451b-a6a6-0bea3103c9d1\") " Nov 03 02:20:00.909288 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:00.909242 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3aa07ac7-fa23-451b-a6a6-0bea3103c9d1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aa07ac7-fa23-451b-a6a6-0bea3103c9d1-istio-podinfo\") pod \"3aa07ac7-fa23-451b-a6a6-0bea3103c9d1\" (UID: \"3aa07ac7-fa23-451b-a6a6-0bea3103c9d1\") " Nov 03 02:20:00.909411 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:00.909378 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aa07ac7-fa23-451b-a6a6-0bea3103c9d1-istio-token\" (UniqueName: \"kubernetes.io/projected/3aa07ac7-fa23-451b-a6a6-0bea3103c9d1-istio-token\") pod \"3aa07ac7-fa23-451b-a6a6-0bea3103c9d1\" (UID: \"3aa07ac7-fa23-451b-a6a6-0bea3103c9d1\") " Nov 03 02:20:00.909411 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:00.909394 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aa07ac7-fa23-451b-a6a6-0bea3103c9d1-workload-socket" pod "3aa07ac7-fa23-451b-a6a6-0bea3103c9d1" (UID: "3aa07ac7-fa23-451b-a6a6-0bea3103c9d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:20:00.909546 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:00.909440 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aa07ac7-fa23-451b-a6a6-0bea3103c9d1-workload-certs" pod "3aa07ac7-fa23-451b-a6a6-0bea3103c9d1" (UID: "3aa07ac7-fa23-451b-a6a6-0bea3103c9d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:20:00.909546 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:00.909448 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aa07ac7-fa23-451b-a6a6-0bea3103c9d1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aa07ac7-fa23-451b-a6a6-0bea3103c9d1-istio-data\") pod \"3aa07ac7-fa23-451b-a6a6-0bea3103c9d1\" (UID: \"3aa07ac7-fa23-451b-a6a6-0bea3103c9d1\") " Nov 03 02:20:00.909546 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:00.909528 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3aa07ac7-fa23-451b-a6a6-0bea3103c9d1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aa07ac7-fa23-451b-a6a6-0bea3103c9d1-istiod-ca-cert\") pod \"3aa07ac7-fa23-451b-a6a6-0bea3103c9d1\" (UID: \"3aa07ac7-fa23-451b-a6a6-0bea3103c9d1\") " Nov 03 02:20:00.909706 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:00.909566 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aa07ac7-fa23-451b-a6a6-0bea3103c9d1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aa07ac7-fa23-451b-a6a6-0bea3103c9d1-istio-envoy\") pod \"3aa07ac7-fa23-451b-a6a6-0bea3103c9d1\" (UID: \"3aa07ac7-fa23-451b-a6a6-0bea3103c9d1\") " Nov 03 02:20:00.909706 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:00.909659 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aa07ac7-fa23-451b-a6a6-0bea3103c9d1-istio-data" pod "3aa07ac7-fa23-451b-a6a6-0bea3103c9d1" (UID: "3aa07ac7-fa23-451b-a6a6-0bea3103c9d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:20:00.910160 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:00.910073 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3aa07ac7-fa23-451b-a6a6-0bea3103c9d1-istiod-ca-cert" pod "3aa07ac7-fa23-451b-a6a6-0bea3103c9d1" (UID: "3aa07ac7-fa23-451b-a6a6-0bea3103c9d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:20:00.910336 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:00.909432 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aa07ac7-fa23-451b-a6a6-0bea3103c9d1-credential-socket" pod "3aa07ac7-fa23-451b-a6a6-0bea3103c9d1" (UID: "3aa07ac7-fa23-451b-a6a6-0bea3103c9d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:20:00.910803 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:00.910490 5605 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aa07ac7-fa23-451b-a6a6-0bea3103c9d1-workload-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:20:00.910892 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:00.910815 5605 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa07ac7-fa23-451b-a6a6-0bea3103c9d1-workload-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:20:00.910892 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:00.910833 5605 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aa07ac7-fa23-451b-a6a6-0bea3103c9d1-istio-data\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:20:00.910892 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:00.910846 5605 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aa07ac7-fa23-451b-a6a6-0bea3103c9d1-istiod-ca-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:20:00.911672 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:00.911634 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3aa07ac7-fa23-451b-a6a6-0bea3103c9d1-istio-podinfo" pod "3aa07ac7-fa23-451b-a6a6-0bea3103c9d1" (UID: "3aa07ac7-fa23-451b-a6a6-0bea3103c9d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:20:00.911777 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:00.911756 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aa07ac7-fa23-451b-a6a6-0bea3103c9d1-kube-api-access-z74sx" pod "3aa07ac7-fa23-451b-a6a6-0bea3103c9d1" (UID: "3aa07ac7-fa23-451b-a6a6-0bea3103c9d1"). InnerVolumeSpecName "kube-api-access-z74sx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:20:00.912680 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:00.912652 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aa07ac7-fa23-451b-a6a6-0bea3103c9d1-istio-envoy" pod "3aa07ac7-fa23-451b-a6a6-0bea3103c9d1" (UID: "3aa07ac7-fa23-451b-a6a6-0bea3103c9d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:20:00.912756 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:00.912721 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aa07ac7-fa23-451b-a6a6-0bea3103c9d1-istio-token" pod "3aa07ac7-fa23-451b-a6a6-0bea3103c9d1" (UID: "3aa07ac7-fa23-451b-a6a6-0bea3103c9d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:20:01.012020 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:01.011995 5605 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3aa07ac7-fa23-451b-a6a6-0bea3103c9d1-istio-token\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:20:01.012020 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:01.012019 5605 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aa07ac7-fa23-451b-a6a6-0bea3103c9d1-istio-envoy\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:20:01.012169 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:01.012031 5605 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa07ac7-fa23-451b-a6a6-0bea3103c9d1-credential-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:20:01.012169 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:01.012040 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z74sx\" (UniqueName: \"kubernetes.io/projected/3aa07ac7-fa23-451b-a6a6-0bea3103c9d1-kube-api-access-z74sx\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:20:01.012169 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:01.012051 5605 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aa07ac7-fa23-451b-a6a6-0bea3103c9d1-istio-podinfo\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:20:01.752907 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:01.752873 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf" event={"ID":"3aa07ac7-fa23-451b-a6a6-0bea3103c9d1","Type":"ContainerDied","Data":"5e4e7fea4dea39540780919d9144df879441976cbaaf032257613421d4c7b4c5"} Nov 03 02:20:01.752907 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:01.752904 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf" Nov 03 02:20:01.753410 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:01.752933 5605 scope.go:122] "RemoveContainer" containerID="c6a47c13c3ccc2ec6b7381ceaaff71c22e6c72edbb545caace97a64e144c4a06" Nov 03 02:20:01.956859 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:01.956828 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf"] Nov 03 02:20:02.001890 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:02.001854 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-701ba30f-openshift-default-55665854f7-4wrwf"] Nov 03 02:20:02.849504 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:02.849474 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3aa07ac7-fa23-451b-a6a6-0bea3103c9d1" path="/var/lib/kubelet/pods/3aa07ac7-fa23-451b-a6a6-0bea3103c9d1/volumes" Nov 03 02:20:03.375482 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:03.375456 5605 scope.go:122] "RemoveContainer" containerID="7eb277de983218730123bc8d9b26000ad4673693adb651326599fe03981597bd" Nov 03 02:20:23.914849 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:20:23.914810 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:20:38.750655 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:38.750623 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:20:49.739022 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:49.738987 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v"] Nov 03 02:20:49.739447 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:49.739222 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v" podUID="3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f" containerName="istio-proxy" containerID="cri-o://248e50158d844363bcfcf798e336953ea1c002f02f86cec88e4b766b50249beb" gracePeriod=30 Nov 03 02:20:53.923782 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:20:53.923748 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:20:55.001113 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:55.001090 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v" Nov 03 02:20:55.123953 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:55.123922 5605 generic.go:356] "Generic (PLEG): container finished" podID="3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f" containerID="248e50158d844363bcfcf798e336953ea1c002f02f86cec88e4b766b50249beb" exitCode=0 Nov 03 02:20:55.124136 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:55.123962 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v" event={"ID":"3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f","Type":"ContainerDied","Data":"248e50158d844363bcfcf798e336953ea1c002f02f86cec88e4b766b50249beb"} Nov 03 02:20:55.124136 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:55.123979 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v" Nov 03 02:20:55.124136 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:55.124001 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v" event={"ID":"3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f","Type":"ContainerDied","Data":"37f115b4fb3baf3eb04bbb70e7a3e34128b2700e1c398921b35dfbb106928b01"} Nov 03 02:20:55.124136 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:55.124025 5605 scope.go:122] "RemoveContainer" containerID="248e50158d844363bcfcf798e336953ea1c002f02f86cec88e4b766b50249beb" Nov 03 02:20:55.132678 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:55.132659 5605 scope.go:122] "RemoveContainer" containerID="37610810acc57ec8d446db9a2193104731dab7ed2131a1f5df057e3360b7aefc" Nov 03 02:20:55.140920 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:55.140902 5605 scope.go:122] "RemoveContainer" containerID="248e50158d844363bcfcf798e336953ea1c002f02f86cec88e4b766b50249beb" Nov 03 02:20:55.141172 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:20:55.141153 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"248e50158d844363bcfcf798e336953ea1c002f02f86cec88e4b766b50249beb\": container with ID starting with 248e50158d844363bcfcf798e336953ea1c002f02f86cec88e4b766b50249beb not found: ID does not exist" containerID="248e50158d844363bcfcf798e336953ea1c002f02f86cec88e4b766b50249beb" Nov 03 02:20:55.141239 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:55.141183 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"248e50158d844363bcfcf798e336953ea1c002f02f86cec88e4b766b50249beb"} err="failed to get container status \"248e50158d844363bcfcf798e336953ea1c002f02f86cec88e4b766b50249beb\": rpc error: code = NotFound desc = could not find container \"248e50158d844363bcfcf798e336953ea1c002f02f86cec88e4b766b50249beb\": container with ID starting with 248e50158d844363bcfcf798e336953ea1c002f02f86cec88e4b766b50249beb not found: ID does not exist" Nov 03 02:20:55.141239 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:55.141208 5605 scope.go:122] "RemoveContainer" containerID="37610810acc57ec8d446db9a2193104731dab7ed2131a1f5df057e3360b7aefc" Nov 03 02:20:55.141468 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:20:55.141450 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37610810acc57ec8d446db9a2193104731dab7ed2131a1f5df057e3360b7aefc\": container with ID starting with 37610810acc57ec8d446db9a2193104731dab7ed2131a1f5df057e3360b7aefc not found: ID does not exist" containerID="37610810acc57ec8d446db9a2193104731dab7ed2131a1f5df057e3360b7aefc" Nov 03 02:20:55.141515 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:55.141474 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37610810acc57ec8d446db9a2193104731dab7ed2131a1f5df057e3360b7aefc"} err="failed to get container status \"37610810acc57ec8d446db9a2193104731dab7ed2131a1f5df057e3360b7aefc\": rpc error: code = NotFound desc = could not find container \"37610810acc57ec8d446db9a2193104731dab7ed2131a1f5df057e3360b7aefc\": container with ID starting with 37610810acc57ec8d446db9a2193104731dab7ed2131a1f5df057e3360b7aefc not found: ID does not exist" Nov 03 02:20:55.164312 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:55.164289 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f-istio-envoy\") pod \"3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f\" (UID: \"3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f\") " Nov 03 02:20:55.164446 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:55.164416 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f-istio-data\") pod \"3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f\" (UID: \"3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f\") " Nov 03 02:20:55.164515 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:55.164499 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f-credential-socket\") pod \"3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f\" (UID: \"3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f\") " Nov 03 02:20:55.164569 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:55.164557 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f-kube-api-access-4x7qx\" (UniqueName: \"kubernetes.io/projected/3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f-kube-api-access-4x7qx\") pod \"3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f\" (UID: \"3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f\") " Nov 03 02:20:55.164625 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:55.164600 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f-istio-podinfo\") pod \"3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f\" (UID: \"3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f\") " Nov 03 02:20:55.164852 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:55.164659 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f-istio-data" pod "3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f" (UID: "3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:20:55.164852 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:55.164677 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f-istio-token\" (UniqueName: \"kubernetes.io/projected/3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f-istio-token\") pod \"3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f\" (UID: \"3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f\") " Nov 03 02:20:55.164852 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:55.164741 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f-workload-socket\") pod \"3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f\" (UID: \"3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f\") " Nov 03 02:20:55.165079 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:55.164867 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f-credential-socket" pod "3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f" (UID: "3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:20:55.165625 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:55.165221 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f-istiod-ca-cert\") pod \"3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f\" (UID: \"3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f\") " Nov 03 02:20:55.165625 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:55.165273 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f-workload-certs\") pod \"3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f\" (UID: \"3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f\") " Nov 03 02:20:55.165625 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:55.165272 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f-workload-socket" pod "3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f" (UID: "3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:20:55.165625 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:55.165569 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f-workload-certs" pod "3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f" (UID: "3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:20:55.165625 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:55.165599 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f-istiod-ca-cert" pod "3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f" (UID: "3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:20:55.165925 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:55.165878 5605 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f-workload-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:20:55.165925 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:55.165899 5605 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f-istiod-ca-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:20:55.165925 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:55.165914 5605 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f-workload-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:20:55.167688 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:55.165928 5605 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f-istio-data\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:20:55.167688 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:55.165942 5605 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f-credential-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:20:55.167688 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:55.167063 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f-istio-envoy" pod "3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f" (UID: "3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:20:55.167688 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:55.167184 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f-kube-api-access-4x7qx" pod "3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f" (UID: "3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f"). InnerVolumeSpecName "kube-api-access-4x7qx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:20:55.167688 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:55.167521 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f-istio-token" pod "3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f" (UID: "3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:20:55.168257 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:55.168219 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f-istio-podinfo" pod "3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f" (UID: "3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:20:55.267663 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:55.267628 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4x7qx\" (UniqueName: \"kubernetes.io/projected/3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f-kube-api-access-4x7qx\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:20:55.267663 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:55.267658 5605 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f-istio-podinfo\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:20:55.267663 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:55.267669 5605 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f-istio-token\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:20:55.267880 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:55.267678 5605 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f-istio-envoy\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:20:55.609510 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:55.609476 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v"] Nov 03 02:20:55.613266 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:55.613239 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-hjp4v"] Nov 03 02:20:56.850556 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:20:56.850527 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f" path="/var/lib/kubelet/pods/3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f/volumes" Nov 03 02:21:03.413398 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:21:03.413352 5605 scope.go:122] "RemoveContainer" containerID="d05d459b581769ef02f92eb25ba6c209e2cf53e3e781552a0f0ffa0b0b0df867" Nov 03 02:21:22.041916 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:21:22.041881 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f911b8f-jl4ph"] Nov 03 02:21:22.088389 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:21:22.088345 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f911b8f-jl4ph"] Nov 03 02:21:22.849862 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:21:22.849833 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c65d37f-f5f0-48a1-a4d8-9cfdeae07e75" path="/var/lib/kubelet/pods/0c65d37f-f5f0-48a1-a4d8-9cfdeae07e75/volumes" Nov 03 02:21:23.915238 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:21:23.915199 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:21:26.395107 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:21:26.395061 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:21:53.915206 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:21:53.915166 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:22:03.437782 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:03.437747 5605 scope.go:122] "RemoveContainer" containerID="c4a7e3d1b97d5ec5460cb2d96e3ebbd196928b507b83c19a636fdbe7a6fbdb98" Nov 03 02:22:23.059637 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:23.059546 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:22:23.914329 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:22:23.914216 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:22:30.220932 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:30.220899 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d63a25-6769697dcf-lzql7"] Nov 03 02:22:30.221347 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:30.221336 5605 cpu_manager.go:414] "containerMap: removing container" podUID="3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f" containerName="istio-proxy" Nov 03 02:22:30.221347 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:30.221347 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f" containerName="istio-proxy" Nov 03 02:22:30.221443 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:30.221371 5605 cpu_manager.go:414] "containerMap: removing container" podUID="2225e2b8-fca7-4a8a-8c47-a687a0d9b214" containerName="istio-proxy" Nov 03 02:22:30.221443 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:30.221376 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2225e2b8-fca7-4a8a-8c47-a687a0d9b214" containerName="istio-proxy" Nov 03 02:22:30.221443 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:30.221386 5605 cpu_manager.go:414] "containerMap: removing container" podUID="451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce" containerName="main" Nov 03 02:22:30.221443 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:30.221391 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce" containerName="main" Nov 03 02:22:30.221443 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:30.221408 5605 cpu_manager.go:414] "containerMap: removing container" podUID="69d57c7a-8495-4d67-9547-dfe082939941" containerName="istio-proxy" Nov 03 02:22:30.221443 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:30.221413 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69d57c7a-8495-4d67-9547-dfe082939941" containerName="istio-proxy" Nov 03 02:22:30.221443 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:30.221420 5605 cpu_manager.go:414] "containerMap: removing container" podUID="58ecd8f7-099d-401b-a63a-6c4e79e56077" containerName="istio-proxy" Nov 03 02:22:30.221443 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:30.221424 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58ecd8f7-099d-401b-a63a-6c4e79e56077" containerName="istio-proxy" Nov 03 02:22:30.221443 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:30.221437 5605 cpu_manager.go:414] "containerMap: removing container" podUID="3e782231-a2f8-45d9-8507-fcc038f1a368" containerName="payload-pre-processing" Nov 03 02:22:30.221443 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:30.221441 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e782231-a2f8-45d9-8507-fcc038f1a368" containerName="payload-pre-processing" Nov 03 02:22:30.221723 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:30.221452 5605 cpu_manager.go:414] "containerMap: removing container" podUID="0c65d37f-f5f0-48a1-a4d8-9cfdeae07e75" containerName="revoke-keys" Nov 03 02:22:30.221723 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:30.221457 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c65d37f-f5f0-48a1-a4d8-9cfdeae07e75" containerName="revoke-keys" Nov 03 02:22:30.221723 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:30.221480 5605 cpu_manager.go:414] "containerMap: removing container" podUID="87ac6502-78a2-41ed-9033-77eecf21086f" containerName="istio-proxy" Nov 03 02:22:30.221723 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:30.221485 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87ac6502-78a2-41ed-9033-77eecf21086f" containerName="istio-proxy" Nov 03 02:22:30.221723 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:30.221493 5605 cpu_manager.go:414] "containerMap: removing container" podUID="3aa07ac7-fa23-451b-a6a6-0bea3103c9d1" containerName="istio-proxy" Nov 03 02:22:30.221723 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:30.221497 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aa07ac7-fa23-451b-a6a6-0bea3103c9d1" containerName="istio-proxy" Nov 03 02:22:30.221723 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:30.221504 5605 cpu_manager.go:414] "containerMap: removing container" podUID="87ac6502-78a2-41ed-9033-77eecf21086f" containerName="istio-proxy" Nov 03 02:22:30.221723 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:30.221509 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87ac6502-78a2-41ed-9033-77eecf21086f" containerName="istio-proxy" Nov 03 02:22:30.221723 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:30.221517 5605 cpu_manager.go:414] "containerMap: removing container" podUID="3aa07ac7-fa23-451b-a6a6-0bea3103c9d1" containerName="istio-proxy" Nov 03 02:22:30.221723 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:30.221521 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aa07ac7-fa23-451b-a6a6-0bea3103c9d1" containerName="istio-proxy" Nov 03 02:22:30.221723 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:30.221527 5605 cpu_manager.go:414] "containerMap: removing container" podUID="5c586c92-59da-421e-8224-45b9a833c2d4" containerName="istio-proxy" Nov 03 02:22:30.221723 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:30.221531 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c586c92-59da-421e-8224-45b9a833c2d4" containerName="istio-proxy" Nov 03 02:22:30.221723 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:30.221609 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e782231-a2f8-45d9-8507-fcc038f1a368" containerName="payload-pre-processing" Nov 03 02:22:30.221723 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:30.221616 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="2225e2b8-fca7-4a8a-8c47-a687a0d9b214" containerName="istio-proxy" Nov 03 02:22:30.221723 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:30.221622 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="451ae5f0-fd8d-4e70-9c67-dadb86d2b0ce" containerName="main" Nov 03 02:22:30.221723 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:30.221628 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c65d37f-f5f0-48a1-a4d8-9cfdeae07e75" containerName="revoke-keys" Nov 03 02:22:30.221723 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:30.221634 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="87ac6502-78a2-41ed-9033-77eecf21086f" containerName="istio-proxy" Nov 03 02:22:30.221723 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:30.221641 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aa07ac7-fa23-451b-a6a6-0bea3103c9d1" containerName="istio-proxy" Nov 03 02:22:30.221723 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:30.221645 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="87ac6502-78a2-41ed-9033-77eecf21086f" containerName="istio-proxy" Nov 03 02:22:30.221723 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:30.221651 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f" containerName="istio-proxy" Nov 03 02:22:30.221723 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:30.221656 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="69d57c7a-8495-4d67-9547-dfe082939941" containerName="istio-proxy" Nov 03 02:22:30.221723 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:30.221661 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="58ecd8f7-099d-401b-a63a-6c4e79e56077" containerName="istio-proxy" Nov 03 02:22:30.221723 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:30.221667 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c586c92-59da-421e-8224-45b9a833c2d4" containerName="istio-proxy" Nov 03 02:22:30.221723 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:30.221672 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aa07ac7-fa23-451b-a6a6-0bea3103c9d1" containerName="istio-proxy" Nov 03 02:22:30.221723 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:30.221678 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f" containerName="istio-proxy" Nov 03 02:22:30.224690 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:30.224669 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d63a25-6769697dcf-lzql7" Nov 03 02:22:30.357536 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:30.324619 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-d63a25-dockercfg-5s5px\"" Nov 03 02:22:30.357536 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:30.324744 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-d63a25\"" Nov 03 02:22:30.357536 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:30.327974 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d63a25-6769697dcf-lzql7"] Nov 03 02:22:30.373088 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:30.373051 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/969764a4-4632-4cda-b6e2-2745ccbf2ab8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d63a25-6769697dcf-lzql7\" (UID: \"969764a4-4632-4cda-b6e2-2745ccbf2ab8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d63a25-6769697dcf-lzql7" Nov 03 02:22:30.426802 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:30.426768 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxm6b\" (UniqueName: \"kubernetes.io/projected/969764a4-4632-4cda-b6e2-2745ccbf2ab8-kube-api-access-mxm6b\") pod \"payload-pre-processing-e2e-api-b-d63a25-6769697dcf-lzql7\" (UID: \"969764a4-4632-4cda-b6e2-2745ccbf2ab8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d63a25-6769697dcf-lzql7" Nov 03 02:22:30.471017 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:30.470928 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d63a25-6ff9c4b65f-jfgrr"] Nov 03 02:22:30.471911 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:30.471883 5605 cpu_manager.go:414] "containerMap: removing container" podUID="3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f" containerName="istio-proxy" Nov 03 02:22:30.472083 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:30.472071 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e97f1e8-d02b-47f5-9eed-4cab3cdcf44f" containerName="istio-proxy" Nov 03 02:22:30.472174 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:30.472164 5605 cpu_manager.go:414] "containerMap: removing container" podUID="2225e2b8-fca7-4a8a-8c47-a687a0d9b214" containerName="istio-proxy" Nov 03 02:22:30.472253 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:30.472243 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2225e2b8-fca7-4a8a-8c47-a687a0d9b214" containerName="istio-proxy" Nov 03 02:22:30.472530 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:30.472513 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="2225e2b8-fca7-4a8a-8c47-a687a0d9b214" containerName="istio-proxy" Nov 03 02:22:30.493567 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:30.493518 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d63a25-6ff9c4b65f-jfgrr"] Nov 03 02:22:30.493733 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:30.493706 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d63a25-6ff9c4b65f-jfgrr" Nov 03 02:22:30.531853 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:30.531824 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/969764a4-4632-4cda-b6e2-2745ccbf2ab8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d63a25-6769697dcf-lzql7\" (UID: \"969764a4-4632-4cda-b6e2-2745ccbf2ab8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d63a25-6769697dcf-lzql7" Nov 03 02:22:30.532014 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:30.531992 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxm6b\" (UniqueName: \"kubernetes.io/projected/969764a4-4632-4cda-b6e2-2745ccbf2ab8-kube-api-access-mxm6b\") pod \"payload-pre-processing-e2e-api-b-d63a25-6769697dcf-lzql7\" (UID: \"969764a4-4632-4cda-b6e2-2745ccbf2ab8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d63a25-6769697dcf-lzql7" Nov 03 02:22:30.532526 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:30.532504 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/969764a4-4632-4cda-b6e2-2745ccbf2ab8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d63a25-6769697dcf-lzql7\" (UID: \"969764a4-4632-4cda-b6e2-2745ccbf2ab8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d63a25-6769697dcf-lzql7" Nov 03 02:22:30.585130 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:30.585106 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxm6b\" (UniqueName: \"kubernetes.io/projected/969764a4-4632-4cda-b6e2-2745ccbf2ab8-kube-api-access-mxm6b\") pod \"payload-pre-processing-e2e-api-b-d63a25-6769697dcf-lzql7\" (UID: \"969764a4-4632-4cda-b6e2-2745ccbf2ab8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d63a25-6769697dcf-lzql7" Nov 03 02:22:30.633275 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:30.633247 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2skhb\" (UniqueName: \"kubernetes.io/projected/b11fa364-4870-438e-861e-46e0f0121f8f-kube-api-access-2skhb\") pod \"payload-processing-e2e-api-b-d63a25-6ff9c4b65f-jfgrr\" (UID: \"b11fa364-4870-438e-861e-46e0f0121f8f\") " pod="openshift-ingress/payload-processing-e2e-api-b-d63a25-6ff9c4b65f-jfgrr" Nov 03 02:22:30.633406 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:30.633390 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b11fa364-4870-438e-861e-46e0f0121f8f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d63a25-6ff9c4b65f-jfgrr\" (UID: \"b11fa364-4870-438e-861e-46e0f0121f8f\") " pod="openshift-ingress/payload-processing-e2e-api-b-d63a25-6ff9c4b65f-jfgrr" Nov 03 02:22:30.734021 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:30.733961 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b11fa364-4870-438e-861e-46e0f0121f8f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d63a25-6ff9c4b65f-jfgrr\" (UID: \"b11fa364-4870-438e-861e-46e0f0121f8f\") " pod="openshift-ingress/payload-processing-e2e-api-b-d63a25-6ff9c4b65f-jfgrr" Nov 03 02:22:30.734126 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:30.734060 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2skhb\" (UniqueName: \"kubernetes.io/projected/b11fa364-4870-438e-861e-46e0f0121f8f-kube-api-access-2skhb\") pod \"payload-processing-e2e-api-b-d63a25-6ff9c4b65f-jfgrr\" (UID: \"b11fa364-4870-438e-861e-46e0f0121f8f\") " pod="openshift-ingress/payload-processing-e2e-api-b-d63a25-6ff9c4b65f-jfgrr" Nov 03 02:22:30.734649 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:30.734630 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b11fa364-4870-438e-861e-46e0f0121f8f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d63a25-6ff9c4b65f-jfgrr\" (UID: \"b11fa364-4870-438e-861e-46e0f0121f8f\") " pod="openshift-ingress/payload-processing-e2e-api-b-d63a25-6ff9c4b65f-jfgrr" Nov 03 02:22:30.786899 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:30.786872 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2skhb\" (UniqueName: \"kubernetes.io/projected/b11fa364-4870-438e-861e-46e0f0121f8f-kube-api-access-2skhb\") pod \"payload-processing-e2e-api-b-d63a25-6ff9c4b65f-jfgrr\" (UID: \"b11fa364-4870-438e-861e-46e0f0121f8f\") " pod="openshift-ingress/payload-processing-e2e-api-b-d63a25-6ff9c4b65f-jfgrr" Nov 03 02:22:30.812322 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:30.812289 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d63a25-6ff9c4b65f-jfgrr" Nov 03 02:22:30.837551 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:30.837528 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d63a25-6769697dcf-lzql7" Nov 03 02:22:31.197275 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:31.197247 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d63a25-6ff9c4b65f-jfgrr"] Nov 03 02:22:31.202125 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:22:31.202093 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb11fa364_4870_438e_861e_46e0f0121f8f.slice/crio-754b85b367910bcb61c43171b752631d60f950f553dc1d4641328a27f89f0fb8 WatchSource:0}: Error finding container 754b85b367910bcb61c43171b752631d60f950f553dc1d4641328a27f89f0fb8: Status 404 returned error can't find the container with id 754b85b367910bcb61c43171b752631d60f950f553dc1d4641328a27f89f0fb8 Nov 03 02:22:31.291536 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:31.289898 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d63a25-6769697dcf-lzql7"] Nov 03 02:22:31.805454 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:31.805410 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d63a25-6769697dcf-lzql7" event={"ID":"969764a4-4632-4cda-b6e2-2745ccbf2ab8","Type":"ContainerStarted","Data":"799ec4184b4ca1388f5a6721cbc6cb6d74265a85e6c7af67e2e6e854152745b2"} Nov 03 02:22:31.805645 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:31.805476 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d63a25-6769697dcf-lzql7" event={"ID":"969764a4-4632-4cda-b6e2-2745ccbf2ab8","Type":"ContainerStarted","Data":"625285ff52965296a2dd04baab8085c600850983a5ce675aaa01fc2fda67b95f"} Nov 03 02:22:31.805645 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:31.805575 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d63a25-6769697dcf-lzql7" Nov 03 02:22:31.807316 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:31.807290 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d63a25-6ff9c4b65f-jfgrr" event={"ID":"b11fa364-4870-438e-861e-46e0f0121f8f","Type":"ContainerStarted","Data":"11e74f795d0be19ff61e0aa4810bc77992b48b8df3d59d0a7f843e5fe17b4408"} Nov 03 02:22:31.807453 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:31.807335 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d63a25-6ff9c4b65f-jfgrr" event={"ID":"b11fa364-4870-438e-861e-46e0f0121f8f","Type":"ContainerStarted","Data":"754b85b367910bcb61c43171b752631d60f950f553dc1d4641328a27f89f0fb8"} Nov 03 02:22:31.807500 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:31.807475 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-d63a25-6ff9c4b65f-jfgrr" Nov 03 02:22:31.913477 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:31.913435 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d63a25-6769697dcf-lzql7" podStartSLOduration=1.913424628 podStartE2EDuration="1.913424628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:22:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:22:31.911375033 +0000 UTC m=+1829.170573197" watchObservedRunningTime="2026-11-03 02:22:31.913424628 +0000 UTC m=+1829.172622780" Nov 03 02:22:32.017534 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:32.017481 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-d63a25-6ff9c4b65f-jfgrr" podStartSLOduration=2.017467467 podStartE2EDuration="2.017467467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:22:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:22:32.016955694 +0000 UTC m=+1829.276153846" watchObservedRunningTime="2026-11-03 02:22:32.017467467 +0000 UTC m=+1829.276665621" Nov 03 02:22:42.844566 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:42.844284 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d63a25-6769697dcf-lzql7" Nov 03 02:22:42.877797 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:22:42.877759 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-d63a25-6ff9c4b65f-jfgrr" Nov 03 02:22:53.913416 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:22:53.913379 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:23:02.925379 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:23:02.925325 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs"] Nov 03 02:23:02.926311 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:23:02.926251 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs" podUID="8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd" containerName="istio-proxy" containerID="cri-o://b76c79178ef2eb3be09b50b575dcb7a45a504facd15c28ba2b061f7ea421c297" gracePeriod=30 Nov 03 02:23:03.459176 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:23:03.459142 5605 scope.go:122] "RemoveContainer" containerID="8f056eccafef0869ada939c19dfe4ad5c9a4e4c8ada1fbe470097f9c64a65eed" Nov 03 02:23:07.064033 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:23:07.063993 5605 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 02:23:07.064426 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:23:07.064054 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs" podUID="8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 02:23:08.100016 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:23:08.099985 5605 generic.go:356] "Generic (PLEG): container finished" podID="8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd" containerID="b76c79178ef2eb3be09b50b575dcb7a45a504facd15c28ba2b061f7ea421c297" exitCode=0 Nov 03 02:23:08.100350 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:23:08.100058 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs" event={"ID":"8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd","Type":"ContainerDied","Data":"b76c79178ef2eb3be09b50b575dcb7a45a504facd15c28ba2b061f7ea421c297"} Nov 03 02:23:08.588914 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:23:08.588888 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs" Nov 03 02:23:08.652793 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:23:08.652763 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd-istio-envoy\") pod \"8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd\" (UID: \"8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd\") " Nov 03 02:23:08.652913 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:23:08.652834 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd-istio-podinfo\") pod \"8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd\" (UID: \"8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd\") " Nov 03 02:23:08.652913 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:23:08.652875 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd-istio-data\") pod \"8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd\" (UID: \"8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd\") " Nov 03 02:23:08.653088 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:23:08.652918 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd-istio-token\" (UniqueName: \"kubernetes.io/projected/8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd-istio-token\") pod \"8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd\" (UID: \"8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd\") " Nov 03 02:23:08.653088 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:23:08.652947 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd-workload-socket\") pod \"8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd\" (UID: \"8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd\") " Nov 03 02:23:08.653088 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:23:08.653038 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd-istiod-ca-cert\") pod \"8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd\" (UID: \"8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd\") " Nov 03 02:23:08.653088 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:23:08.653083 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd-kube-api-access-kvvsg\" (UniqueName: \"kubernetes.io/projected/8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd-kube-api-access-kvvsg\") pod \"8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd\" (UID: \"8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd\") " Nov 03 02:23:08.653288 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:23:08.653117 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd-istio-data" pod "8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd" (UID: "8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:23:08.653288 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:23:08.653145 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd-credential-socket\") pod \"8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd\" (UID: \"8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd\") " Nov 03 02:23:08.653288 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:23:08.653243 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd-workload-certs\") pod \"8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd\" (UID: \"8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd\") " Nov 03 02:23:08.653478 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:23:08.653286 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd-workload-socket" pod "8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd" (UID: "8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:23:08.653478 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:23:08.653409 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd-istiod-ca-cert" pod "8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd" (UID: "8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:23:08.653641 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:23:08.653579 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd-workload-certs" pod "8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd" (UID: "8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:23:08.653727 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:23:08.653646 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd-credential-socket" pod "8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd" (UID: "8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:23:08.653842 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:23:08.653823 5605 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd-istio-data\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:08.653909 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:23:08.653849 5605 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd-workload-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:08.653909 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:23:08.653864 5605 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd-istiod-ca-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:08.653909 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:23:08.653878 5605 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd-credential-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:08.653909 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:23:08.653892 5605 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd-workload-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:08.655062 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:23:08.655037 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd-istio-podinfo" pod "8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd" (UID: "8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:23:08.655163 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:23:08.655082 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd-istio-envoy" pod "8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd" (UID: "8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:23:08.655240 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:23:08.655212 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd-kube-api-access-kvvsg" pod "8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd" (UID: "8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd"). InnerVolumeSpecName "kube-api-access-kvvsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:23:08.655328 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:23:08.655308 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd-istio-token" pod "8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd" (UID: "8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:23:08.755061 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:23:08.754997 5605 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd-istio-envoy\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:08.755061 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:23:08.755023 5605 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd-istio-podinfo\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:08.755061 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:23:08.755032 5605 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd-istio-token\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:08.755061 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:23:08.755040 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvvsg\" (UniqueName: \"kubernetes.io/projected/8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd-kube-api-access-kvvsg\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:09.106555 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:23:09.106522 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs" event={"ID":"8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd","Type":"ContainerDied","Data":"98478ae1e5247e8b38d3728dab4694938569da9b1d38974212b8f0ce8bb38cbe"} Nov 03 02:23:09.106959 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:23:09.106582 5605 scope.go:122] "RemoveContainer" containerID="b76c79178ef2eb3be09b50b575dcb7a45a504facd15c28ba2b061f7ea421c297" Nov 03 02:23:09.106959 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:23:09.106537 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs" Nov 03 02:23:09.261232 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:23:09.261208 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs"] Nov 03 02:23:09.307616 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:23:09.307597 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tc9qs"] Nov 03 02:23:10.853173 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:23:10.853135 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd" path="/var/lib/kubelet/pods/8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd/volumes" Nov 03 02:23:13.044114 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:23:13.044071 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:23:23.919214 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:23:23.919165 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:23:53.914297 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:23:53.914204 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:23:57.644426 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:23:57.644387 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jl8k2"] Nov 03 02:23:57.644893 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:23:57.644677 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jl8k2" podUID="92507c2a-dbae-47e0-96f7-d50a7ccbd312" containerName="istio-proxy" containerID="cri-o://5d6a51e34152092feb4e7da6b0ca6e8c717b620de6ffe073271b83e1bed9eb0c" gracePeriod=30 Nov 03 02:23:59.176917 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:23:59.176881 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:24:00.047388 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:24:00.047331 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-jl8k2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 02:24:00.047564 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:24:00.047409 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jl8k2" podUID="92507c2a-dbae-47e0-96f7-d50a7ccbd312" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 02:24:02.893018 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:24:02.892991 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jl8k2" Nov 03 02:24:02.947642 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:24:02.947582 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/92507c2a-dbae-47e0-96f7-d50a7ccbd312-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92507c2a-dbae-47e0-96f7-d50a7ccbd312-istio-podinfo\") pod \"92507c2a-dbae-47e0-96f7-d50a7ccbd312\" (UID: \"92507c2a-dbae-47e0-96f7-d50a7ccbd312\") " Nov 03 02:24:02.947642 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:24:02.947619 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92507c2a-dbae-47e0-96f7-d50a7ccbd312-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92507c2a-dbae-47e0-96f7-d50a7ccbd312-istio-data\") pod \"92507c2a-dbae-47e0-96f7-d50a7ccbd312\" (UID: \"92507c2a-dbae-47e0-96f7-d50a7ccbd312\") " Nov 03 02:24:02.947834 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:24:02.947654 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92507c2a-dbae-47e0-96f7-d50a7ccbd312-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92507c2a-dbae-47e0-96f7-d50a7ccbd312-credential-socket\") pod \"92507c2a-dbae-47e0-96f7-d50a7ccbd312\" (UID: \"92507c2a-dbae-47e0-96f7-d50a7ccbd312\") " Nov 03 02:24:02.947834 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:24:02.947712 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92507c2a-dbae-47e0-96f7-d50a7ccbd312-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92507c2a-dbae-47e0-96f7-d50a7ccbd312-workload-certs\") pod \"92507c2a-dbae-47e0-96f7-d50a7ccbd312\" (UID: \"92507c2a-dbae-47e0-96f7-d50a7ccbd312\") " Nov 03 02:24:02.947834 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:24:02.947752 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92507c2a-dbae-47e0-96f7-d50a7ccbd312-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92507c2a-dbae-47e0-96f7-d50a7ccbd312-istio-envoy\") pod \"92507c2a-dbae-47e0-96f7-d50a7ccbd312\" (UID: \"92507c2a-dbae-47e0-96f7-d50a7ccbd312\") " Nov 03 02:24:02.947989 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:24:02.947844 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92507c2a-dbae-47e0-96f7-d50a7ccbd312-istio-token\" (UniqueName: \"kubernetes.io/projected/92507c2a-dbae-47e0-96f7-d50a7ccbd312-istio-token\") pod \"92507c2a-dbae-47e0-96f7-d50a7ccbd312\" (UID: \"92507c2a-dbae-47e0-96f7-d50a7ccbd312\") " Nov 03 02:24:02.947989 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:24:02.947895 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92507c2a-dbae-47e0-96f7-d50a7ccbd312-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92507c2a-dbae-47e0-96f7-d50a7ccbd312-istiod-ca-cert\") pod \"92507c2a-dbae-47e0-96f7-d50a7ccbd312\" (UID: \"92507c2a-dbae-47e0-96f7-d50a7ccbd312\") " Nov 03 02:24:02.947989 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:24:02.947925 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92507c2a-dbae-47e0-96f7-d50a7ccbd312-kube-api-access-frwwm\" (UniqueName: \"kubernetes.io/projected/92507c2a-dbae-47e0-96f7-d50a7ccbd312-kube-api-access-frwwm\") pod \"92507c2a-dbae-47e0-96f7-d50a7ccbd312\" (UID: \"92507c2a-dbae-47e0-96f7-d50a7ccbd312\") " Nov 03 02:24:02.947989 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:24:02.947912 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92507c2a-dbae-47e0-96f7-d50a7ccbd312-istio-data" pod "92507c2a-dbae-47e0-96f7-d50a7ccbd312" (UID: "92507c2a-dbae-47e0-96f7-d50a7ccbd312"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:24:02.947989 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:24:02.947957 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92507c2a-dbae-47e0-96f7-d50a7ccbd312-credential-socket" pod "92507c2a-dbae-47e0-96f7-d50a7ccbd312" (UID: "92507c2a-dbae-47e0-96f7-d50a7ccbd312"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:24:02.948256 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:24:02.948017 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92507c2a-dbae-47e0-96f7-d50a7ccbd312-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92507c2a-dbae-47e0-96f7-d50a7ccbd312-workload-socket\") pod \"92507c2a-dbae-47e0-96f7-d50a7ccbd312\" (UID: \"92507c2a-dbae-47e0-96f7-d50a7ccbd312\") " Nov 03 02:24:02.948256 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:24:02.948227 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92507c2a-dbae-47e0-96f7-d50a7ccbd312-istiod-ca-cert" pod "92507c2a-dbae-47e0-96f7-d50a7ccbd312" (UID: "92507c2a-dbae-47e0-96f7-d50a7ccbd312"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:24:02.948390 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:24:02.948272 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92507c2a-dbae-47e0-96f7-d50a7ccbd312-workload-certs" pod "92507c2a-dbae-47e0-96f7-d50a7ccbd312" (UID: "92507c2a-dbae-47e0-96f7-d50a7ccbd312"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:24:02.948681 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:24:02.948654 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92507c2a-dbae-47e0-96f7-d50a7ccbd312-workload-socket" pod "92507c2a-dbae-47e0-96f7-d50a7ccbd312" (UID: "92507c2a-dbae-47e0-96f7-d50a7ccbd312"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:24:02.948814 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:24:02.948721 5605 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92507c2a-dbae-47e0-96f7-d50a7ccbd312-istio-data\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:24:02.948814 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:24:02.948742 5605 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92507c2a-dbae-47e0-96f7-d50a7ccbd312-credential-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:24:02.948814 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:24:02.948759 5605 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92507c2a-dbae-47e0-96f7-d50a7ccbd312-workload-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:24:02.948814 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:24:02.948774 5605 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92507c2a-dbae-47e0-96f7-d50a7ccbd312-istiod-ca-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:24:02.950053 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:24:02.950022 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92507c2a-dbae-47e0-96f7-d50a7ccbd312-istio-envoy" pod "92507c2a-dbae-47e0-96f7-d50a7ccbd312" (UID: "92507c2a-dbae-47e0-96f7-d50a7ccbd312"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:24:02.950171 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:24:02.950063 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/92507c2a-dbae-47e0-96f7-d50a7ccbd312-istio-podinfo" pod "92507c2a-dbae-47e0-96f7-d50a7ccbd312" (UID: "92507c2a-dbae-47e0-96f7-d50a7ccbd312"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:24:02.950236 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:24:02.950201 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92507c2a-dbae-47e0-96f7-d50a7ccbd312-istio-token" pod "92507c2a-dbae-47e0-96f7-d50a7ccbd312" (UID: "92507c2a-dbae-47e0-96f7-d50a7ccbd312"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:24:02.950841 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:24:02.950818 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92507c2a-dbae-47e0-96f7-d50a7ccbd312-kube-api-access-frwwm" pod "92507c2a-dbae-47e0-96f7-d50a7ccbd312" (UID: "92507c2a-dbae-47e0-96f7-d50a7ccbd312"). InnerVolumeSpecName "kube-api-access-frwwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:24:03.049796 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:24:03.049768 5605 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92507c2a-dbae-47e0-96f7-d50a7ccbd312-istio-podinfo\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:24:03.049926 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:24:03.049797 5605 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92507c2a-dbae-47e0-96f7-d50a7ccbd312-istio-envoy\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:24:03.049926 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:24:03.049811 5605 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92507c2a-dbae-47e0-96f7-d50a7ccbd312-istio-token\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:24:03.049926 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:24:03.049822 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frwwm\" (UniqueName: \"kubernetes.io/projected/92507c2a-dbae-47e0-96f7-d50a7ccbd312-kube-api-access-frwwm\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:24:03.049926 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:24:03.049835 5605 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92507c2a-dbae-47e0-96f7-d50a7ccbd312-workload-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:24:03.478604 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:24:03.478577 5605 scope.go:122] "RemoveContainer" containerID="5d6a51e34152092feb4e7da6b0ca6e8c717b620de6ffe073271b83e1bed9eb0c" Nov 03 02:24:03.567196 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:24:03.567161 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jl8k2" event={"ID":"92507c2a-dbae-47e0-96f7-d50a7ccbd312","Type":"ContainerDied","Data":"5d6a51e34152092feb4e7da6b0ca6e8c717b620de6ffe073271b83e1bed9eb0c"} Nov 03 02:24:03.567293 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:24:03.567227 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jl8k2" event={"ID":"92507c2a-dbae-47e0-96f7-d50a7ccbd312","Type":"ContainerDied","Data":"f304fe693881b91ac4ef6bf0dd59002963d71b406b5573dc0b073e279cd28e3a"} Nov 03 02:24:03.567293 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:24:03.567181 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jl8k2" Nov 03 02:24:03.827903 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:24:03.827877 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jl8k2"] Nov 03 02:24:03.831301 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:24:03.831277 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jl8k2"] Nov 03 02:24:04.903947 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:24:04.903908 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92507c2a-dbae-47e0-96f7-d50a7ccbd312" path="/var/lib/kubelet/pods/92507c2a-dbae-47e0-96f7-d50a7ccbd312/volumes" Nov 03 02:24:23.914681 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:24:23.914641 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:24:44.871569 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:24:44.871534 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:24:53.913639 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:24:53.913591 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:25:17.051973 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:17.051892 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:25:19.933547 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:19.933508 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2"] Nov 03 02:25:19.934018 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:19.933714 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2" podUID="ae6b294b-6d5b-4bd7-820b-7e179011a2f8" containerName="istio-proxy" containerID="cri-o://3c69dd3953d802332e6cac7a63d9989e286cd77454f05713df39eec6d02f426d" gracePeriod=30 Nov 03 02:25:22.289519 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:22.289483 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s"] Nov 03 02:25:22.290013 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:22.289690 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s" podUID="b8c9203e-5102-452b-bbc7-af1b50ea3ec7" containerName="istio-proxy" containerID="cri-o://f6a7b2b01926b8282b17862cf0fcc588a6a8ca0086adfb62f8928d2a06255216" gracePeriod=30 Nov 03 02:25:23.649130 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:23.649080 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22"] Nov 03 02:25:23.649592 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:23.649319 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22" podUID="fdbfc7ca-1928-4b51-8967-cb5576ca80cb" containerName="istio-proxy" containerID="cri-o://4817d715e5eedfff169b9ee48028f6efbce0d42dc0a03116f95f99fe8448e052" gracePeriod=30 Nov 03 02:25:23.853094 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:23.853059 5605 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 02:25:23.853260 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:23.853112 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2" podUID="ae6b294b-6d5b-4bd7-820b-7e179011a2f8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 02:25:23.914907 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:25:23.914808 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:25:24.993524 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:24.993490 5605 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 02:25:24.993919 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:24.993554 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s" podUID="b8c9203e-5102-452b-bbc7-af1b50ea3ec7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 02:25:25.187137 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:25.187111 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2" Nov 03 02:25:25.213048 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:25.212960 5605 generic.go:356] "Generic (PLEG): container finished" podID="ae6b294b-6d5b-4bd7-820b-7e179011a2f8" containerID="3c69dd3953d802332e6cac7a63d9989e286cd77454f05713df39eec6d02f426d" exitCode=0 Nov 03 02:25:25.213048 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:25.213007 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2" Nov 03 02:25:25.213048 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:25.213027 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2" event={"ID":"ae6b294b-6d5b-4bd7-820b-7e179011a2f8","Type":"ContainerDied","Data":"3c69dd3953d802332e6cac7a63d9989e286cd77454f05713df39eec6d02f426d"} Nov 03 02:25:25.213296 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:25.213083 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2" event={"ID":"ae6b294b-6d5b-4bd7-820b-7e179011a2f8","Type":"ContainerDied","Data":"60a9404eec397b9e99c96cad05d507599a2520fa18f87246596015ee12df2ac2"} Nov 03 02:25:25.213296 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:25.213112 5605 scope.go:122] "RemoveContainer" containerID="3c69dd3953d802332e6cac7a63d9989e286cd77454f05713df39eec6d02f426d" Nov 03 02:25:25.222608 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:25.222591 5605 scope.go:122] "RemoveContainer" containerID="3c69dd3953d802332e6cac7a63d9989e286cd77454f05713df39eec6d02f426d" Nov 03 02:25:25.222879 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:25:25.222856 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c69dd3953d802332e6cac7a63d9989e286cd77454f05713df39eec6d02f426d\": container with ID starting with 3c69dd3953d802332e6cac7a63d9989e286cd77454f05713df39eec6d02f426d not found: ID does not exist" containerID="3c69dd3953d802332e6cac7a63d9989e286cd77454f05713df39eec6d02f426d" Nov 03 02:25:25.222916 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:25.222891 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c69dd3953d802332e6cac7a63d9989e286cd77454f05713df39eec6d02f426d"} err="failed to get container status \"3c69dd3953d802332e6cac7a63d9989e286cd77454f05713df39eec6d02f426d\": rpc error: code = NotFound desc = could not find container \"3c69dd3953d802332e6cac7a63d9989e286cd77454f05713df39eec6d02f426d\": container with ID starting with 3c69dd3953d802332e6cac7a63d9989e286cd77454f05713df39eec6d02f426d not found: ID does not exist" Nov 03 02:25:25.255312 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:25.255281 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae6b294b-6d5b-4bd7-820b-7e179011a2f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6b294b-6d5b-4bd7-820b-7e179011a2f8-credential-socket\") pod \"ae6b294b-6d5b-4bd7-820b-7e179011a2f8\" (UID: \"ae6b294b-6d5b-4bd7-820b-7e179011a2f8\") " Nov 03 02:25:25.255312 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:25.255317 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae6b294b-6d5b-4bd7-820b-7e179011a2f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6b294b-6d5b-4bd7-820b-7e179011a2f8-workload-socket\") pod \"ae6b294b-6d5b-4bd7-820b-7e179011a2f8\" (UID: \"ae6b294b-6d5b-4bd7-820b-7e179011a2f8\") " Nov 03 02:25:25.255538 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:25.255346 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae6b294b-6d5b-4bd7-820b-7e179011a2f8-istio-token\" (UniqueName: \"kubernetes.io/projected/ae6b294b-6d5b-4bd7-820b-7e179011a2f8-istio-token\") pod \"ae6b294b-6d5b-4bd7-820b-7e179011a2f8\" (UID: \"ae6b294b-6d5b-4bd7-820b-7e179011a2f8\") " Nov 03 02:25:25.255538 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:25.255401 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae6b294b-6d5b-4bd7-820b-7e179011a2f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae6b294b-6d5b-4bd7-820b-7e179011a2f8-istio-data\") pod \"ae6b294b-6d5b-4bd7-820b-7e179011a2f8\" (UID: \"ae6b294b-6d5b-4bd7-820b-7e179011a2f8\") " Nov 03 02:25:25.255538 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:25.255473 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae6b294b-6d5b-4bd7-820b-7e179011a2f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae6b294b-6d5b-4bd7-820b-7e179011a2f8-workload-certs\") pod \"ae6b294b-6d5b-4bd7-820b-7e179011a2f8\" (UID: \"ae6b294b-6d5b-4bd7-820b-7e179011a2f8\") " Nov 03 02:25:25.255678 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:25.255641 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae6b294b-6d5b-4bd7-820b-7e179011a2f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae6b294b-6d5b-4bd7-820b-7e179011a2f8-istio-envoy\") pod \"ae6b294b-6d5b-4bd7-820b-7e179011a2f8\" (UID: \"ae6b294b-6d5b-4bd7-820b-7e179011a2f8\") " Nov 03 02:25:25.255678 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:25.255647 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae6b294b-6d5b-4bd7-820b-7e179011a2f8-credential-socket" pod "ae6b294b-6d5b-4bd7-820b-7e179011a2f8" (UID: "ae6b294b-6d5b-4bd7-820b-7e179011a2f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:25:25.255678 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:25.255652 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae6b294b-6d5b-4bd7-820b-7e179011a2f8-workload-socket" pod "ae6b294b-6d5b-4bd7-820b-7e179011a2f8" (UID: "ae6b294b-6d5b-4bd7-820b-7e179011a2f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:25:25.255828 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:25.255708 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae6b294b-6d5b-4bd7-820b-7e179011a2f8-istio-data" pod "ae6b294b-6d5b-4bd7-820b-7e179011a2f8" (UID: "ae6b294b-6d5b-4bd7-820b-7e179011a2f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:25:25.255828 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:25.255737 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae6b294b-6d5b-4bd7-820b-7e179011a2f8-kube-api-access-llmt4\" (UniqueName: \"kubernetes.io/projected/ae6b294b-6d5b-4bd7-820b-7e179011a2f8-kube-api-access-llmt4\") pod \"ae6b294b-6d5b-4bd7-820b-7e179011a2f8\" (UID: \"ae6b294b-6d5b-4bd7-820b-7e179011a2f8\") " Nov 03 02:25:25.255828 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:25.255788 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae6b294b-6d5b-4bd7-820b-7e179011a2f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae6b294b-6d5b-4bd7-820b-7e179011a2f8-istiod-ca-cert\") pod \"ae6b294b-6d5b-4bd7-820b-7e179011a2f8\" (UID: \"ae6b294b-6d5b-4bd7-820b-7e179011a2f8\") " Nov 03 02:25:25.255828 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:25.255827 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ae6b294b-6d5b-4bd7-820b-7e179011a2f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae6b294b-6d5b-4bd7-820b-7e179011a2f8-istio-podinfo\") pod \"ae6b294b-6d5b-4bd7-820b-7e179011a2f8\" (UID: \"ae6b294b-6d5b-4bd7-820b-7e179011a2f8\") " Nov 03 02:25:25.256010 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:25.255841 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae6b294b-6d5b-4bd7-820b-7e179011a2f8-workload-certs" pod "ae6b294b-6d5b-4bd7-820b-7e179011a2f8" (UID: "ae6b294b-6d5b-4bd7-820b-7e179011a2f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:25:25.256170 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:25.256146 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae6b294b-6d5b-4bd7-820b-7e179011a2f8-istiod-ca-cert" pod "ae6b294b-6d5b-4bd7-820b-7e179011a2f8" (UID: "ae6b294b-6d5b-4bd7-820b-7e179011a2f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:25:25.256471 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:25.256450 5605 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae6b294b-6d5b-4bd7-820b-7e179011a2f8-istiod-ca-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:25.256552 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:25.256477 5605 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6b294b-6d5b-4bd7-820b-7e179011a2f8-credential-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:25.256552 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:25.256492 5605 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6b294b-6d5b-4bd7-820b-7e179011a2f8-workload-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:25.256552 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:25.256505 5605 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae6b294b-6d5b-4bd7-820b-7e179011a2f8-istio-data\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:25.256552 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:25.256518 5605 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae6b294b-6d5b-4bd7-820b-7e179011a2f8-workload-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:25.257771 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:25.257743 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae6b294b-6d5b-4bd7-820b-7e179011a2f8-istio-token" pod "ae6b294b-6d5b-4bd7-820b-7e179011a2f8" (UID: "ae6b294b-6d5b-4bd7-820b-7e179011a2f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:25:25.257898 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:25.257870 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae6b294b-6d5b-4bd7-820b-7e179011a2f8-istio-podinfo" pod "ae6b294b-6d5b-4bd7-820b-7e179011a2f8" (UID: "ae6b294b-6d5b-4bd7-820b-7e179011a2f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:25:25.258154 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:25.258137 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae6b294b-6d5b-4bd7-820b-7e179011a2f8-kube-api-access-llmt4" pod "ae6b294b-6d5b-4bd7-820b-7e179011a2f8" (UID: "ae6b294b-6d5b-4bd7-820b-7e179011a2f8"). InnerVolumeSpecName "kube-api-access-llmt4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:25:25.258205 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:25.258175 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae6b294b-6d5b-4bd7-820b-7e179011a2f8-istio-envoy" pod "ae6b294b-6d5b-4bd7-820b-7e179011a2f8" (UID: "ae6b294b-6d5b-4bd7-820b-7e179011a2f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:25:25.357916 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:25.357872 5605 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae6b294b-6d5b-4bd7-820b-7e179011a2f8-istio-envoy\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:25.357916 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:25.357906 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llmt4\" (UniqueName: \"kubernetes.io/projected/ae6b294b-6d5b-4bd7-820b-7e179011a2f8-kube-api-access-llmt4\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:25.357916 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:25.357922 5605 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae6b294b-6d5b-4bd7-820b-7e179011a2f8-istio-podinfo\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:25.358202 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:25.357934 5605 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae6b294b-6d5b-4bd7-820b-7e179011a2f8-istio-token\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:25.578339 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:25.578299 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf"] Nov 03 02:25:25.578612 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:25.578585 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf" podUID="cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d" containerName="istio-proxy" containerID="cri-o://643d31b51ebc8524645f42031921943ab9e4f46c5fec97cae44d153343ab555f" gracePeriod=30 Nov 03 02:25:25.580468 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:25.580430 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b"] Nov 03 02:25:25.580652 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:25.580617 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b" podUID="338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17" containerName="istio-proxy" containerID="cri-o://7599139daa8c656f02beccfaf750a431ba0ed088a5ed594b1c33c9eea7a7c8f1" gracePeriod=30 Nov 03 02:25:25.626707 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:25.626673 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2"] Nov 03 02:25:25.632859 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:25.632832 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-4zwb2"] Nov 03 02:25:26.880979 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:26.880939 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae6b294b-6d5b-4bd7-820b-7e179011a2f8" path="/var/lib/kubelet/pods/ae6b294b-6d5b-4bd7-820b-7e179011a2f8/volumes" Nov 03 02:25:27.548319 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:27.548290 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s" Nov 03 02:25:27.599144 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:27.599069 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8c9203e-5102-452b-bbc7-af1b50ea3ec7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8c9203e-5102-452b-bbc7-af1b50ea3ec7-istio-envoy\") pod \"b8c9203e-5102-452b-bbc7-af1b50ea3ec7\" (UID: \"b8c9203e-5102-452b-bbc7-af1b50ea3ec7\") " Nov 03 02:25:27.599144 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:27.599108 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8c9203e-5102-452b-bbc7-af1b50ea3ec7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8c9203e-5102-452b-bbc7-af1b50ea3ec7-credential-socket\") pod \"b8c9203e-5102-452b-bbc7-af1b50ea3ec7\" (UID: \"b8c9203e-5102-452b-bbc7-af1b50ea3ec7\") " Nov 03 02:25:27.599318 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:27.599153 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8c9203e-5102-452b-bbc7-af1b50ea3ec7-kube-api-access-kxhh7\" (UniqueName: \"kubernetes.io/projected/b8c9203e-5102-452b-bbc7-af1b50ea3ec7-kube-api-access-kxhh7\") pod \"b8c9203e-5102-452b-bbc7-af1b50ea3ec7\" (UID: \"b8c9203e-5102-452b-bbc7-af1b50ea3ec7\") " Nov 03 02:25:27.599318 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:27.599203 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8c9203e-5102-452b-bbc7-af1b50ea3ec7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8c9203e-5102-452b-bbc7-af1b50ea3ec7-istiod-ca-cert\") pod \"b8c9203e-5102-452b-bbc7-af1b50ea3ec7\" (UID: \"b8c9203e-5102-452b-bbc7-af1b50ea3ec7\") " Nov 03 02:25:27.599318 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:27.599232 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8c9203e-5102-452b-bbc7-af1b50ea3ec7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8c9203e-5102-452b-bbc7-af1b50ea3ec7-istio-data\") pod \"b8c9203e-5102-452b-bbc7-af1b50ea3ec7\" (UID: \"b8c9203e-5102-452b-bbc7-af1b50ea3ec7\") " Nov 03 02:25:27.599318 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:27.599278 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b8c9203e-5102-452b-bbc7-af1b50ea3ec7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8c9203e-5102-452b-bbc7-af1b50ea3ec7-istio-podinfo\") pod \"b8c9203e-5102-452b-bbc7-af1b50ea3ec7\" (UID: \"b8c9203e-5102-452b-bbc7-af1b50ea3ec7\") " Nov 03 02:25:27.599541 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:27.599332 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8c9203e-5102-452b-bbc7-af1b50ea3ec7-istio-token\" (UniqueName: \"kubernetes.io/projected/b8c9203e-5102-452b-bbc7-af1b50ea3ec7-istio-token\") pod \"b8c9203e-5102-452b-bbc7-af1b50ea3ec7\" (UID: \"b8c9203e-5102-452b-bbc7-af1b50ea3ec7\") " Nov 03 02:25:27.599541 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:27.599438 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8c9203e-5102-452b-bbc7-af1b50ea3ec7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8c9203e-5102-452b-bbc7-af1b50ea3ec7-workload-certs\") pod \"b8c9203e-5102-452b-bbc7-af1b50ea3ec7\" (UID: \"b8c9203e-5102-452b-bbc7-af1b50ea3ec7\") " Nov 03 02:25:27.599541 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:27.599487 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8c9203e-5102-452b-bbc7-af1b50ea3ec7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8c9203e-5102-452b-bbc7-af1b50ea3ec7-workload-socket\") pod \"b8c9203e-5102-452b-bbc7-af1b50ea3ec7\" (UID: \"b8c9203e-5102-452b-bbc7-af1b50ea3ec7\") " Nov 03 02:25:27.599696 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:27.599532 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8c9203e-5102-452b-bbc7-af1b50ea3ec7-credential-socket" pod "b8c9203e-5102-452b-bbc7-af1b50ea3ec7" (UID: "b8c9203e-5102-452b-bbc7-af1b50ea3ec7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:25:27.599757 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:27.599736 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8c9203e-5102-452b-bbc7-af1b50ea3ec7-istio-data" pod "b8c9203e-5102-452b-bbc7-af1b50ea3ec7" (UID: "b8c9203e-5102-452b-bbc7-af1b50ea3ec7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:25:27.599814 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:27.599749 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8c9203e-5102-452b-bbc7-af1b50ea3ec7-workload-certs" pod "b8c9203e-5102-452b-bbc7-af1b50ea3ec7" (UID: "b8c9203e-5102-452b-bbc7-af1b50ea3ec7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:25:27.599984 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:27.599949 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8c9203e-5102-452b-bbc7-af1b50ea3ec7-workload-socket" pod "b8c9203e-5102-452b-bbc7-af1b50ea3ec7" (UID: "b8c9203e-5102-452b-bbc7-af1b50ea3ec7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:25:27.600096 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:27.599981 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8c9203e-5102-452b-bbc7-af1b50ea3ec7-istiod-ca-cert" pod "b8c9203e-5102-452b-bbc7-af1b50ea3ec7" (UID: "b8c9203e-5102-452b-bbc7-af1b50ea3ec7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:25:27.600096 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:27.600049 5605 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8c9203e-5102-452b-bbc7-af1b50ea3ec7-credential-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:27.600096 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:27.600072 5605 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8c9203e-5102-452b-bbc7-af1b50ea3ec7-istio-data\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:27.600096 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:27.600088 5605 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8c9203e-5102-452b-bbc7-af1b50ea3ec7-workload-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:27.600308 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:27.600102 5605 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8c9203e-5102-452b-bbc7-af1b50ea3ec7-workload-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:27.601447 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:27.601419 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8c9203e-5102-452b-bbc7-af1b50ea3ec7-istio-podinfo" pod "b8c9203e-5102-452b-bbc7-af1b50ea3ec7" (UID: "b8c9203e-5102-452b-bbc7-af1b50ea3ec7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:25:27.601571 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:27.601467 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8c9203e-5102-452b-bbc7-af1b50ea3ec7-kube-api-access-kxhh7" pod "b8c9203e-5102-452b-bbc7-af1b50ea3ec7" (UID: "b8c9203e-5102-452b-bbc7-af1b50ea3ec7"). InnerVolumeSpecName "kube-api-access-kxhh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:25:27.601571 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:27.601523 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8c9203e-5102-452b-bbc7-af1b50ea3ec7-istio-token" pod "b8c9203e-5102-452b-bbc7-af1b50ea3ec7" (UID: "b8c9203e-5102-452b-bbc7-af1b50ea3ec7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:25:27.601571 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:27.601541 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8c9203e-5102-452b-bbc7-af1b50ea3ec7-istio-envoy" pod "b8c9203e-5102-452b-bbc7-af1b50ea3ec7" (UID: "b8c9203e-5102-452b-bbc7-af1b50ea3ec7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:25:27.701830 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:27.701791 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxhh7\" (UniqueName: \"kubernetes.io/projected/b8c9203e-5102-452b-bbc7-af1b50ea3ec7-kube-api-access-kxhh7\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:27.701830 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:27.701823 5605 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8c9203e-5102-452b-bbc7-af1b50ea3ec7-istiod-ca-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:27.701830 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:27.701834 5605 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8c9203e-5102-452b-bbc7-af1b50ea3ec7-istio-podinfo\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:27.702057 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:27.701842 5605 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8c9203e-5102-452b-bbc7-af1b50ea3ec7-istio-token\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:27.702057 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:27.701853 5605 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8c9203e-5102-452b-bbc7-af1b50ea3ec7-istio-envoy\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:28.231455 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:28.231416 5605 generic.go:356] "Generic (PLEG): container finished" podID="b8c9203e-5102-452b-bbc7-af1b50ea3ec7" containerID="f6a7b2b01926b8282b17862cf0fcc588a6a8ca0086adfb62f8928d2a06255216" exitCode=0 Nov 03 02:25:28.231936 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:28.231470 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s" Nov 03 02:25:28.231936 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:28.231499 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s" event={"ID":"b8c9203e-5102-452b-bbc7-af1b50ea3ec7","Type":"ContainerDied","Data":"f6a7b2b01926b8282b17862cf0fcc588a6a8ca0086adfb62f8928d2a06255216"} Nov 03 02:25:28.231936 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:28.231565 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s" event={"ID":"b8c9203e-5102-452b-bbc7-af1b50ea3ec7","Type":"ContainerDied","Data":"30aa9a73a05b703100900422a7eb19e61e8cc69ed499514918555711edd547fe"} Nov 03 02:25:28.231936 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:28.231594 5605 scope.go:122] "RemoveContainer" containerID="f6a7b2b01926b8282b17862cf0fcc588a6a8ca0086adfb62f8928d2a06255216" Nov 03 02:25:28.241617 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:28.241596 5605 scope.go:122] "RemoveContainer" containerID="f6a7b2b01926b8282b17862cf0fcc588a6a8ca0086adfb62f8928d2a06255216" Nov 03 02:25:28.241895 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:25:28.241875 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6a7b2b01926b8282b17862cf0fcc588a6a8ca0086adfb62f8928d2a06255216\": container with ID starting with f6a7b2b01926b8282b17862cf0fcc588a6a8ca0086adfb62f8928d2a06255216 not found: ID does not exist" containerID="f6a7b2b01926b8282b17862cf0fcc588a6a8ca0086adfb62f8928d2a06255216" Nov 03 02:25:28.241987 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:28.241908 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6a7b2b01926b8282b17862cf0fcc588a6a8ca0086adfb62f8928d2a06255216"} err="failed to get container status \"f6a7b2b01926b8282b17862cf0fcc588a6a8ca0086adfb62f8928d2a06255216\": rpc error: code = NotFound desc = could not find container \"f6a7b2b01926b8282b17862cf0fcc588a6a8ca0086adfb62f8928d2a06255216\": container with ID starting with f6a7b2b01926b8282b17862cf0fcc588a6a8ca0086adfb62f8928d2a06255216 not found: ID does not exist" Nov 03 02:25:28.408860 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:28.408818 5605 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 02:25:28.409044 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:28.408881 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22" podUID="fdbfc7ca-1928-4b51-8967-cb5576ca80cb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 02:25:28.448911 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:28.448862 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s"] Nov 03 02:25:28.494076 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:28.493994 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-qxc4s"] Nov 03 02:25:28.850401 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:28.850350 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8c9203e-5102-452b-bbc7-af1b50ea3ec7" path="/var/lib/kubelet/pods/b8c9203e-5102-452b-bbc7-af1b50ea3ec7/volumes" Nov 03 02:25:28.940278 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:28.940251 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22" Nov 03 02:25:29.018028 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:29.017993 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdbfc7ca-1928-4b51-8967-cb5576ca80cb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdbfc7ca-1928-4b51-8967-cb5576ca80cb-istio-envoy\") pod \"fdbfc7ca-1928-4b51-8967-cb5576ca80cb\" (UID: \"fdbfc7ca-1928-4b51-8967-cb5576ca80cb\") " Nov 03 02:25:29.018237 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:29.018049 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdbfc7ca-1928-4b51-8967-cb5576ca80cb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdbfc7ca-1928-4b51-8967-cb5576ca80cb-istiod-ca-cert\") pod \"fdbfc7ca-1928-4b51-8967-cb5576ca80cb\" (UID: \"fdbfc7ca-1928-4b51-8967-cb5576ca80cb\") " Nov 03 02:25:29.018237 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:29.018073 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdbfc7ca-1928-4b51-8967-cb5576ca80cb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdbfc7ca-1928-4b51-8967-cb5576ca80cb-credential-socket\") pod \"fdbfc7ca-1928-4b51-8967-cb5576ca80cb\" (UID: \"fdbfc7ca-1928-4b51-8967-cb5576ca80cb\") " Nov 03 02:25:29.018237 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:29.018123 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdbfc7ca-1928-4b51-8967-cb5576ca80cb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdbfc7ca-1928-4b51-8967-cb5576ca80cb-workload-certs\") pod \"fdbfc7ca-1928-4b51-8967-cb5576ca80cb\" (UID: \"fdbfc7ca-1928-4b51-8967-cb5576ca80cb\") " Nov 03 02:25:29.018406 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:29.018269 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdbfc7ca-1928-4b51-8967-cb5576ca80cb-kube-api-access-j86gx\" (UniqueName: \"kubernetes.io/projected/fdbfc7ca-1928-4b51-8967-cb5576ca80cb-kube-api-access-j86gx\") pod \"fdbfc7ca-1928-4b51-8967-cb5576ca80cb\" (UID: \"fdbfc7ca-1928-4b51-8967-cb5576ca80cb\") " Nov 03 02:25:29.018406 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:29.018320 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fdbfc7ca-1928-4b51-8967-cb5576ca80cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdbfc7ca-1928-4b51-8967-cb5576ca80cb-istio-podinfo\") pod \"fdbfc7ca-1928-4b51-8967-cb5576ca80cb\" (UID: \"fdbfc7ca-1928-4b51-8967-cb5576ca80cb\") " Nov 03 02:25:29.018515 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:29.018411 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdbfc7ca-1928-4b51-8967-cb5576ca80cb-workload-certs" pod "fdbfc7ca-1928-4b51-8967-cb5576ca80cb" (UID: "fdbfc7ca-1928-4b51-8967-cb5576ca80cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:25:29.018515 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:29.018427 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdbfc7ca-1928-4b51-8967-cb5576ca80cb-credential-socket" pod "fdbfc7ca-1928-4b51-8967-cb5576ca80cb" (UID: "fdbfc7ca-1928-4b51-8967-cb5576ca80cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:25:29.018515 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:29.018453 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdbfc7ca-1928-4b51-8967-cb5576ca80cb-istio-token\" (UniqueName: \"kubernetes.io/projected/fdbfc7ca-1928-4b51-8967-cb5576ca80cb-istio-token\") pod \"fdbfc7ca-1928-4b51-8967-cb5576ca80cb\" (UID: \"fdbfc7ca-1928-4b51-8967-cb5576ca80cb\") " Nov 03 02:25:29.018515 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:29.018500 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdbfc7ca-1928-4b51-8967-cb5576ca80cb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdbfc7ca-1928-4b51-8967-cb5576ca80cb-istio-data\") pod \"fdbfc7ca-1928-4b51-8967-cb5576ca80cb\" (UID: \"fdbfc7ca-1928-4b51-8967-cb5576ca80cb\") " Nov 03 02:25:29.018710 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:29.018525 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdbfc7ca-1928-4b51-8967-cb5576ca80cb-istiod-ca-cert" pod "fdbfc7ca-1928-4b51-8967-cb5576ca80cb" (UID: "fdbfc7ca-1928-4b51-8967-cb5576ca80cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:25:29.018710 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:29.018532 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdbfc7ca-1928-4b51-8967-cb5576ca80cb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdbfc7ca-1928-4b51-8967-cb5576ca80cb-workload-socket\") pod \"fdbfc7ca-1928-4b51-8967-cb5576ca80cb\" (UID: \"fdbfc7ca-1928-4b51-8967-cb5576ca80cb\") " Nov 03 02:25:29.018811 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:29.018788 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdbfc7ca-1928-4b51-8967-cb5576ca80cb-istio-data" pod "fdbfc7ca-1928-4b51-8967-cb5576ca80cb" (UID: "fdbfc7ca-1928-4b51-8967-cb5576ca80cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:25:29.018860 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:29.018834 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdbfc7ca-1928-4b51-8967-cb5576ca80cb-workload-socket" pod "fdbfc7ca-1928-4b51-8967-cb5576ca80cb" (UID: "fdbfc7ca-1928-4b51-8967-cb5576ca80cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:25:29.019063 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:29.019040 5605 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdbfc7ca-1928-4b51-8967-cb5576ca80cb-workload-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:29.019137 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:29.019067 5605 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdbfc7ca-1928-4b51-8967-cb5576ca80cb-istiod-ca-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:29.019137 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:29.019082 5605 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdbfc7ca-1928-4b51-8967-cb5576ca80cb-credential-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:29.019137 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:29.019096 5605 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdbfc7ca-1928-4b51-8967-cb5576ca80cb-workload-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:29.019137 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:29.019110 5605 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdbfc7ca-1928-4b51-8967-cb5576ca80cb-istio-data\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:29.020403 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:29.020347 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdbfc7ca-1928-4b51-8967-cb5576ca80cb-istio-envoy" pod "fdbfc7ca-1928-4b51-8967-cb5576ca80cb" (UID: "fdbfc7ca-1928-4b51-8967-cb5576ca80cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:25:29.020596 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:29.020574 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdbfc7ca-1928-4b51-8967-cb5576ca80cb-istio-token" pod "fdbfc7ca-1928-4b51-8967-cb5576ca80cb" (UID: "fdbfc7ca-1928-4b51-8967-cb5576ca80cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:25:29.020828 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:29.020801 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fdbfc7ca-1928-4b51-8967-cb5576ca80cb-istio-podinfo" pod "fdbfc7ca-1928-4b51-8967-cb5576ca80cb" (UID: "fdbfc7ca-1928-4b51-8967-cb5576ca80cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:25:29.020944 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:29.020856 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdbfc7ca-1928-4b51-8967-cb5576ca80cb-kube-api-access-j86gx" pod "fdbfc7ca-1928-4b51-8967-cb5576ca80cb" (UID: "fdbfc7ca-1928-4b51-8967-cb5576ca80cb"). InnerVolumeSpecName "kube-api-access-j86gx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:25:29.120122 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:29.120083 5605 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdbfc7ca-1928-4b51-8967-cb5576ca80cb-istio-token\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:29.120122 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:29.120115 5605 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdbfc7ca-1928-4b51-8967-cb5576ca80cb-istio-envoy\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:29.120122 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:29.120125 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j86gx\" (UniqueName: \"kubernetes.io/projected/fdbfc7ca-1928-4b51-8967-cb5576ca80cb-kube-api-access-j86gx\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:29.120420 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:29.120136 5605 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdbfc7ca-1928-4b51-8967-cb5576ca80cb-istio-podinfo\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:29.238202 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:29.238160 5605 generic.go:356] "Generic (PLEG): container finished" podID="fdbfc7ca-1928-4b51-8967-cb5576ca80cb" containerID="4817d715e5eedfff169b9ee48028f6efbce0d42dc0a03116f95f99fe8448e052" exitCode=0 Nov 03 02:25:29.238690 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:29.238216 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22" Nov 03 02:25:29.238690 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:29.238245 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22" event={"ID":"fdbfc7ca-1928-4b51-8967-cb5576ca80cb","Type":"ContainerDied","Data":"4817d715e5eedfff169b9ee48028f6efbce0d42dc0a03116f95f99fe8448e052"} Nov 03 02:25:29.238690 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:29.238302 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22" event={"ID":"fdbfc7ca-1928-4b51-8967-cb5576ca80cb","Type":"ContainerDied","Data":"2468a114429c5f57ed35b0f11cdfb99aac9555b29c93d383d880d82430cde7d3"} Nov 03 02:25:29.238690 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:29.238325 5605 scope.go:122] "RemoveContainer" containerID="4817d715e5eedfff169b9ee48028f6efbce0d42dc0a03116f95f99fe8448e052" Nov 03 02:25:29.248135 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:29.248114 5605 scope.go:122] "RemoveContainer" containerID="4817d715e5eedfff169b9ee48028f6efbce0d42dc0a03116f95f99fe8448e052" Nov 03 02:25:29.248489 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:25:29.248461 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4817d715e5eedfff169b9ee48028f6efbce0d42dc0a03116f95f99fe8448e052\": container with ID starting with 4817d715e5eedfff169b9ee48028f6efbce0d42dc0a03116f95f99fe8448e052 not found: ID does not exist" containerID="4817d715e5eedfff169b9ee48028f6efbce0d42dc0a03116f95f99fe8448e052" Nov 03 02:25:29.248543 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:29.248496 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4817d715e5eedfff169b9ee48028f6efbce0d42dc0a03116f95f99fe8448e052"} err="failed to get container status \"4817d715e5eedfff169b9ee48028f6efbce0d42dc0a03116f95f99fe8448e052\": rpc error: code = NotFound desc = could not find container \"4817d715e5eedfff169b9ee48028f6efbce0d42dc0a03116f95f99fe8448e052\": container with ID starting with 4817d715e5eedfff169b9ee48028f6efbce0d42dc0a03116f95f99fe8448e052 not found: ID does not exist" Nov 03 02:25:29.391183 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:29.391098 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22"] Nov 03 02:25:29.395293 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:29.395258 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xnv22"] Nov 03 02:25:30.834237 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:30.834205 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf" Nov 03 02:25:30.849998 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:30.849922 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdbfc7ca-1928-4b51-8967-cb5576ca80cb" path="/var/lib/kubelet/pods/fdbfc7ca-1928-4b51-8967-cb5576ca80cb/volumes" Nov 03 02:25:30.900994 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:30.900969 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b" Nov 03 02:25:30.944095 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:30.944062 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d-istio-data\") pod \"cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d\" (UID: \"cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d\") " Nov 03 02:25:30.944230 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:30.944124 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17-istio-podinfo\") pod \"338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17\" (UID: \"338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17\") " Nov 03 02:25:30.944230 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:30.944158 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17-istio-data\") pod \"338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17\" (UID: \"338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17\") " Nov 03 02:25:30.944230 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:30.944191 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17-kube-api-access-ddwdx\" (UniqueName: \"kubernetes.io/projected/338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17-kube-api-access-ddwdx\") pod \"338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17\" (UID: \"338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17\") " Nov 03 02:25:30.944230 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:30.944221 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d-kube-api-access-zjtct\" (UniqueName: \"kubernetes.io/projected/cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d-kube-api-access-zjtct\") pod \"cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d\" (UID: \"cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d\") " Nov 03 02:25:30.944502 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:30.944264 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17-istiod-ca-cert\") pod \"338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17\" (UID: \"338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17\") " Nov 03 02:25:30.944502 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:30.944313 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d-workload-certs\") pod \"cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d\" (UID: \"cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d\") " Nov 03 02:25:30.944502 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:30.944391 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d-istio-token\" (UniqueName: \"kubernetes.io/projected/cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d-istio-token\") pod \"cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d\" (UID: \"cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d\") " Nov 03 02:25:30.944502 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:30.944402 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d-istio-data" pod "cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d" (UID: "cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:25:30.944502 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:30.944463 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d-istio-podinfo\") pod \"cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d\" (UID: \"cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d\") " Nov 03 02:25:30.944502 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:30.944469 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17-istio-data" pod "338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17" (UID: "338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:25:30.944797 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:30.944514 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17-credential-socket\") pod \"338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17\" (UID: \"338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17\") " Nov 03 02:25:30.944797 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:30.944554 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d-credential-socket\") pod \"cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d\" (UID: \"cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d\") " Nov 03 02:25:30.944797 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:30.944626 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d-istiod-ca-cert\") pod \"cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d\" (UID: \"cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d\") " Nov 03 02:25:30.944797 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:30.944675 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d-workload-socket\") pod \"cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d\" (UID: \"cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d\") " Nov 03 02:25:30.944797 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:30.944703 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17-istio-envoy\") pod \"338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17\" (UID: \"338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17\") " Nov 03 02:25:30.944797 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:30.944746 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17-workload-certs\") pod \"338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17\" (UID: \"338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17\") " Nov 03 02:25:30.944797 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:30.944795 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17-workload-socket\") pod \"338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17\" (UID: \"338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17\") " Nov 03 02:25:30.945130 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:30.944826 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d-istio-envoy\") pod \"cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d\" (UID: \"cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d\") " Nov 03 02:25:30.945130 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:30.944825 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d-workload-certs" pod "cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d" (UID: "cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:25:30.945130 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:30.944835 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17-credential-socket" pod "338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17" (UID: "338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:25:30.945130 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:30.944892 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17-istio-token\" (UniqueName: \"kubernetes.io/projected/338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17-istio-token\") pod \"338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17\" (UID: \"338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17\") " Nov 03 02:25:30.945130 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:30.945066 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d-credential-socket" pod "cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d" (UID: "cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:25:30.945391 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:30.945263 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17-istiod-ca-cert" pod "338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17" (UID: "338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:25:30.945455 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:30.945386 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d-istiod-ca-cert" pod "cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d" (UID: "cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:25:30.945950 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:30.945674 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17-workload-certs" pod "338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17" (UID: "338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:25:30.945950 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:30.945817 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17-workload-socket" pod "338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17" (UID: "338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:25:30.945950 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:30.945532 5605 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d-istio-data\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:30.945950 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:30.945899 5605 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17-istio-data\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:30.945950 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:30.945915 5605 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17-istiod-ca-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:30.945950 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:30.945918 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d-workload-socket" pod "cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d" (UID: "cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:25:30.945950 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:30.945928 5605 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d-workload-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:30.946382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:30.945973 5605 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17-credential-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:30.946382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:30.945992 5605 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d-credential-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:30.946382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:30.946009 5605 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d-istiod-ca-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:30.947434 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:30.946971 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d-istio-token" pod "cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d" (UID: "cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:25:30.947434 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:30.947340 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17-istio-podinfo" pod "338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17" (UID: "338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:25:30.947434 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:30.947348 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17-kube-api-access-ddwdx" pod "338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17" (UID: "338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17"). InnerVolumeSpecName "kube-api-access-ddwdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:25:30.947677 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:30.947483 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17-istio-token" pod "338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17" (UID: "338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:25:30.947911 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:30.947893 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d-istio-envoy" pod "cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d" (UID: "cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:25:30.948085 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:30.948070 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17-istio-envoy" pod "338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17" (UID: "338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:25:30.948770 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:30.948749 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d-kube-api-access-zjtct" pod "cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d" (UID: "cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d"). InnerVolumeSpecName "kube-api-access-zjtct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:25:30.948848 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:30.948777 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d-istio-podinfo" pod "cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d" (UID: "cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:25:31.046748 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:31.046715 5605 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d-istio-token\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:31.046748 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:31.046743 5605 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d-istio-podinfo\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:31.046748 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:31.046755 5605 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d-workload-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:31.046977 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:31.046764 5605 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17-istio-envoy\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:31.046977 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:31.046773 5605 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17-workload-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:31.046977 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:31.046785 5605 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17-workload-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:31.046977 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:31.046793 5605 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d-istio-envoy\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:31.046977 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:31.046800 5605 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17-istio-token\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:31.046977 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:31.046810 5605 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17-istio-podinfo\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:31.046977 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:31.046819 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddwdx\" (UniqueName: \"kubernetes.io/projected/338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17-kube-api-access-ddwdx\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:31.046977 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:31.046828 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjtct\" (UniqueName: \"kubernetes.io/projected/cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d-kube-api-access-zjtct\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:31.252520 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:31.252371 5605 generic.go:356] "Generic (PLEG): container finished" podID="338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17" containerID="7599139daa8c656f02beccfaf750a431ba0ed088a5ed594b1c33c9eea7a7c8f1" exitCode=0 Nov 03 02:25:31.252520 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:31.252443 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b" Nov 03 02:25:31.252520 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:31.252445 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b" event={"ID":"338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17","Type":"ContainerDied","Data":"7599139daa8c656f02beccfaf750a431ba0ed088a5ed594b1c33c9eea7a7c8f1"} Nov 03 02:25:31.252520 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:31.252504 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b" event={"ID":"338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17","Type":"ContainerDied","Data":"0b8d8e60ca95699795154e6dab5debacc39728e3397b20de28595659fedbe846"} Nov 03 02:25:31.252839 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:31.252528 5605 scope.go:122] "RemoveContainer" containerID="7599139daa8c656f02beccfaf750a431ba0ed088a5ed594b1c33c9eea7a7c8f1" Nov 03 02:25:31.254015 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:31.253990 5605 generic.go:356] "Generic (PLEG): container finished" podID="cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d" containerID="643d31b51ebc8524645f42031921943ab9e4f46c5fec97cae44d153343ab555f" exitCode=0 Nov 03 02:25:31.254145 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:31.254037 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf" Nov 03 02:25:31.254145 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:31.254087 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf" event={"ID":"cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d","Type":"ContainerDied","Data":"643d31b51ebc8524645f42031921943ab9e4f46c5fec97cae44d153343ab555f"} Nov 03 02:25:31.254257 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:31.254159 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf" event={"ID":"cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d","Type":"ContainerDied","Data":"9806cd2a0e192b87e5adfcd80422dbb2d382a30375ae3d06ade6cb8bec89282d"} Nov 03 02:25:31.264292 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:31.264267 5605 scope.go:122] "RemoveContainer" containerID="7599139daa8c656f02beccfaf750a431ba0ed088a5ed594b1c33c9eea7a7c8f1" Nov 03 02:25:31.264730 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:25:31.264705 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7599139daa8c656f02beccfaf750a431ba0ed088a5ed594b1c33c9eea7a7c8f1\": container with ID starting with 7599139daa8c656f02beccfaf750a431ba0ed088a5ed594b1c33c9eea7a7c8f1 not found: ID does not exist" containerID="7599139daa8c656f02beccfaf750a431ba0ed088a5ed594b1c33c9eea7a7c8f1" Nov 03 02:25:31.264829 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:31.264743 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7599139daa8c656f02beccfaf750a431ba0ed088a5ed594b1c33c9eea7a7c8f1"} err="failed to get container status \"7599139daa8c656f02beccfaf750a431ba0ed088a5ed594b1c33c9eea7a7c8f1\": rpc error: code = NotFound desc = could not find container \"7599139daa8c656f02beccfaf750a431ba0ed088a5ed594b1c33c9eea7a7c8f1\": container with ID starting with 7599139daa8c656f02beccfaf750a431ba0ed088a5ed594b1c33c9eea7a7c8f1 not found: ID does not exist" Nov 03 02:25:31.264829 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:31.264768 5605 scope.go:122] "RemoveContainer" containerID="643d31b51ebc8524645f42031921943ab9e4f46c5fec97cae44d153343ab555f" Nov 03 02:25:31.274607 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:31.274405 5605 scope.go:122] "RemoveContainer" containerID="643d31b51ebc8524645f42031921943ab9e4f46c5fec97cae44d153343ab555f" Nov 03 02:25:31.274772 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:25:31.274748 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"643d31b51ebc8524645f42031921943ab9e4f46c5fec97cae44d153343ab555f\": container with ID starting with 643d31b51ebc8524645f42031921943ab9e4f46c5fec97cae44d153343ab555f not found: ID does not exist" containerID="643d31b51ebc8524645f42031921943ab9e4f46c5fec97cae44d153343ab555f" Nov 03 02:25:31.274831 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:31.274782 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"643d31b51ebc8524645f42031921943ab9e4f46c5fec97cae44d153343ab555f"} err="failed to get container status \"643d31b51ebc8524645f42031921943ab9e4f46c5fec97cae44d153343ab555f\": rpc error: code = NotFound desc = could not find container \"643d31b51ebc8524645f42031921943ab9e4f46c5fec97cae44d153343ab555f\": container with ID starting with 643d31b51ebc8524645f42031921943ab9e4f46c5fec97cae44d153343ab555f not found: ID does not exist" Nov 03 02:25:31.407334 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:31.407295 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b"] Nov 03 02:25:31.412919 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:31.412895 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sbf5b"] Nov 03 02:25:31.514601 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:31.514520 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf"] Nov 03 02:25:31.518721 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:31.518692 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-pn4kf"] Nov 03 02:25:32.850829 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:32.850796 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17" path="/var/lib/kubelet/pods/338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17/volumes" Nov 03 02:25:32.851275 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:32.851260 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d" path="/var/lib/kubelet/pods/cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d/volumes" Nov 03 02:25:42.775017 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:42.774974 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f760d3-984756bb7-5v2b6"] Nov 03 02:25:42.775470 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:42.775454 5605 cpu_manager.go:414] "containerMap: removing container" podUID="fdbfc7ca-1928-4b51-8967-cb5576ca80cb" containerName="istio-proxy" Nov 03 02:25:42.775540 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:42.775472 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdbfc7ca-1928-4b51-8967-cb5576ca80cb" containerName="istio-proxy" Nov 03 02:25:42.775540 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:42.775487 5605 cpu_manager.go:414] "containerMap: removing container" podUID="92507c2a-dbae-47e0-96f7-d50a7ccbd312" containerName="istio-proxy" Nov 03 02:25:42.775540 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:42.775495 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92507c2a-dbae-47e0-96f7-d50a7ccbd312" containerName="istio-proxy" Nov 03 02:25:42.775540 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:42.775518 5605 cpu_manager.go:414] "containerMap: removing container" podUID="338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17" containerName="istio-proxy" Nov 03 02:25:42.775540 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:42.775526 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17" containerName="istio-proxy" Nov 03 02:25:42.775724 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:42.775544 5605 cpu_manager.go:414] "containerMap: removing container" podUID="b8c9203e-5102-452b-bbc7-af1b50ea3ec7" containerName="istio-proxy" Nov 03 02:25:42.775724 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:42.775551 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8c9203e-5102-452b-bbc7-af1b50ea3ec7" containerName="istio-proxy" Nov 03 02:25:42.775724 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:42.775572 5605 cpu_manager.go:414] "containerMap: removing container" podUID="8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd" containerName="istio-proxy" Nov 03 02:25:42.775724 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:42.775580 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd" containerName="istio-proxy" Nov 03 02:25:42.775724 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:42.775592 5605 cpu_manager.go:414] "containerMap: removing container" podUID="cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d" containerName="istio-proxy" Nov 03 02:25:42.775724 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:42.775599 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d" containerName="istio-proxy" Nov 03 02:25:42.775724 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:42.775621 5605 cpu_manager.go:414] "containerMap: removing container" podUID="ae6b294b-6d5b-4bd7-820b-7e179011a2f8" containerName="istio-proxy" Nov 03 02:25:42.775724 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:42.775631 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae6b294b-6d5b-4bd7-820b-7e179011a2f8" containerName="istio-proxy" Nov 03 02:25:42.776094 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:42.775764 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="92507c2a-dbae-47e0-96f7-d50a7ccbd312" containerName="istio-proxy" Nov 03 02:25:42.776094 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:42.775780 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc02e9e7-6ef2-4b5f-b246-0d4ca1719f6d" containerName="istio-proxy" Nov 03 02:25:42.776094 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:42.775794 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae6b294b-6d5b-4bd7-820b-7e179011a2f8" containerName="istio-proxy" Nov 03 02:25:42.776094 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:42.775803 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="338113d0-1ddb-4f8b-9cfa-94c2ed4a5c17" containerName="istio-proxy" Nov 03 02:25:42.776094 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:42.775813 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8c9203e-5102-452b-bbc7-af1b50ea3ec7" containerName="istio-proxy" Nov 03 02:25:42.776094 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:42.775825 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdbfc7ca-1928-4b51-8967-cb5576ca80cb" containerName="istio-proxy" Nov 03 02:25:42.776094 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:42.775834 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e17c6ad-4cc3-4cfd-a961-c14fcb4241cd" containerName="istio-proxy" Nov 03 02:25:42.781054 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:42.781026 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-f760d3-984756bb7-5v2b6" Nov 03 02:25:42.826297 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:42.826248 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8afe8b7-91ea-4e07-a283-1ce1f81a814a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-f760d3-984756bb7-5v2b6\" (UID: \"d8afe8b7-91ea-4e07-a283-1ce1f81a814a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f760d3-984756bb7-5v2b6" Nov 03 02:25:42.827057 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:42.827020 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xspqg\" (UniqueName: \"kubernetes.io/projected/d8afe8b7-91ea-4e07-a283-1ce1f81a814a-kube-api-access-xspqg\") pod \"payload-processing-e2e-praxis-a-f760d3-984756bb7-5v2b6\" (UID: \"d8afe8b7-91ea-4e07-a283-1ce1f81a814a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f760d3-984756bb7-5v2b6" Nov 03 02:25:42.843705 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:42.839614 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f760d3-984756bb7-5v2b6"] Nov 03 02:25:42.902057 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:42.899122 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-f760d3-dockercfg-pj7z4\"" Nov 03 02:25:42.902697 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:42.902670 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-f760d3\"" Nov 03 02:25:42.936944 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:42.933159 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8afe8b7-91ea-4e07-a283-1ce1f81a814a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-f760d3-984756bb7-5v2b6\" (UID: \"d8afe8b7-91ea-4e07-a283-1ce1f81a814a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f760d3-984756bb7-5v2b6" Nov 03 02:25:42.936944 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:42.933281 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xspqg\" (UniqueName: \"kubernetes.io/projected/d8afe8b7-91ea-4e07-a283-1ce1f81a814a-kube-api-access-xspqg\") pod \"payload-processing-e2e-praxis-a-f760d3-984756bb7-5v2b6\" (UID: \"d8afe8b7-91ea-4e07-a283-1ce1f81a814a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f760d3-984756bb7-5v2b6" Nov 03 02:25:42.936944 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:42.934925 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8afe8b7-91ea-4e07-a283-1ce1f81a814a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-f760d3-984756bb7-5v2b6\" (UID: \"d8afe8b7-91ea-4e07-a283-1ce1f81a814a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f760d3-984756bb7-5v2b6" Nov 03 02:25:43.020422 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:43.018274 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xspqg\" (UniqueName: \"kubernetes.io/projected/d8afe8b7-91ea-4e07-a283-1ce1f81a814a-kube-api-access-xspqg\") pod \"payload-processing-e2e-praxis-a-f760d3-984756bb7-5v2b6\" (UID: \"d8afe8b7-91ea-4e07-a283-1ce1f81a814a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f760d3-984756bb7-5v2b6" Nov 03 02:25:43.127606 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:43.127567 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-f760d3-984756bb7-5v2b6" Nov 03 02:25:43.521924 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:43.520323 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f760d3-984756bb7-5v2b6"] Nov 03 02:25:44.393875 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:44.393827 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-f760d3-984756bb7-5v2b6" event={"ID":"d8afe8b7-91ea-4e07-a283-1ce1f81a814a","Type":"ContainerStarted","Data":"84631ae50a4b0e3cf6b72ed63288761e0c45b9342ee873bc8e77b4af79b65d1a"} Nov 03 02:25:44.394288 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:44.393885 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-f760d3-984756bb7-5v2b6" event={"ID":"d8afe8b7-91ea-4e07-a283-1ce1f81a814a","Type":"ContainerStarted","Data":"d93a7604d4f7a2b045f47f9a7d0250c1acf5fa329377f55ca7e14b46d1ff7f52"} Nov 03 02:25:44.394288 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:44.393946 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-f760d3-984756bb7-5v2b6" Nov 03 02:25:44.500240 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:44.500184 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-f760d3-984756bb7-5v2b6" podStartSLOduration=2.5001695379999997 podStartE2EDuration="2.500169538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:25:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:25:44.499944692 +0000 UTC m=+2021.759142859" watchObservedRunningTime="2026-11-03 02:25:44.500169538 +0000 UTC m=+2021.759367691" Nov 03 02:25:48.057118 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:48.055172 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:25:53.915874 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:25:53.915833 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:25:55.403889 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:55.403859 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-f760d3-984756bb7-5v2b6" Nov 03 02:25:56.548382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:56.544462 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7f9d9d-794bb5df47-hlcwq"] Nov 03 02:25:56.566744 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:56.566710 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-7f9d9d-794bb5df47-hlcwq" Nov 03 02:25:56.604136 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:56.604100 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7f9d9d-794bb5df47-hlcwq"] Nov 03 02:25:56.624852 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:56.624814 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ngc6\" (UniqueName: \"kubernetes.io/projected/f0e676c5-fdda-4636-86de-941149320c96-kube-api-access-5ngc6\") pod \"payload-processing-e2e-praxis-b-7f9d9d-794bb5df47-hlcwq\" (UID: \"f0e676c5-fdda-4636-86de-941149320c96\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7f9d9d-794bb5df47-hlcwq" Nov 03 02:25:56.625035 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:56.624886 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0e676c5-fdda-4636-86de-941149320c96-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-7f9d9d-794bb5df47-hlcwq\" (UID: \"f0e676c5-fdda-4636-86de-941149320c96\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7f9d9d-794bb5df47-hlcwq" Nov 03 02:25:56.648373 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:56.648331 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-7f9d9d-dockercfg-gqdx5\"" Nov 03 02:25:56.648373 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:56.648350 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-7f9d9d\"" Nov 03 02:25:56.716786 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:56.716743 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7f9d9d-55455d767-xpmtx"] Nov 03 02:25:56.720696 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:56.720668 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7f9d9d-55455d767-xpmtx" Nov 03 02:25:56.726693 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:56.726664 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ngc6\" (UniqueName: \"kubernetes.io/projected/f0e676c5-fdda-4636-86de-941149320c96-kube-api-access-5ngc6\") pod \"payload-processing-e2e-praxis-b-7f9d9d-794bb5df47-hlcwq\" (UID: \"f0e676c5-fdda-4636-86de-941149320c96\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7f9d9d-794bb5df47-hlcwq" Nov 03 02:25:56.726854 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:56.726741 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0e676c5-fdda-4636-86de-941149320c96-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-7f9d9d-794bb5df47-hlcwq\" (UID: \"f0e676c5-fdda-4636-86de-941149320c96\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7f9d9d-794bb5df47-hlcwq" Nov 03 02:25:56.727675 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:56.727474 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0e676c5-fdda-4636-86de-941149320c96-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-7f9d9d-794bb5df47-hlcwq\" (UID: \"f0e676c5-fdda-4636-86de-941149320c96\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7f9d9d-794bb5df47-hlcwq" Nov 03 02:25:56.773710 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:56.773671 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7f9d9d-55455d767-xpmtx"] Nov 03 02:25:56.780053 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:56.780020 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ngc6\" (UniqueName: \"kubernetes.io/projected/f0e676c5-fdda-4636-86de-941149320c96-kube-api-access-5ngc6\") pod \"payload-processing-e2e-praxis-b-7f9d9d-794bb5df47-hlcwq\" (UID: \"f0e676c5-fdda-4636-86de-941149320c96\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7f9d9d-794bb5df47-hlcwq" Nov 03 02:25:56.827596 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:56.827499 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e429518f-695c-46c8-839b-fade2d52c105-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-7f9d9d-55455d767-xpmtx\" (UID: \"e429518f-695c-46c8-839b-fade2d52c105\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7f9d9d-55455d767-xpmtx" Nov 03 02:25:56.827779 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:56.827737 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmqlf\" (UniqueName: \"kubernetes.io/projected/e429518f-695c-46c8-839b-fade2d52c105-kube-api-access-bmqlf\") pod \"payload-pre-processing-e2e-praxis-b-7f9d9d-55455d767-xpmtx\" (UID: \"e429518f-695c-46c8-839b-fade2d52c105\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7f9d9d-55455d767-xpmtx" Nov 03 02:25:56.893472 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:56.893430 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-7f9d9d-794bb5df47-hlcwq" Nov 03 02:25:56.928985 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:56.928949 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmqlf\" (UniqueName: \"kubernetes.io/projected/e429518f-695c-46c8-839b-fade2d52c105-kube-api-access-bmqlf\") pod \"payload-pre-processing-e2e-praxis-b-7f9d9d-55455d767-xpmtx\" (UID: \"e429518f-695c-46c8-839b-fade2d52c105\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7f9d9d-55455d767-xpmtx" Nov 03 02:25:56.929181 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:56.929016 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e429518f-695c-46c8-839b-fade2d52c105-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-7f9d9d-55455d767-xpmtx\" (UID: \"e429518f-695c-46c8-839b-fade2d52c105\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7f9d9d-55455d767-xpmtx" Nov 03 02:25:56.929727 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:56.929706 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e429518f-695c-46c8-839b-fade2d52c105-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-7f9d9d-55455d767-xpmtx\" (UID: \"e429518f-695c-46c8-839b-fade2d52c105\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7f9d9d-55455d767-xpmtx" Nov 03 02:25:56.982479 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:56.982445 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmqlf\" (UniqueName: \"kubernetes.io/projected/e429518f-695c-46c8-839b-fade2d52c105-kube-api-access-bmqlf\") pod \"payload-pre-processing-e2e-praxis-b-7f9d9d-55455d767-xpmtx\" (UID: \"e429518f-695c-46c8-839b-fade2d52c105\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7f9d9d-55455d767-xpmtx" Nov 03 02:25:57.033642 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:57.033606 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7f9d9d-55455d767-xpmtx" Nov 03 02:25:57.266553 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:57.266494 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7f9d9d-794bb5df47-hlcwq"] Nov 03 02:25:57.274735 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:25:57.274693 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0e676c5_fdda_4636_86de_941149320c96.slice/crio-82b4d96cd78434c818d1f9704941ad6b83aef9212c840401e82f1d3f8b9bb45d WatchSource:0}: Error finding container 82b4d96cd78434c818d1f9704941ad6b83aef9212c840401e82f1d3f8b9bb45d: Status 404 returned error can't find the container with id 82b4d96cd78434c818d1f9704941ad6b83aef9212c840401e82f1d3f8b9bb45d Nov 03 02:25:57.410726 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:57.410624 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7f9d9d-55455d767-xpmtx"] Nov 03 02:25:57.416495 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:25:57.416462 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode429518f_695c_46c8_839b_fade2d52c105.slice/crio-60ae9eddeb11b5a19bf7de232b130b32e99d39c1b5ea6d319245ac50265abb05 WatchSource:0}: Error finding container 60ae9eddeb11b5a19bf7de232b130b32e99d39c1b5ea6d319245ac50265abb05: Status 404 returned error can't find the container with id 60ae9eddeb11b5a19bf7de232b130b32e99d39c1b5ea6d319245ac50265abb05 Nov 03 02:25:57.467961 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:57.467928 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7f9d9d-55455d767-xpmtx" event={"ID":"e429518f-695c-46c8-839b-fade2d52c105","Type":"ContainerStarted","Data":"60ae9eddeb11b5a19bf7de232b130b32e99d39c1b5ea6d319245ac50265abb05"} Nov 03 02:25:57.469663 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:57.469633 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-7f9d9d-794bb5df47-hlcwq" event={"ID":"f0e676c5-fdda-4636-86de-941149320c96","Type":"ContainerStarted","Data":"924cd65533e200ed886e359b3556cfe9d5579b0b23fb9228aed2964d6aa43c2d"} Nov 03 02:25:57.469831 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:57.469691 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-7f9d9d-794bb5df47-hlcwq" event={"ID":"f0e676c5-fdda-4636-86de-941149320c96","Type":"ContainerStarted","Data":"82b4d96cd78434c818d1f9704941ad6b83aef9212c840401e82f1d3f8b9bb45d"} Nov 03 02:25:57.469893 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:57.469835 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-7f9d9d-794bb5df47-hlcwq" Nov 03 02:25:57.576236 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:57.576182 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-7f9d9d-794bb5df47-hlcwq" podStartSLOduration=1.5761671910000001 podStartE2EDuration="1.576167191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:25:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:25:57.574957746 +0000 UTC m=+2034.834155902" watchObservedRunningTime="2026-11-03 02:25:57.576167191 +0000 UTC m=+2034.835365345" Nov 03 02:25:58.476648 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:58.476598 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7f9d9d-55455d767-xpmtx" event={"ID":"e429518f-695c-46c8-839b-fade2d52c105","Type":"ContainerStarted","Data":"45b423504e372f01c9069813d85a32d1ff6133ff6b5c69d9762014f2cd38d84c"} Nov 03 02:25:58.476860 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:58.476826 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7f9d9d-55455d767-xpmtx" Nov 03 02:25:58.583145 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:58.583079 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7f9d9d-55455d767-xpmtx" podStartSLOduration=2.583058879 podStartE2EDuration="2.583058879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:25:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:25:58.582250803 +0000 UTC m=+2035.841448957" watchObservedRunningTime="2026-11-03 02:25:58.583058879 +0000 UTC m=+2035.842257033" Nov 03 02:25:59.519684 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:59.519647 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d63a25-6ff9c4b65f-jfgrr"] Nov 03 02:25:59.630401 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:25:59.630334 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d63a25-6769697dcf-lzql7"] Nov 03 02:26:04.520642 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:26:04.520597 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-d63a25-6ff9c4b65f-jfgrr" podUID="b11fa364-4870-438e-861e-46e0f0121f8f" containerName="payload-processing" containerID="cri-o://11e74f795d0be19ff61e0aa4810bc77992b48b8df3d59d0a7f843e5fe17b4408" gracePeriod=25 Nov 03 02:26:04.630703 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:26:04.630645 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d63a25-6769697dcf-lzql7" podUID="969764a4-4632-4cda-b6e2-2745ccbf2ab8" containerName="payload-pre-processing" containerID="cri-o://799ec4184b4ca1388f5a6721cbc6cb6d74265a85e6c7af67e2e6e854152745b2" gracePeriod=25 Nov 03 02:26:04.799220 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:26:04.799191 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d63a25-6ff9c4b65f-jfgrr" Nov 03 02:26:04.902713 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:26:04.902678 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d63a25-6769697dcf-lzql7" Nov 03 02:26:04.943835 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:26:04.943796 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b11fa364-4870-438e-861e-46e0f0121f8f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b11fa364-4870-438e-861e-46e0f0121f8f-plugins-config-volume\") pod \"b11fa364-4870-438e-861e-46e0f0121f8f\" (UID: \"b11fa364-4870-438e-861e-46e0f0121f8f\") " Nov 03 02:26:04.944019 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:26:04.943865 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b11fa364-4870-438e-861e-46e0f0121f8f-kube-api-access-2skhb\" (UniqueName: \"kubernetes.io/projected/b11fa364-4870-438e-861e-46e0f0121f8f-kube-api-access-2skhb\") pod \"b11fa364-4870-438e-861e-46e0f0121f8f\" (UID: \"b11fa364-4870-438e-861e-46e0f0121f8f\") " Nov 03 02:26:04.944163 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:26:04.944137 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b11fa364-4870-438e-861e-46e0f0121f8f-plugins-config-volume" pod "b11fa364-4870-438e-861e-46e0f0121f8f" (UID: "b11fa364-4870-438e-861e-46e0f0121f8f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:26:04.944475 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:26:04.944451 5605 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b11fa364-4870-438e-861e-46e0f0121f8f-plugins-config-volume\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:26:04.946142 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:26:04.946113 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b11fa364-4870-438e-861e-46e0f0121f8f-kube-api-access-2skhb" pod "b11fa364-4870-438e-861e-46e0f0121f8f" (UID: "b11fa364-4870-438e-861e-46e0f0121f8f"). InnerVolumeSpecName "kube-api-access-2skhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:26:05.045120 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:26:05.045038 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/969764a4-4632-4cda-b6e2-2745ccbf2ab8-kube-api-access-mxm6b\" (UniqueName: \"kubernetes.io/projected/969764a4-4632-4cda-b6e2-2745ccbf2ab8-kube-api-access-mxm6b\") pod \"969764a4-4632-4cda-b6e2-2745ccbf2ab8\" (UID: \"969764a4-4632-4cda-b6e2-2745ccbf2ab8\") " Nov 03 02:26:05.045120 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:26:05.045110 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/969764a4-4632-4cda-b6e2-2745ccbf2ab8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/969764a4-4632-4cda-b6e2-2745ccbf2ab8-plugins-config-volume\") pod \"969764a4-4632-4cda-b6e2-2745ccbf2ab8\" (UID: \"969764a4-4632-4cda-b6e2-2745ccbf2ab8\") " Nov 03 02:26:05.045571 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:26:05.045542 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/969764a4-4632-4cda-b6e2-2745ccbf2ab8-plugins-config-volume" pod "969764a4-4632-4cda-b6e2-2745ccbf2ab8" (UID: "969764a4-4632-4cda-b6e2-2745ccbf2ab8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:26:05.045676 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:26:05.045660 5605 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/969764a4-4632-4cda-b6e2-2745ccbf2ab8-plugins-config-volume\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:26:05.045732 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:26:05.045678 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2skhb\" (UniqueName: \"kubernetes.io/projected/b11fa364-4870-438e-861e-46e0f0121f8f-kube-api-access-2skhb\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:26:05.047387 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:26:05.047331 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/969764a4-4632-4cda-b6e2-2745ccbf2ab8-kube-api-access-mxm6b" pod "969764a4-4632-4cda-b6e2-2745ccbf2ab8" (UID: "969764a4-4632-4cda-b6e2-2745ccbf2ab8"). InnerVolumeSpecName "kube-api-access-mxm6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:26:05.147192 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:26:05.147155 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxm6b\" (UniqueName: \"kubernetes.io/projected/969764a4-4632-4cda-b6e2-2745ccbf2ab8-kube-api-access-mxm6b\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:26:05.516031 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:26:05.515997 5605 generic.go:356] "Generic (PLEG): container finished" podID="969764a4-4632-4cda-b6e2-2745ccbf2ab8" containerID="799ec4184b4ca1388f5a6721cbc6cb6d74265a85e6c7af67e2e6e854152745b2" exitCode=0 Nov 03 02:26:05.516258 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:26:05.516049 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d63a25-6769697dcf-lzql7" Nov 03 02:26:05.516258 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:26:05.516080 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d63a25-6769697dcf-lzql7" event={"ID":"969764a4-4632-4cda-b6e2-2745ccbf2ab8","Type":"ContainerDied","Data":"799ec4184b4ca1388f5a6721cbc6cb6d74265a85e6c7af67e2e6e854152745b2"} Nov 03 02:26:05.516258 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:26:05.516140 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d63a25-6769697dcf-lzql7" event={"ID":"969764a4-4632-4cda-b6e2-2745ccbf2ab8","Type":"ContainerDied","Data":"625285ff52965296a2dd04baab8085c600850983a5ce675aaa01fc2fda67b95f"} Nov 03 02:26:05.516258 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:26:05.516178 5605 scope.go:122] "RemoveContainer" containerID="799ec4184b4ca1388f5a6721cbc6cb6d74265a85e6c7af67e2e6e854152745b2" Nov 03 02:26:05.517555 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:26:05.517530 5605 generic.go:356] "Generic (PLEG): container finished" podID="b11fa364-4870-438e-861e-46e0f0121f8f" containerID="11e74f795d0be19ff61e0aa4810bc77992b48b8df3d59d0a7f843e5fe17b4408" exitCode=0 Nov 03 02:26:05.517640 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:26:05.517575 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d63a25-6ff9c4b65f-jfgrr" Nov 03 02:26:05.517640 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:26:05.517614 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d63a25-6ff9c4b65f-jfgrr" event={"ID":"b11fa364-4870-438e-861e-46e0f0121f8f","Type":"ContainerDied","Data":"11e74f795d0be19ff61e0aa4810bc77992b48b8df3d59d0a7f843e5fe17b4408"} Nov 03 02:26:05.517715 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:26:05.517657 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d63a25-6ff9c4b65f-jfgrr" event={"ID":"b11fa364-4870-438e-861e-46e0f0121f8f","Type":"ContainerDied","Data":"754b85b367910bcb61c43171b752631d60f950f553dc1d4641328a27f89f0fb8"} Nov 03 02:26:05.526405 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:26:05.526385 5605 scope.go:122] "RemoveContainer" containerID="799ec4184b4ca1388f5a6721cbc6cb6d74265a85e6c7af67e2e6e854152745b2" Nov 03 02:26:05.526708 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:26:05.526691 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"799ec4184b4ca1388f5a6721cbc6cb6d74265a85e6c7af67e2e6e854152745b2\": container with ID starting with 799ec4184b4ca1388f5a6721cbc6cb6d74265a85e6c7af67e2e6e854152745b2 not found: ID does not exist" containerID="799ec4184b4ca1388f5a6721cbc6cb6d74265a85e6c7af67e2e6e854152745b2" Nov 03 02:26:05.526752 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:26:05.526716 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"799ec4184b4ca1388f5a6721cbc6cb6d74265a85e6c7af67e2e6e854152745b2"} err="failed to get container status \"799ec4184b4ca1388f5a6721cbc6cb6d74265a85e6c7af67e2e6e854152745b2\": rpc error: code = NotFound desc = could not find container \"799ec4184b4ca1388f5a6721cbc6cb6d74265a85e6c7af67e2e6e854152745b2\": container with ID starting with 799ec4184b4ca1388f5a6721cbc6cb6d74265a85e6c7af67e2e6e854152745b2 not found: ID does not exist" Nov 03 02:26:05.526752 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:26:05.526733 5605 scope.go:122] "RemoveContainer" containerID="11e74f795d0be19ff61e0aa4810bc77992b48b8df3d59d0a7f843e5fe17b4408" Nov 03 02:26:05.544417 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:26:05.544386 5605 scope.go:122] "RemoveContainer" containerID="11e74f795d0be19ff61e0aa4810bc77992b48b8df3d59d0a7f843e5fe17b4408" Nov 03 02:26:05.544759 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:26:05.544738 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11e74f795d0be19ff61e0aa4810bc77992b48b8df3d59d0a7f843e5fe17b4408\": container with ID starting with 11e74f795d0be19ff61e0aa4810bc77992b48b8df3d59d0a7f843e5fe17b4408 not found: ID does not exist" containerID="11e74f795d0be19ff61e0aa4810bc77992b48b8df3d59d0a7f843e5fe17b4408" Nov 03 02:26:05.544821 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:26:05.544768 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11e74f795d0be19ff61e0aa4810bc77992b48b8df3d59d0a7f843e5fe17b4408"} err="failed to get container status \"11e74f795d0be19ff61e0aa4810bc77992b48b8df3d59d0a7f843e5fe17b4408\": rpc error: code = NotFound desc = could not find container \"11e74f795d0be19ff61e0aa4810bc77992b48b8df3d59d0a7f843e5fe17b4408\": container with ID starting with 11e74f795d0be19ff61e0aa4810bc77992b48b8df3d59d0a7f843e5fe17b4408 not found: ID does not exist" Nov 03 02:26:05.668928 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:26:05.668891 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d63a25-6ff9c4b65f-jfgrr"] Nov 03 02:26:05.709543 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:26:05.709510 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d63a25-6ff9c4b65f-jfgrr"] Nov 03 02:26:05.774977 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:26:05.774878 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d63a25-6769697dcf-lzql7"] Nov 03 02:26:05.776788 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:26:05.776761 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d63a25-6769697dcf-lzql7"] Nov 03 02:26:06.851060 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:26:06.851024 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="969764a4-4632-4cda-b6e2-2745ccbf2ab8" path="/var/lib/kubelet/pods/969764a4-4632-4cda-b6e2-2745ccbf2ab8/volumes" Nov 03 02:26:06.851514 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:26:06.851444 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b11fa364-4870-438e-861e-46e0f0121f8f" path="/var/lib/kubelet/pods/b11fa364-4870-438e-861e-46e0f0121f8f/volumes" Nov 03 02:26:08.480581 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:26:08.480551 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-7f9d9d-794bb5df47-hlcwq" Nov 03 02:26:09.484553 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:26:09.484522 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7f9d9d-55455d767-xpmtx" Nov 03 02:26:23.918018 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:26:23.917975 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:26:30.717047 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:26:30.717008 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:26:53.935047 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:26:53.934958 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:27:19.248519 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:19.248480 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:27:23.913506 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:27:23.913459 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:27:48.839379 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:48.839328 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn"] Nov 03 02:27:48.840055 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:48.840033 5605 cpu_manager.go:414] "containerMap: removing container" podUID="969764a4-4632-4cda-b6e2-2745ccbf2ab8" containerName="payload-pre-processing" Nov 03 02:27:48.840119 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:48.840058 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="969764a4-4632-4cda-b6e2-2745ccbf2ab8" containerName="payload-pre-processing" Nov 03 02:27:48.840119 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:48.840084 5605 cpu_manager.go:414] "containerMap: removing container" podUID="b11fa364-4870-438e-861e-46e0f0121f8f" containerName="payload-processing" Nov 03 02:27:48.840119 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:48.840092 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b11fa364-4870-438e-861e-46e0f0121f8f" containerName="payload-processing" Nov 03 02:27:48.840221 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:48.840202 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="969764a4-4632-4cda-b6e2-2745ccbf2ab8" containerName="payload-pre-processing" Nov 03 02:27:48.840253 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:48.840219 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="b11fa364-4870-438e-861e-46e0f0121f8f" containerName="payload-processing" Nov 03 02:27:48.844165 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:48.844134 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn" Nov 03 02:27:48.871496 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:48.871468 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb2bd860-6aa6-4f34-a90a-d599c74c4cba-istio-data\") pod \"e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn\" (UID: \"fb2bd860-6aa6-4f34-a90a-d599c74c4cba\") " pod="openshift-ingress/e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn" Nov 03 02:27:48.871662 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:48.871499 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb2bd860-6aa6-4f34-a90a-d599c74c4cba-istio-token\") pod \"e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn\" (UID: \"fb2bd860-6aa6-4f34-a90a-d599c74c4cba\") " pod="openshift-ingress/e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn" Nov 03 02:27:48.871662 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:48.871589 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb2bd860-6aa6-4f34-a90a-d599c74c4cba-credential-socket\") pod \"e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn\" (UID: \"fb2bd860-6aa6-4f34-a90a-d599c74c4cba\") " pod="openshift-ingress/e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn" Nov 03 02:27:48.871751 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:48.871686 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb2bd860-6aa6-4f34-a90a-d599c74c4cba-istio-podinfo\") pod \"e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn\" (UID: \"fb2bd860-6aa6-4f34-a90a-d599c74c4cba\") " pod="openshift-ingress/e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn" Nov 03 02:27:48.871807 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:48.871791 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb2bd860-6aa6-4f34-a90a-d599c74c4cba-workload-socket\") pod \"e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn\" (UID: \"fb2bd860-6aa6-4f34-a90a-d599c74c4cba\") " pod="openshift-ingress/e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn" Nov 03 02:27:48.871886 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:48.871872 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb2bd860-6aa6-4f34-a90a-d599c74c4cba-istiod-ca-cert\") pod \"e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn\" (UID: \"fb2bd860-6aa6-4f34-a90a-d599c74c4cba\") " pod="openshift-ingress/e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn" Nov 03 02:27:48.871939 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:48.871925 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb2bd860-6aa6-4f34-a90a-d599c74c4cba-workload-certs\") pod \"e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn\" (UID: \"fb2bd860-6aa6-4f34-a90a-d599c74c4cba\") " pod="openshift-ingress/e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn" Nov 03 02:27:48.871988 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:48.871976 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb2bd860-6aa6-4f34-a90a-d599c74c4cba-istio-envoy\") pod \"e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn\" (UID: \"fb2bd860-6aa6-4f34-a90a-d599c74c4cba\") " pod="openshift-ingress/e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn" Nov 03 02:27:48.872113 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:48.872093 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbkdq\" (UniqueName: \"kubernetes.io/projected/fb2bd860-6aa6-4f34-a90a-d599c74c4cba-kube-api-access-tbkdq\") pod \"e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn\" (UID: \"fb2bd860-6aa6-4f34-a90a-d599c74c4cba\") " pod="openshift-ingress/e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn" Nov 03 02:27:48.938956 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:48.938914 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn"] Nov 03 02:27:48.948707 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:48.948666 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-496177-openshift-default-dockercfg-cbvzh\"" Nov 03 02:27:48.972942 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:48.972895 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb2bd860-6aa6-4f34-a90a-d599c74c4cba-credential-socket\") pod \"e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn\" (UID: \"fb2bd860-6aa6-4f34-a90a-d599c74c4cba\") " pod="openshift-ingress/e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn" Nov 03 02:27:48.973756 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:48.973731 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb2bd860-6aa6-4f34-a90a-d599c74c4cba-istio-podinfo\") pod \"e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn\" (UID: \"fb2bd860-6aa6-4f34-a90a-d599c74c4cba\") " pod="openshift-ingress/e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn" Nov 03 02:27:48.973981 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:48.973959 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb2bd860-6aa6-4f34-a90a-d599c74c4cba-workload-socket\") pod \"e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn\" (UID: \"fb2bd860-6aa6-4f34-a90a-d599c74c4cba\") " pod="openshift-ingress/e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn" Nov 03 02:27:48.974472 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:48.974453 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb2bd860-6aa6-4f34-a90a-d599c74c4cba-istiod-ca-cert\") pod \"e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn\" (UID: \"fb2bd860-6aa6-4f34-a90a-d599c74c4cba\") " pod="openshift-ingress/e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn" Nov 03 02:27:48.974624 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:48.974605 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb2bd860-6aa6-4f34-a90a-d599c74c4cba-workload-certs\") pod \"e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn\" (UID: \"fb2bd860-6aa6-4f34-a90a-d599c74c4cba\") " pod="openshift-ingress/e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn" Nov 03 02:27:48.974766 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:48.974749 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb2bd860-6aa6-4f34-a90a-d599c74c4cba-istio-envoy\") pod \"e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn\" (UID: \"fb2bd860-6aa6-4f34-a90a-d599c74c4cba\") " pod="openshift-ingress/e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn" Nov 03 02:27:48.974940 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:48.974921 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbkdq\" (UniqueName: \"kubernetes.io/projected/fb2bd860-6aa6-4f34-a90a-d599c74c4cba-kube-api-access-tbkdq\") pod \"e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn\" (UID: \"fb2bd860-6aa6-4f34-a90a-d599c74c4cba\") " pod="openshift-ingress/e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn" Nov 03 02:27:48.976669 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:48.976648 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb2bd860-6aa6-4f34-a90a-d599c74c4cba-istio-data\") pod \"e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn\" (UID: \"fb2bd860-6aa6-4f34-a90a-d599c74c4cba\") " pod="openshift-ingress/e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn" Nov 03 02:27:48.977524 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:48.977431 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb2bd860-6aa6-4f34-a90a-d599c74c4cba-istio-token\") pod \"e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn\" (UID: \"fb2bd860-6aa6-4f34-a90a-d599c74c4cba\") " pod="openshift-ingress/e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn" Nov 03 02:27:48.978066 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:48.975666 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb2bd860-6aa6-4f34-a90a-d599c74c4cba-istiod-ca-cert\") pod \"e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn\" (UID: \"fb2bd860-6aa6-4f34-a90a-d599c74c4cba\") " pod="openshift-ingress/e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn" Nov 03 02:27:48.978343 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:48.975900 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb2bd860-6aa6-4f34-a90a-d599c74c4cba-workload-certs\") pod \"e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn\" (UID: \"fb2bd860-6aa6-4f34-a90a-d599c74c4cba\") " pod="openshift-ingress/e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn" Nov 03 02:27:48.978517 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:48.974375 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb2bd860-6aa6-4f34-a90a-d599c74c4cba-workload-socket\") pod \"e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn\" (UID: \"fb2bd860-6aa6-4f34-a90a-d599c74c4cba\") " pod="openshift-ingress/e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn" Nov 03 02:27:48.979389 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:48.977374 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb2bd860-6aa6-4f34-a90a-d599c74c4cba-istio-data\") pod \"e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn\" (UID: \"fb2bd860-6aa6-4f34-a90a-d599c74c4cba\") " pod="openshift-ingress/e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn" Nov 03 02:27:48.979525 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:48.973589 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb2bd860-6aa6-4f34-a90a-d599c74c4cba-credential-socket\") pod \"e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn\" (UID: \"fb2bd860-6aa6-4f34-a90a-d599c74c4cba\") " pod="openshift-ingress/e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn" Nov 03 02:27:49.058383 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:49.032629 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb2bd860-6aa6-4f34-a90a-d599c74c4cba-istio-podinfo\") pod \"e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn\" (UID: \"fb2bd860-6aa6-4f34-a90a-d599c74c4cba\") " pod="openshift-ingress/e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn" Nov 03 02:27:49.058687 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:49.047040 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb2bd860-6aa6-4f34-a90a-d599c74c4cba-istio-token\") pod \"e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn\" (UID: \"fb2bd860-6aa6-4f34-a90a-d599c74c4cba\") " pod="openshift-ingress/e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn" Nov 03 02:27:49.059696 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:49.059665 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbkdq\" (UniqueName: \"kubernetes.io/projected/fb2bd860-6aa6-4f34-a90a-d599c74c4cba-kube-api-access-tbkdq\") pod \"e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn\" (UID: \"fb2bd860-6aa6-4f34-a90a-d599c74c4cba\") " pod="openshift-ingress/e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn" Nov 03 02:27:49.065175 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:49.064986 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb2bd860-6aa6-4f34-a90a-d599c74c4cba-istio-envoy\") pod \"e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn\" (UID: \"fb2bd860-6aa6-4f34-a90a-d599c74c4cba\") " pod="openshift-ingress/e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn" Nov 03 02:27:49.158323 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:49.158236 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn" Nov 03 02:27:49.586610 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:49.586571 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn"] Nov 03 02:27:49.595596 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:27:49.595560 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb2bd860_6aa6_4f34_a90a_d599c74c4cba.slice/crio-eda378f618aabd3f9c116d0e649f320bc085d69d858b9f42ab1fa40565408050 WatchSource:0}: Error finding container eda378f618aabd3f9c116d0e649f320bc085d69d858b9f42ab1fa40565408050: Status 404 returned error can't find the container with id eda378f618aabd3f9c116d0e649f320bc085d69d858b9f42ab1fa40565408050 Nov 03 02:27:49.598560 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:49.598512 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:27:49.598656 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:49.598632 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:27:49.598697 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:49.598678 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:27:50.163947 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:50.163912 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn" event={"ID":"fb2bd860-6aa6-4f34-a90a-d599c74c4cba","Type":"ContainerStarted","Data":"5f3cb2962cf3a2c311567b52bdf4f52f4217e88dceb011d141514e2b6096f540"} Nov 03 02:27:50.164404 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:50.163965 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn" event={"ID":"fb2bd860-6aa6-4f34-a90a-d599c74c4cba","Type":"ContainerStarted","Data":"eda378f618aabd3f9c116d0e649f320bc085d69d858b9f42ab1fa40565408050"} Nov 03 02:27:50.276088 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:50.276027 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn" podStartSLOduration=2.27601047 podStartE2EDuration="2.27601047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:27:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:27:50.273825628 +0000 UTC m=+2147.533023828" watchObservedRunningTime="2026-11-03 02:27:50.27601047 +0000 UTC m=+2147.535208710" Nov 03 02:27:51.158782 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:51.158735 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn" Nov 03 02:27:51.164390 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:51.164338 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn" Nov 03 02:27:51.174772 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:51.174729 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn" Nov 03 02:27:51.175855 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:51.175831 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn" Nov 03 02:27:53.915271 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:27:53.915229 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:27:56.905656 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:56.905604 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-496177-5b44d87cbf-5p6qp"] Nov 03 02:27:56.922902 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:56.922699 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-496177-5b44d87cbf-5p6qp" Nov 03 02:27:56.961061 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:56.961024 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-496177-5b44d87cbf-5p6qp"] Nov 03 02:27:56.982323 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:56.982287 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0714b248-bec2-44ad-9ca3-e99d0bb89659-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-496177-5b44d87cbf-5p6qp\" (UID: \"0714b248-bec2-44ad-9ca3-e99d0bb89659\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-496177-5b44d87cbf-5p6qp" Nov 03 02:27:56.982487 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:56.982348 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp4jt\" (UniqueName: \"kubernetes.io/projected/0714b248-bec2-44ad-9ca3-e99d0bb89659-kube-api-access-dp4jt\") pod \"payload-processing-e2e-praxis-b-496177-5b44d87cbf-5p6qp\" (UID: \"0714b248-bec2-44ad-9ca3-e99d0bb89659\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-496177-5b44d87cbf-5p6qp" Nov 03 02:27:57.005319 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:57.005289 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-496177\"" Nov 03 02:27:57.005470 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:57.005449 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-496177-dockercfg-mdtxg\"" Nov 03 02:27:57.083712 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:57.083679 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0714b248-bec2-44ad-9ca3-e99d0bb89659-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-496177-5b44d87cbf-5p6qp\" (UID: \"0714b248-bec2-44ad-9ca3-e99d0bb89659\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-496177-5b44d87cbf-5p6qp" Nov 03 02:27:57.083898 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:57.083736 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp4jt\" (UniqueName: \"kubernetes.io/projected/0714b248-bec2-44ad-9ca3-e99d0bb89659-kube-api-access-dp4jt\") pod \"payload-processing-e2e-praxis-b-496177-5b44d87cbf-5p6qp\" (UID: \"0714b248-bec2-44ad-9ca3-e99d0bb89659\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-496177-5b44d87cbf-5p6qp" Nov 03 02:27:57.084437 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:57.084417 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0714b248-bec2-44ad-9ca3-e99d0bb89659-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-496177-5b44d87cbf-5p6qp\" (UID: \"0714b248-bec2-44ad-9ca3-e99d0bb89659\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-496177-5b44d87cbf-5p6qp" Nov 03 02:27:57.137820 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:57.137788 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp4jt\" (UniqueName: \"kubernetes.io/projected/0714b248-bec2-44ad-9ca3-e99d0bb89659-kube-api-access-dp4jt\") pod \"payload-processing-e2e-praxis-b-496177-5b44d87cbf-5p6qp\" (UID: \"0714b248-bec2-44ad-9ca3-e99d0bb89659\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-496177-5b44d87cbf-5p6qp" Nov 03 02:27:57.239860 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:57.239769 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-496177-5b44d87cbf-5p6qp" Nov 03 02:27:57.607947 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:57.607903 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-496177-5b44d87cbf-5p6qp"] Nov 03 02:27:57.615467 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:27:57.615426 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0714b248_bec2_44ad_9ca3_e99d0bb89659.slice/crio-49442e37d3e7eb7c6c841596f46650c1b91a176fb758e7f1cbd7f9e1c0a0ec96 WatchSource:0}: Error finding container 49442e37d3e7eb7c6c841596f46650c1b91a176fb758e7f1cbd7f9e1c0a0ec96: Status 404 returned error can't find the container with id 49442e37d3e7eb7c6c841596f46650c1b91a176fb758e7f1cbd7f9e1c0a0ec96 Nov 03 02:27:58.210448 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:58.210403 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-496177-5b44d87cbf-5p6qp" event={"ID":"0714b248-bec2-44ad-9ca3-e99d0bb89659","Type":"ContainerStarted","Data":"bc5092b8966dc8c47569fd929c55c83434407e6cfbae1cc9dc5a1f813715c92b"} Nov 03 02:27:58.210873 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:58.210474 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-496177-5b44d87cbf-5p6qp" event={"ID":"0714b248-bec2-44ad-9ca3-e99d0bb89659","Type":"ContainerStarted","Data":"49442e37d3e7eb7c6c841596f46650c1b91a176fb758e7f1cbd7f9e1c0a0ec96"} Nov 03 02:27:58.210873 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:58.210609 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-496177-5b44d87cbf-5p6qp" Nov 03 02:27:58.318507 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:27:58.318438 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-496177-5b44d87cbf-5p6qp" podStartSLOduration=2.318419362 podStartE2EDuration="2.318419362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:27:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:27:58.315804786 +0000 UTC m=+2155.575002940" watchObservedRunningTime="2026-11-03 02:27:58.318419362 +0000 UTC m=+2155.577617517" Nov 03 02:28:09.218883 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:09.218844 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-496177-5b44d87cbf-5p6qp" Nov 03 02:28:16.251724 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:16.251639 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:28:23.937709 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:28:23.937667 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:28:29.572792 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:29.572753 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4gq8z"] Nov 03 02:28:29.573256 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:29.572954 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4gq8z" podUID="1d83861f-abc5-452e-a1d9-3148c6cd3774" containerName="istio-proxy" containerID="cri-o://52987e50d164950fed2ae55ed3a5a64e4e1443e07d9fb4ab698587690ca7825d" gracePeriod=30 Nov 03 02:28:30.128011 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:30.127965 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-4gq8z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 02:28:30.128185 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:30.128029 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4gq8z" podUID="1d83861f-abc5-452e-a1d9-3148c6cd3774" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 02:28:34.839453 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:34.839422 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4gq8z" Nov 03 02:28:34.937267 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:34.937167 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d83861f-abc5-452e-a1d9-3148c6cd3774-istio-token\" (UniqueName: \"kubernetes.io/projected/1d83861f-abc5-452e-a1d9-3148c6cd3774-istio-token\") pod \"1d83861f-abc5-452e-a1d9-3148c6cd3774\" (UID: \"1d83861f-abc5-452e-a1d9-3148c6cd3774\") " Nov 03 02:28:34.937267 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:34.937220 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d83861f-abc5-452e-a1d9-3148c6cd3774-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d83861f-abc5-452e-a1d9-3148c6cd3774-istio-data\") pod \"1d83861f-abc5-452e-a1d9-3148c6cd3774\" (UID: \"1d83861f-abc5-452e-a1d9-3148c6cd3774\") " Nov 03 02:28:34.937267 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:34.937261 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d83861f-abc5-452e-a1d9-3148c6cd3774-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d83861f-abc5-452e-a1d9-3148c6cd3774-credential-socket\") pod \"1d83861f-abc5-452e-a1d9-3148c6cd3774\" (UID: \"1d83861f-abc5-452e-a1d9-3148c6cd3774\") " Nov 03 02:28:34.937583 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:34.937300 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d83861f-abc5-452e-a1d9-3148c6cd3774-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d83861f-abc5-452e-a1d9-3148c6cd3774-workload-certs\") pod \"1d83861f-abc5-452e-a1d9-3148c6cd3774\" (UID: \"1d83861f-abc5-452e-a1d9-3148c6cd3774\") " Nov 03 02:28:34.937583 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:34.937389 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1d83861f-abc5-452e-a1d9-3148c6cd3774-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d83861f-abc5-452e-a1d9-3148c6cd3774-istio-podinfo\") pod \"1d83861f-abc5-452e-a1d9-3148c6cd3774\" (UID: \"1d83861f-abc5-452e-a1d9-3148c6cd3774\") " Nov 03 02:28:34.937583 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:34.937421 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d83861f-abc5-452e-a1d9-3148c6cd3774-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d83861f-abc5-452e-a1d9-3148c6cd3774-workload-socket\") pod \"1d83861f-abc5-452e-a1d9-3148c6cd3774\" (UID: \"1d83861f-abc5-452e-a1d9-3148c6cd3774\") " Nov 03 02:28:34.937732 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:34.937578 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d83861f-abc5-452e-a1d9-3148c6cd3774-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d83861f-abc5-452e-a1d9-3148c6cd3774-istio-envoy\") pod \"1d83861f-abc5-452e-a1d9-3148c6cd3774\" (UID: \"1d83861f-abc5-452e-a1d9-3148c6cd3774\") " Nov 03 02:28:34.937732 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:34.937676 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d83861f-abc5-452e-a1d9-3148c6cd3774-credential-socket" pod "1d83861f-abc5-452e-a1d9-3148c6cd3774" (UID: "1d83861f-abc5-452e-a1d9-3148c6cd3774"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:28:34.937732 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:34.937691 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d83861f-abc5-452e-a1d9-3148c6cd3774-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d83861f-abc5-452e-a1d9-3148c6cd3774-istiod-ca-cert\") pod \"1d83861f-abc5-452e-a1d9-3148c6cd3774\" (UID: \"1d83861f-abc5-452e-a1d9-3148c6cd3774\") " Nov 03 02:28:34.937732 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:34.937695 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d83861f-abc5-452e-a1d9-3148c6cd3774-workload-certs" pod "1d83861f-abc5-452e-a1d9-3148c6cd3774" (UID: "1d83861f-abc5-452e-a1d9-3148c6cd3774"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:28:34.937933 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:34.937728 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d83861f-abc5-452e-a1d9-3148c6cd3774-kube-api-access-fb7v5\" (UniqueName: \"kubernetes.io/projected/1d83861f-abc5-452e-a1d9-3148c6cd3774-kube-api-access-fb7v5\") pod \"1d83861f-abc5-452e-a1d9-3148c6cd3774\" (UID: \"1d83861f-abc5-452e-a1d9-3148c6cd3774\") " Nov 03 02:28:34.937933 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:34.937750 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d83861f-abc5-452e-a1d9-3148c6cd3774-istio-data" pod "1d83861f-abc5-452e-a1d9-3148c6cd3774" (UID: "1d83861f-abc5-452e-a1d9-3148c6cd3774"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:28:34.938038 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:34.938023 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d83861f-abc5-452e-a1d9-3148c6cd3774-workload-socket" pod "1d83861f-abc5-452e-a1d9-3148c6cd3774" (UID: "1d83861f-abc5-452e-a1d9-3148c6cd3774"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:28:34.938091 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:34.938074 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d83861f-abc5-452e-a1d9-3148c6cd3774-istiod-ca-cert" pod "1d83861f-abc5-452e-a1d9-3148c6cd3774" (UID: "1d83861f-abc5-452e-a1d9-3148c6cd3774"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:28:34.938445 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:34.938420 5605 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d83861f-abc5-452e-a1d9-3148c6cd3774-istio-data\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:28:34.938630 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:34.938611 5605 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d83861f-abc5-452e-a1d9-3148c6cd3774-credential-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:28:34.938770 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:34.938747 5605 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d83861f-abc5-452e-a1d9-3148c6cd3774-workload-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:28:34.938770 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:34.938773 5605 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d83861f-abc5-452e-a1d9-3148c6cd3774-workload-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:28:34.938770 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:34.938788 5605 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d83861f-abc5-452e-a1d9-3148c6cd3774-istiod-ca-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:28:34.939775 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:34.939749 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d83861f-abc5-452e-a1d9-3148c6cd3774-istio-token" pod "1d83861f-abc5-452e-a1d9-3148c6cd3774" (UID: "1d83861f-abc5-452e-a1d9-3148c6cd3774"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:28:34.939987 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:34.939958 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d83861f-abc5-452e-a1d9-3148c6cd3774-istio-podinfo" pod "1d83861f-abc5-452e-a1d9-3148c6cd3774" (UID: "1d83861f-abc5-452e-a1d9-3148c6cd3774"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:28:34.939987 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:34.939965 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d83861f-abc5-452e-a1d9-3148c6cd3774-istio-envoy" pod "1d83861f-abc5-452e-a1d9-3148c6cd3774" (UID: "1d83861f-abc5-452e-a1d9-3148c6cd3774"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:28:34.940231 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:34.940214 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d83861f-abc5-452e-a1d9-3148c6cd3774-kube-api-access-fb7v5" pod "1d83861f-abc5-452e-a1d9-3148c6cd3774" (UID: "1d83861f-abc5-452e-a1d9-3148c6cd3774"). InnerVolumeSpecName "kube-api-access-fb7v5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:28:34.949221 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:34.949197 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr"] Nov 03 02:28:34.949451 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:34.949428 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr" podUID="e3b0cfb1-29ab-4570-81b6-4b55627e8067" containerName="istio-proxy" containerID="cri-o://3257f498235885415411dd42ed8752bdbf810d415bb432c5d9d4c59dba99dfb4" gracePeriod=30 Nov 03 02:28:35.039493 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:35.039459 5605 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d83861f-abc5-452e-a1d9-3148c6cd3774-istio-token\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:28:35.039493 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:35.039490 5605 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d83861f-abc5-452e-a1d9-3148c6cd3774-istio-podinfo\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:28:35.039493 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:35.039500 5605 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d83861f-abc5-452e-a1d9-3148c6cd3774-istio-envoy\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:28:35.039732 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:35.039512 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fb7v5\" (UniqueName: \"kubernetes.io/projected/1d83861f-abc5-452e-a1d9-3148c6cd3774-kube-api-access-fb7v5\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:28:35.423040 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:35.422999 5605 generic.go:356] "Generic (PLEG): container finished" podID="1d83861f-abc5-452e-a1d9-3148c6cd3774" containerID="52987e50d164950fed2ae55ed3a5a64e4e1443e07d9fb4ab698587690ca7825d" exitCode=0 Nov 03 02:28:35.423248 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:35.423069 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4gq8z" Nov 03 02:28:35.423248 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:35.423080 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4gq8z" event={"ID":"1d83861f-abc5-452e-a1d9-3148c6cd3774","Type":"ContainerDied","Data":"52987e50d164950fed2ae55ed3a5a64e4e1443e07d9fb4ab698587690ca7825d"} Nov 03 02:28:35.423248 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:35.423139 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4gq8z" event={"ID":"1d83861f-abc5-452e-a1d9-3148c6cd3774","Type":"ContainerDied","Data":"3ef26fb668d20b73c481bb6e525e40cb3afb9f5dad0727a00603ab5d825af592"} Nov 03 02:28:35.423248 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:35.423163 5605 scope.go:122] "RemoveContainer" containerID="52987e50d164950fed2ae55ed3a5a64e4e1443e07d9fb4ab698587690ca7825d" Nov 03 02:28:35.432800 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:35.432781 5605 scope.go:122] "RemoveContainer" containerID="52987e50d164950fed2ae55ed3a5a64e4e1443e07d9fb4ab698587690ca7825d" Nov 03 02:28:35.433101 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:28:35.433080 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52987e50d164950fed2ae55ed3a5a64e4e1443e07d9fb4ab698587690ca7825d\": container with ID starting with 52987e50d164950fed2ae55ed3a5a64e4e1443e07d9fb4ab698587690ca7825d not found: ID does not exist" containerID="52987e50d164950fed2ae55ed3a5a64e4e1443e07d9fb4ab698587690ca7825d" Nov 03 02:28:35.433170 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:35.433109 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52987e50d164950fed2ae55ed3a5a64e4e1443e07d9fb4ab698587690ca7825d"} err="failed to get container status \"52987e50d164950fed2ae55ed3a5a64e4e1443e07d9fb4ab698587690ca7825d\": rpc error: code = NotFound desc = could not find container \"52987e50d164950fed2ae55ed3a5a64e4e1443e07d9fb4ab698587690ca7825d\": container with ID starting with 52987e50d164950fed2ae55ed3a5a64e4e1443e07d9fb4ab698587690ca7825d not found: ID does not exist" Nov 03 02:28:35.575817 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:35.575779 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4gq8z"] Nov 03 02:28:35.622664 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:35.622625 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4gq8z"] Nov 03 02:28:36.849644 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:36.849613 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d83861f-abc5-452e-a1d9-3148c6cd3774" path="/var/lib/kubelet/pods/1d83861f-abc5-452e-a1d9-3148c6cd3774/volumes" Nov 03 02:28:38.873429 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:38.873384 5605 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 02:28:38.873845 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:38.873469 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr" podUID="e3b0cfb1-29ab-4570-81b6-4b55627e8067" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 02:28:40.208951 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:40.208925 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr" Nov 03 02:28:40.321813 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:40.321778 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3b0cfb1-29ab-4570-81b6-4b55627e8067-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3b0cfb1-29ab-4570-81b6-4b55627e8067-istio-data\") pod \"e3b0cfb1-29ab-4570-81b6-4b55627e8067\" (UID: \"e3b0cfb1-29ab-4570-81b6-4b55627e8067\") " Nov 03 02:28:40.321987 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:40.321864 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3b0cfb1-29ab-4570-81b6-4b55627e8067-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b0cfb1-29ab-4570-81b6-4b55627e8067-credential-socket\") pod \"e3b0cfb1-29ab-4570-81b6-4b55627e8067\" (UID: \"e3b0cfb1-29ab-4570-81b6-4b55627e8067\") " Nov 03 02:28:40.321987 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:40.321889 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3b0cfb1-29ab-4570-81b6-4b55627e8067-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3b0cfb1-29ab-4570-81b6-4b55627e8067-workload-certs\") pod \"e3b0cfb1-29ab-4570-81b6-4b55627e8067\" (UID: \"e3b0cfb1-29ab-4570-81b6-4b55627e8067\") " Nov 03 02:28:40.321987 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:40.321909 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3b0cfb1-29ab-4570-81b6-4b55627e8067-istio-token\" (UniqueName: \"kubernetes.io/projected/e3b0cfb1-29ab-4570-81b6-4b55627e8067-istio-token\") pod \"e3b0cfb1-29ab-4570-81b6-4b55627e8067\" (UID: \"e3b0cfb1-29ab-4570-81b6-4b55627e8067\") " Nov 03 02:28:40.321987 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:40.321953 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3b0cfb1-29ab-4570-81b6-4b55627e8067-kube-api-access-h7ctj\" (UniqueName: \"kubernetes.io/projected/e3b0cfb1-29ab-4570-81b6-4b55627e8067-kube-api-access-h7ctj\") pod \"e3b0cfb1-29ab-4570-81b6-4b55627e8067\" (UID: \"e3b0cfb1-29ab-4570-81b6-4b55627e8067\") " Nov 03 02:28:40.322309 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:40.321993 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3b0cfb1-29ab-4570-81b6-4b55627e8067-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b0cfb1-29ab-4570-81b6-4b55627e8067-workload-socket\") pod \"e3b0cfb1-29ab-4570-81b6-4b55627e8067\" (UID: \"e3b0cfb1-29ab-4570-81b6-4b55627e8067\") " Nov 03 02:28:40.322309 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:40.322036 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3b0cfb1-29ab-4570-81b6-4b55627e8067-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3b0cfb1-29ab-4570-81b6-4b55627e8067-istiod-ca-cert\") pod \"e3b0cfb1-29ab-4570-81b6-4b55627e8067\" (UID: \"e3b0cfb1-29ab-4570-81b6-4b55627e8067\") " Nov 03 02:28:40.322309 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:40.322064 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3b0cfb1-29ab-4570-81b6-4b55627e8067-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3b0cfb1-29ab-4570-81b6-4b55627e8067-istio-envoy\") pod \"e3b0cfb1-29ab-4570-81b6-4b55627e8067\" (UID: \"e3b0cfb1-29ab-4570-81b6-4b55627e8067\") " Nov 03 02:28:40.322309 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:40.322077 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3b0cfb1-29ab-4570-81b6-4b55627e8067-istio-data" pod "e3b0cfb1-29ab-4570-81b6-4b55627e8067" (UID: "e3b0cfb1-29ab-4570-81b6-4b55627e8067"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:28:40.322309 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:40.322191 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3b0cfb1-29ab-4570-81b6-4b55627e8067-credential-socket" pod "e3b0cfb1-29ab-4570-81b6-4b55627e8067" (UID: "e3b0cfb1-29ab-4570-81b6-4b55627e8067"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:28:40.322309 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:40.322251 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3b0cfb1-29ab-4570-81b6-4b55627e8067-workload-certs" pod "e3b0cfb1-29ab-4570-81b6-4b55627e8067" (UID: "e3b0cfb1-29ab-4570-81b6-4b55627e8067"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:28:40.322309 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:40.322272 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e3b0cfb1-29ab-4570-81b6-4b55627e8067-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3b0cfb1-29ab-4570-81b6-4b55627e8067-istio-podinfo\") pod \"e3b0cfb1-29ab-4570-81b6-4b55627e8067\" (UID: \"e3b0cfb1-29ab-4570-81b6-4b55627e8067\") " Nov 03 02:28:40.322712 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:40.322511 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3b0cfb1-29ab-4570-81b6-4b55627e8067-istiod-ca-cert" pod "e3b0cfb1-29ab-4570-81b6-4b55627e8067" (UID: "e3b0cfb1-29ab-4570-81b6-4b55627e8067"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:28:40.322712 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:40.322564 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3b0cfb1-29ab-4570-81b6-4b55627e8067-workload-socket" pod "e3b0cfb1-29ab-4570-81b6-4b55627e8067" (UID: "e3b0cfb1-29ab-4570-81b6-4b55627e8067"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:28:40.322884 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:40.322861 5605 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3b0cfb1-29ab-4570-81b6-4b55627e8067-istio-data\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:28:40.322945 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:40.322890 5605 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b0cfb1-29ab-4570-81b6-4b55627e8067-credential-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:28:40.322945 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:40.322906 5605 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3b0cfb1-29ab-4570-81b6-4b55627e8067-workload-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:28:40.322945 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:40.322920 5605 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b0cfb1-29ab-4570-81b6-4b55627e8067-workload-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:28:40.322945 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:40.322935 5605 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3b0cfb1-29ab-4570-81b6-4b55627e8067-istiod-ca-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:28:40.324334 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:40.324307 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3b0cfb1-29ab-4570-81b6-4b55627e8067-kube-api-access-h7ctj" pod "e3b0cfb1-29ab-4570-81b6-4b55627e8067" (UID: "e3b0cfb1-29ab-4570-81b6-4b55627e8067"). InnerVolumeSpecName "kube-api-access-h7ctj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:28:40.324478 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:40.324313 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3b0cfb1-29ab-4570-81b6-4b55627e8067-istio-envoy" pod "e3b0cfb1-29ab-4570-81b6-4b55627e8067" (UID: "e3b0cfb1-29ab-4570-81b6-4b55627e8067"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:28:40.324478 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:40.324424 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3b0cfb1-29ab-4570-81b6-4b55627e8067-istio-podinfo" pod "e3b0cfb1-29ab-4570-81b6-4b55627e8067" (UID: "e3b0cfb1-29ab-4570-81b6-4b55627e8067"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:28:40.324563 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:40.324505 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3b0cfb1-29ab-4570-81b6-4b55627e8067-istio-token" pod "e3b0cfb1-29ab-4570-81b6-4b55627e8067" (UID: "e3b0cfb1-29ab-4570-81b6-4b55627e8067"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:28:40.424578 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:40.424544 5605 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3b0cfb1-29ab-4570-81b6-4b55627e8067-istio-envoy\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:28:40.424578 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:40.424574 5605 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3b0cfb1-29ab-4570-81b6-4b55627e8067-istio-podinfo\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:28:40.424578 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:40.424583 5605 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3b0cfb1-29ab-4570-81b6-4b55627e8067-istio-token\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:28:40.424819 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:40.424592 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7ctj\" (UniqueName: \"kubernetes.io/projected/e3b0cfb1-29ab-4570-81b6-4b55627e8067-kube-api-access-h7ctj\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:28:40.451731 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:40.451691 5605 generic.go:356] "Generic (PLEG): container finished" podID="e3b0cfb1-29ab-4570-81b6-4b55627e8067" containerID="3257f498235885415411dd42ed8752bdbf810d415bb432c5d9d4c59dba99dfb4" exitCode=0 Nov 03 02:28:40.451908 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:40.451738 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr" Nov 03 02:28:40.451908 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:40.451778 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr" event={"ID":"e3b0cfb1-29ab-4570-81b6-4b55627e8067","Type":"ContainerDied","Data":"3257f498235885415411dd42ed8752bdbf810d415bb432c5d9d4c59dba99dfb4"} Nov 03 02:28:40.451908 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:40.451836 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr" event={"ID":"e3b0cfb1-29ab-4570-81b6-4b55627e8067","Type":"ContainerDied","Data":"fa9236f574cfa889f563f7e4711012c26c896614ae89498852573a9f75699318"} Nov 03 02:28:40.451908 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:40.451866 5605 scope.go:122] "RemoveContainer" containerID="3257f498235885415411dd42ed8752bdbf810d415bb432c5d9d4c59dba99dfb4" Nov 03 02:28:40.461989 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:40.461964 5605 scope.go:122] "RemoveContainer" containerID="3257f498235885415411dd42ed8752bdbf810d415bb432c5d9d4c59dba99dfb4" Nov 03 02:28:40.462275 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:28:40.462255 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3257f498235885415411dd42ed8752bdbf810d415bb432c5d9d4c59dba99dfb4\": container with ID starting with 3257f498235885415411dd42ed8752bdbf810d415bb432c5d9d4c59dba99dfb4 not found: ID does not exist" containerID="3257f498235885415411dd42ed8752bdbf810d415bb432c5d9d4c59dba99dfb4" Nov 03 02:28:40.462320 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:40.462283 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3257f498235885415411dd42ed8752bdbf810d415bb432c5d9d4c59dba99dfb4"} err="failed to get container status \"3257f498235885415411dd42ed8752bdbf810d415bb432c5d9d4c59dba99dfb4\": rpc error: code = NotFound desc = could not find container \"3257f498235885415411dd42ed8752bdbf810d415bb432c5d9d4c59dba99dfb4\": container with ID starting with 3257f498235885415411dd42ed8752bdbf810d415bb432c5d9d4c59dba99dfb4 not found: ID does not exist" Nov 03 02:28:40.604929 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:40.604842 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr"] Nov 03 02:28:40.607769 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:40.607734 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-vz6gr"] Nov 03 02:28:40.853845 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:40.853812 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3b0cfb1-29ab-4570-81b6-4b55627e8067" path="/var/lib/kubelet/pods/e3b0cfb1-29ab-4570-81b6-4b55627e8067/volumes" Nov 03 02:28:49.234473 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:49.234428 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5"] Nov 03 02:28:49.235014 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:49.234885 5605 cpu_manager.go:414] "containerMap: removing container" podUID="1d83861f-abc5-452e-a1d9-3148c6cd3774" containerName="istio-proxy" Nov 03 02:28:49.235014 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:49.234896 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d83861f-abc5-452e-a1d9-3148c6cd3774" containerName="istio-proxy" Nov 03 02:28:49.235014 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:49.234924 5605 cpu_manager.go:414] "containerMap: removing container" podUID="e3b0cfb1-29ab-4570-81b6-4b55627e8067" containerName="istio-proxy" Nov 03 02:28:49.235014 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:49.234929 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3b0cfb1-29ab-4570-81b6-4b55627e8067" containerName="istio-proxy" Nov 03 02:28:49.235014 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:49.234996 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d83861f-abc5-452e-a1d9-3148c6cd3774" containerName="istio-proxy" Nov 03 02:28:49.235014 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:49.235006 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3b0cfb1-29ab-4570-81b6-4b55627e8067" containerName="istio-proxy" Nov 03 02:28:49.239665 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:49.239638 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5" Nov 03 02:28:49.350040 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:49.350000 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5"] Nov 03 02:28:49.398407 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:49.392335 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-711318-openshift-default-dockercfg-ln2vj\"" Nov 03 02:28:49.430028 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:49.429984 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e8485b2-ec6f-4344-bee7-37495006a34c-credential-socket\") pod \"e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5\" (UID: \"6e8485b2-ec6f-4344-bee7-37495006a34c\") " pod="openshift-ingress/e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5" Nov 03 02:28:49.430280 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:49.430076 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e8485b2-ec6f-4344-bee7-37495006a34c-istio-envoy\") pod \"e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5\" (UID: \"6e8485b2-ec6f-4344-bee7-37495006a34c\") " pod="openshift-ingress/e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5" Nov 03 02:28:49.430280 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:49.430120 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e8485b2-ec6f-4344-bee7-37495006a34c-workload-certs\") pod \"e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5\" (UID: \"6e8485b2-ec6f-4344-bee7-37495006a34c\") " pod="openshift-ingress/e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5" Nov 03 02:28:49.430280 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:49.430148 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkzx8\" (UniqueName: \"kubernetes.io/projected/6e8485b2-ec6f-4344-bee7-37495006a34c-kube-api-access-wkzx8\") pod \"e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5\" (UID: \"6e8485b2-ec6f-4344-bee7-37495006a34c\") " pod="openshift-ingress/e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5" Nov 03 02:28:49.430280 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:49.430180 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e8485b2-ec6f-4344-bee7-37495006a34c-workload-socket\") pod \"e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5\" (UID: \"6e8485b2-ec6f-4344-bee7-37495006a34c\") " pod="openshift-ingress/e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5" Nov 03 02:28:49.430280 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:49.430262 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e8485b2-ec6f-4344-bee7-37495006a34c-istiod-ca-cert\") pod \"e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5\" (UID: \"6e8485b2-ec6f-4344-bee7-37495006a34c\") " pod="openshift-ingress/e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5" Nov 03 02:28:49.430280 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:49.430299 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e8485b2-ec6f-4344-bee7-37495006a34c-istio-token\") pod \"e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5\" (UID: \"6e8485b2-ec6f-4344-bee7-37495006a34c\") " pod="openshift-ingress/e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5" Nov 03 02:28:49.430736 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:49.430382 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e8485b2-ec6f-4344-bee7-37495006a34c-istio-data\") pod \"e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5\" (UID: \"6e8485b2-ec6f-4344-bee7-37495006a34c\") " pod="openshift-ingress/e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5" Nov 03 02:28:49.430736 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:49.430422 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e8485b2-ec6f-4344-bee7-37495006a34c-istio-podinfo\") pod \"e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5\" (UID: \"6e8485b2-ec6f-4344-bee7-37495006a34c\") " pod="openshift-ingress/e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5" Nov 03 02:28:49.531497 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:49.531395 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e8485b2-ec6f-4344-bee7-37495006a34c-istiod-ca-cert\") pod \"e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5\" (UID: \"6e8485b2-ec6f-4344-bee7-37495006a34c\") " pod="openshift-ingress/e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5" Nov 03 02:28:49.531497 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:49.531455 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e8485b2-ec6f-4344-bee7-37495006a34c-istio-token\") pod \"e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5\" (UID: \"6e8485b2-ec6f-4344-bee7-37495006a34c\") " pod="openshift-ingress/e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5" Nov 03 02:28:49.531738 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:49.531522 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e8485b2-ec6f-4344-bee7-37495006a34c-istio-data\") pod \"e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5\" (UID: \"6e8485b2-ec6f-4344-bee7-37495006a34c\") " pod="openshift-ingress/e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5" Nov 03 02:28:49.531738 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:49.531558 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e8485b2-ec6f-4344-bee7-37495006a34c-istio-podinfo\") pod \"e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5\" (UID: \"6e8485b2-ec6f-4344-bee7-37495006a34c\") " pod="openshift-ingress/e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5" Nov 03 02:28:49.531738 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:49.531618 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e8485b2-ec6f-4344-bee7-37495006a34c-credential-socket\") pod \"e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5\" (UID: \"6e8485b2-ec6f-4344-bee7-37495006a34c\") " pod="openshift-ingress/e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5" Nov 03 02:28:49.531738 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:49.531684 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e8485b2-ec6f-4344-bee7-37495006a34c-istio-envoy\") pod \"e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5\" (UID: \"6e8485b2-ec6f-4344-bee7-37495006a34c\") " pod="openshift-ingress/e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5" Nov 03 02:28:49.531738 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:49.531729 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e8485b2-ec6f-4344-bee7-37495006a34c-workload-certs\") pod \"e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5\" (UID: \"6e8485b2-ec6f-4344-bee7-37495006a34c\") " pod="openshift-ingress/e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5" Nov 03 02:28:49.532006 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:49.531759 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkzx8\" (UniqueName: \"kubernetes.io/projected/6e8485b2-ec6f-4344-bee7-37495006a34c-kube-api-access-wkzx8\") pod \"e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5\" (UID: \"6e8485b2-ec6f-4344-bee7-37495006a34c\") " pod="openshift-ingress/e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5" Nov 03 02:28:49.532006 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:49.531792 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e8485b2-ec6f-4344-bee7-37495006a34c-workload-socket\") pod \"e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5\" (UID: \"6e8485b2-ec6f-4344-bee7-37495006a34c\") " pod="openshift-ingress/e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5" Nov 03 02:28:49.532006 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:49.531904 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e8485b2-ec6f-4344-bee7-37495006a34c-istio-data\") pod \"e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5\" (UID: \"6e8485b2-ec6f-4344-bee7-37495006a34c\") " pod="openshift-ingress/e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5" Nov 03 02:28:49.532164 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:49.532139 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e8485b2-ec6f-4344-bee7-37495006a34c-workload-socket\") pod \"e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5\" (UID: \"6e8485b2-ec6f-4344-bee7-37495006a34c\") " pod="openshift-ingress/e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5" Nov 03 02:28:49.532226 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:49.532171 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e8485b2-ec6f-4344-bee7-37495006a34c-credential-socket\") pod \"e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5\" (UID: \"6e8485b2-ec6f-4344-bee7-37495006a34c\") " pod="openshift-ingress/e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5" Nov 03 02:28:49.532282 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:49.532226 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e8485b2-ec6f-4344-bee7-37495006a34c-istiod-ca-cert\") pod \"e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5\" (UID: \"6e8485b2-ec6f-4344-bee7-37495006a34c\") " pod="openshift-ingress/e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5" Nov 03 02:28:49.532458 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:49.532409 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e8485b2-ec6f-4344-bee7-37495006a34c-workload-certs\") pod \"e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5\" (UID: \"6e8485b2-ec6f-4344-bee7-37495006a34c\") " pod="openshift-ingress/e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5" Nov 03 02:28:49.533994 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:49.533969 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e8485b2-ec6f-4344-bee7-37495006a34c-istio-envoy\") pod \"e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5\" (UID: \"6e8485b2-ec6f-4344-bee7-37495006a34c\") " pod="openshift-ingress/e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5" Nov 03 02:28:49.534432 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:49.534335 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e8485b2-ec6f-4344-bee7-37495006a34c-istio-podinfo\") pod \"e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5\" (UID: \"6e8485b2-ec6f-4344-bee7-37495006a34c\") " pod="openshift-ingress/e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5" Nov 03 02:28:49.587099 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:49.587058 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e8485b2-ec6f-4344-bee7-37495006a34c-istio-token\") pod \"e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5\" (UID: \"6e8485b2-ec6f-4344-bee7-37495006a34c\") " pod="openshift-ingress/e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5" Nov 03 02:28:49.587390 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:49.587347 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkzx8\" (UniqueName: \"kubernetes.io/projected/6e8485b2-ec6f-4344-bee7-37495006a34c-kube-api-access-wkzx8\") pod \"e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5\" (UID: \"6e8485b2-ec6f-4344-bee7-37495006a34c\") " pod="openshift-ingress/e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5" Nov 03 02:28:49.852711 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:49.852673 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5" Nov 03 02:28:50.287371 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:50.287271 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5"] Nov 03 02:28:50.293027 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:28:50.292994 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e8485b2_ec6f_4344_bee7_37495006a34c.slice/crio-eef375138aaff8b9c291d4b1d8f1e9880d2477511f7a8992417e7b9e1788eb38 WatchSource:0}: Error finding container eef375138aaff8b9c291d4b1d8f1e9880d2477511f7a8992417e7b9e1788eb38: Status 404 returned error can't find the container with id eef375138aaff8b9c291d4b1d8f1e9880d2477511f7a8992417e7b9e1788eb38 Nov 03 02:28:50.295792 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:50.295757 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:28:50.295895 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:50.295831 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:28:50.295895 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:50.295862 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:28:50.511208 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:50.511164 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5" event={"ID":"6e8485b2-ec6f-4344-bee7-37495006a34c","Type":"ContainerStarted","Data":"d0a29f070628364b4e12e88048780351f44c00f620f53883a89bebec92ab2128"} Nov 03 02:28:50.511394 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:50.511217 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5" event={"ID":"6e8485b2-ec6f-4344-bee7-37495006a34c","Type":"ContainerStarted","Data":"eef375138aaff8b9c291d4b1d8f1e9880d2477511f7a8992417e7b9e1788eb38"} Nov 03 02:28:50.621992 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:50.621925 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5" podStartSLOduration=1.6219112629999999 podStartE2EDuration="1.621911263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:28:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:28:50.620454684 +0000 UTC m=+2207.879652842" watchObservedRunningTime="2026-11-03 02:28:50.621911263 +0000 UTC m=+2207.881109416" Nov 03 02:28:50.853116 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:50.853087 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5" Nov 03 02:28:51.034750 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:51.034658 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:28:51.859180 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:51.859152 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5" Nov 03 02:28:52.523574 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:52.523544 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5" Nov 03 02:28:52.524492 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:52.524470 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5" Nov 03 02:28:53.917284 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:28:53.917231 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:28:57.270485 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:57.270449 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-711318-5967965975-wf4df"] Nov 03 02:28:57.274411 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:57.274385 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-711318-5967965975-wf4df" Nov 03 02:28:57.325008 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:57.324974 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-711318\"" Nov 03 02:28:57.341452 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:57.332990 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89lv4\" (UniqueName: \"kubernetes.io/projected/15cdb9a2-9888-463a-85c0-a2b70d760b5d-kube-api-access-89lv4\") pod \"payload-pre-processing-e2e-praxis-b-711318-5967965975-wf4df\" (UID: \"15cdb9a2-9888-463a-85c0-a2b70d760b5d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-711318-5967965975-wf4df" Nov 03 02:28:57.341452 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:57.333084 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15cdb9a2-9888-463a-85c0-a2b70d760b5d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-711318-5967965975-wf4df\" (UID: \"15cdb9a2-9888-463a-85c0-a2b70d760b5d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-711318-5967965975-wf4df" Nov 03 02:28:57.376146 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:57.376098 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-711318-dockercfg-gvpkl\"" Nov 03 02:28:57.386640 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:57.386609 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-711318-5967965975-wf4df"] Nov 03 02:28:57.434034 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:57.433998 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89lv4\" (UniqueName: \"kubernetes.io/projected/15cdb9a2-9888-463a-85c0-a2b70d760b5d-kube-api-access-89lv4\") pod \"payload-pre-processing-e2e-praxis-b-711318-5967965975-wf4df\" (UID: \"15cdb9a2-9888-463a-85c0-a2b70d760b5d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-711318-5967965975-wf4df" Nov 03 02:28:57.434207 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:57.434054 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15cdb9a2-9888-463a-85c0-a2b70d760b5d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-711318-5967965975-wf4df\" (UID: \"15cdb9a2-9888-463a-85c0-a2b70d760b5d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-711318-5967965975-wf4df" Nov 03 02:28:57.434808 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:57.434786 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15cdb9a2-9888-463a-85c0-a2b70d760b5d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-711318-5967965975-wf4df\" (UID: \"15cdb9a2-9888-463a-85c0-a2b70d760b5d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-711318-5967965975-wf4df" Nov 03 02:28:57.487327 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:57.487281 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89lv4\" (UniqueName: \"kubernetes.io/projected/15cdb9a2-9888-463a-85c0-a2b70d760b5d-kube-api-access-89lv4\") pod \"payload-pre-processing-e2e-praxis-b-711318-5967965975-wf4df\" (UID: \"15cdb9a2-9888-463a-85c0-a2b70d760b5d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-711318-5967965975-wf4df" Nov 03 02:28:57.588605 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:57.588568 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-711318-5967965975-wf4df" Nov 03 02:28:57.964061 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:57.963973 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-711318-5967965975-wf4df"] Nov 03 02:28:57.970385 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:28:57.970338 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15cdb9a2_9888_463a_85c0_a2b70d760b5d.slice/crio-9e77f0b376003419b3e1ab87dcfe5615be86c179c50a4e01b75c0561974ddbbb WatchSource:0}: Error finding container 9e77f0b376003419b3e1ab87dcfe5615be86c179c50a4e01b75c0561974ddbbb: Status 404 returned error can't find the container with id 9e77f0b376003419b3e1ab87dcfe5615be86c179c50a4e01b75c0561974ddbbb Nov 03 02:28:57.974401 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:57.974377 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7f9d9d-794bb5df47-hlcwq"] Nov 03 02:28:58.087949 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:58.087913 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7f9d9d-55455d767-xpmtx"] Nov 03 02:28:58.556729 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:58.556695 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-711318-5967965975-wf4df" event={"ID":"15cdb9a2-9888-463a-85c0-a2b70d760b5d","Type":"ContainerStarted","Data":"7463dcf8859ed35c39434318b9dd77fd79245d9e61e29752c2fb6fdbe15deee1"} Nov 03 02:28:58.557193 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:58.556758 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-711318-5967965975-wf4df" event={"ID":"15cdb9a2-9888-463a-85c0-a2b70d760b5d","Type":"ContainerStarted","Data":"9e77f0b376003419b3e1ab87dcfe5615be86c179c50a4e01b75c0561974ddbbb"} Nov 03 02:28:58.557193 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:58.556798 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-711318-5967965975-wf4df" Nov 03 02:28:58.662481 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:28:58.662422 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-711318-5967965975-wf4df" podStartSLOduration=1.662406237 podStartE2EDuration="1.662406237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:28:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:28:58.661966411 +0000 UTC m=+2215.921164591" watchObservedRunningTime="2026-11-03 02:28:58.662406237 +0000 UTC m=+2215.921604390" Nov 03 02:29:02.974669 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:02.974621 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-7f9d9d-794bb5df47-hlcwq" podUID="f0e676c5-fdda-4636-86de-941149320c96" containerName="payload-processing" containerID="cri-o://924cd65533e200ed886e359b3556cfe9d5579b0b23fb9228aed2964d6aa43c2d" gracePeriod=25 Nov 03 02:29:03.088373 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:03.088319 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7f9d9d-55455d767-xpmtx" podUID="e429518f-695c-46c8-839b-fade2d52c105" containerName="payload-pre-processing" containerID="cri-o://45b423504e372f01c9069813d85a32d1ff6133ff6b5c69d9762014f2cd38d84c" gracePeriod=25 Nov 03 02:29:03.229470 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:03.229397 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-7f9d9d-794bb5df47-hlcwq" Nov 03 02:29:03.320152 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:03.320113 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0e676c5-fdda-4636-86de-941149320c96-kube-api-access-5ngc6\" (UniqueName: \"kubernetes.io/projected/f0e676c5-fdda-4636-86de-941149320c96-kube-api-access-5ngc6\") pod \"f0e676c5-fdda-4636-86de-941149320c96\" (UID: \"f0e676c5-fdda-4636-86de-941149320c96\") " Nov 03 02:29:03.320666 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:03.320275 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0e676c5-fdda-4636-86de-941149320c96-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0e676c5-fdda-4636-86de-941149320c96-plugins-config-volume\") pod \"f0e676c5-fdda-4636-86de-941149320c96\" (UID: \"f0e676c5-fdda-4636-86de-941149320c96\") " Nov 03 02:29:03.320772 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:03.320749 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0e676c5-fdda-4636-86de-941149320c96-plugins-config-volume" pod "f0e676c5-fdda-4636-86de-941149320c96" (UID: "f0e676c5-fdda-4636-86de-941149320c96"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:29:03.322892 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:03.322862 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0e676c5-fdda-4636-86de-941149320c96-kube-api-access-5ngc6" pod "f0e676c5-fdda-4636-86de-941149320c96" (UID: "f0e676c5-fdda-4636-86de-941149320c96"). InnerVolumeSpecName "kube-api-access-5ngc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:29:03.378854 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:03.378820 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7f9d9d-55455d767-xpmtx" Nov 03 02:29:03.422009 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:03.421973 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ngc6\" (UniqueName: \"kubernetes.io/projected/f0e676c5-fdda-4636-86de-941149320c96-kube-api-access-5ngc6\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:29:03.422009 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:03.422003 5605 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0e676c5-fdda-4636-86de-941149320c96-plugins-config-volume\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:29:03.522611 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:03.522526 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e429518f-695c-46c8-839b-fade2d52c105-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e429518f-695c-46c8-839b-fade2d52c105-plugins-config-volume\") pod \"e429518f-695c-46c8-839b-fade2d52c105\" (UID: \"e429518f-695c-46c8-839b-fade2d52c105\") " Nov 03 02:29:03.522765 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:03.522628 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e429518f-695c-46c8-839b-fade2d52c105-kube-api-access-bmqlf\" (UniqueName: \"kubernetes.io/projected/e429518f-695c-46c8-839b-fade2d52c105-kube-api-access-bmqlf\") pod \"e429518f-695c-46c8-839b-fade2d52c105\" (UID: \"e429518f-695c-46c8-839b-fade2d52c105\") " Nov 03 02:29:03.523004 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:03.522975 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e429518f-695c-46c8-839b-fade2d52c105-plugins-config-volume" pod "e429518f-695c-46c8-839b-fade2d52c105" (UID: "e429518f-695c-46c8-839b-fade2d52c105"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:29:03.524742 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:03.524719 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e429518f-695c-46c8-839b-fade2d52c105-kube-api-access-bmqlf" pod "e429518f-695c-46c8-839b-fade2d52c105" (UID: "e429518f-695c-46c8-839b-fade2d52c105"). InnerVolumeSpecName "kube-api-access-bmqlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:29:03.592041 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:03.591994 5605 generic.go:356] "Generic (PLEG): container finished" podID="f0e676c5-fdda-4636-86de-941149320c96" containerID="924cd65533e200ed886e359b3556cfe9d5579b0b23fb9228aed2964d6aa43c2d" exitCode=0 Nov 03 02:29:03.592410 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:03.592344 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-7f9d9d-794bb5df47-hlcwq" event={"ID":"f0e676c5-fdda-4636-86de-941149320c96","Type":"ContainerDied","Data":"924cd65533e200ed886e359b3556cfe9d5579b0b23fb9228aed2964d6aa43c2d"} Nov 03 02:29:03.592601 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:03.592498 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-7f9d9d-794bb5df47-hlcwq" event={"ID":"f0e676c5-fdda-4636-86de-941149320c96","Type":"ContainerDied","Data":"82b4d96cd78434c818d1f9704941ad6b83aef9212c840401e82f1d3f8b9bb45d"} Nov 03 02:29:03.592601 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:03.592538 5605 scope.go:122] "RemoveContainer" containerID="924cd65533e200ed886e359b3556cfe9d5579b0b23fb9228aed2964d6aa43c2d" Nov 03 02:29:03.592714 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:03.592396 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-7f9d9d-794bb5df47-hlcwq" Nov 03 02:29:03.594532 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:03.594504 5605 generic.go:356] "Generic (PLEG): container finished" podID="e429518f-695c-46c8-839b-fade2d52c105" containerID="45b423504e372f01c9069813d85a32d1ff6133ff6b5c69d9762014f2cd38d84c" exitCode=0 Nov 03 02:29:03.594661 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:03.594539 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7f9d9d-55455d767-xpmtx" event={"ID":"e429518f-695c-46c8-839b-fade2d52c105","Type":"ContainerDied","Data":"45b423504e372f01c9069813d85a32d1ff6133ff6b5c69d9762014f2cd38d84c"} Nov 03 02:29:03.594661 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:03.594578 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7f9d9d-55455d767-xpmtx" event={"ID":"e429518f-695c-46c8-839b-fade2d52c105","Type":"ContainerDied","Data":"60ae9eddeb11b5a19bf7de232b130b32e99d39c1b5ea6d319245ac50265abb05"} Nov 03 02:29:03.594661 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:03.594584 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7f9d9d-55455d767-xpmtx" Nov 03 02:29:03.607497 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:03.607424 5605 scope.go:122] "RemoveContainer" containerID="924cd65533e200ed886e359b3556cfe9d5579b0b23fb9228aed2964d6aa43c2d" Nov 03 02:29:03.607697 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:29:03.607671 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"924cd65533e200ed886e359b3556cfe9d5579b0b23fb9228aed2964d6aa43c2d\": container with ID starting with 924cd65533e200ed886e359b3556cfe9d5579b0b23fb9228aed2964d6aa43c2d not found: ID does not exist" containerID="924cd65533e200ed886e359b3556cfe9d5579b0b23fb9228aed2964d6aa43c2d" Nov 03 02:29:03.607737 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:03.607700 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"924cd65533e200ed886e359b3556cfe9d5579b0b23fb9228aed2964d6aa43c2d"} err="failed to get container status \"924cd65533e200ed886e359b3556cfe9d5579b0b23fb9228aed2964d6aa43c2d\": rpc error: code = NotFound desc = could not find container \"924cd65533e200ed886e359b3556cfe9d5579b0b23fb9228aed2964d6aa43c2d\": container with ID starting with 924cd65533e200ed886e359b3556cfe9d5579b0b23fb9228aed2964d6aa43c2d not found: ID does not exist" Nov 03 02:29:03.607737 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:03.607719 5605 scope.go:122] "RemoveContainer" containerID="45b423504e372f01c9069813d85a32d1ff6133ff6b5c69d9762014f2cd38d84c" Nov 03 02:29:03.617145 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:03.617124 5605 scope.go:122] "RemoveContainer" containerID="45b423504e372f01c9069813d85a32d1ff6133ff6b5c69d9762014f2cd38d84c" Nov 03 02:29:03.617445 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:29:03.617417 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45b423504e372f01c9069813d85a32d1ff6133ff6b5c69d9762014f2cd38d84c\": container with ID starting with 45b423504e372f01c9069813d85a32d1ff6133ff6b5c69d9762014f2cd38d84c not found: ID does not exist" containerID="45b423504e372f01c9069813d85a32d1ff6133ff6b5c69d9762014f2cd38d84c" Nov 03 02:29:03.617551 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:03.617449 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45b423504e372f01c9069813d85a32d1ff6133ff6b5c69d9762014f2cd38d84c"} err="failed to get container status \"45b423504e372f01c9069813d85a32d1ff6133ff6b5c69d9762014f2cd38d84c\": rpc error: code = NotFound desc = could not find container \"45b423504e372f01c9069813d85a32d1ff6133ff6b5c69d9762014f2cd38d84c\": container with ID starting with 45b423504e372f01c9069813d85a32d1ff6133ff6b5c69d9762014f2cd38d84c not found: ID does not exist" Nov 03 02:29:03.624137 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:03.624112 5605 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e429518f-695c-46c8-839b-fade2d52c105-plugins-config-volume\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:29:03.624215 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:03.624140 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmqlf\" (UniqueName: \"kubernetes.io/projected/e429518f-695c-46c8-839b-fade2d52c105-kube-api-access-bmqlf\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:29:03.747487 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:03.747445 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7f9d9d-55455d767-xpmtx"] Nov 03 02:29:03.750772 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:03.750743 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7f9d9d-55455d767-xpmtx"] Nov 03 02:29:03.887518 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:03.887443 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7f9d9d-794bb5df47-hlcwq"] Nov 03 02:29:03.889423 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:03.889337 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7f9d9d-794bb5df47-hlcwq"] Nov 03 02:29:04.850009 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:04.849974 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e429518f-695c-46c8-839b-fade2d52c105" path="/var/lib/kubelet/pods/e429518f-695c-46c8-839b-fade2d52c105/volumes" Nov 03 02:29:04.850421 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:04.850393 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0e676c5-fdda-4636-86de-941149320c96" path="/var/lib/kubelet/pods/f0e676c5-fdda-4636-86de-941149320c96/volumes" Nov 03 02:29:09.566670 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:09.566640 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-711318-5967965975-wf4df" Nov 03 02:29:22.533471 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:22.533436 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:29:23.927068 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:29:23.927018 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:29:26.506108 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:26.506072 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e013133c-f9hgs"] Nov 03 02:29:26.506590 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:26.506575 5605 cpu_manager.go:414] "containerMap: removing container" podUID="f0e676c5-fdda-4636-86de-941149320c96" containerName="payload-processing" Nov 03 02:29:26.506654 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:26.506591 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0e676c5-fdda-4636-86de-941149320c96" containerName="payload-processing" Nov 03 02:29:26.506654 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:26.506610 5605 cpu_manager.go:414] "containerMap: removing container" podUID="e429518f-695c-46c8-839b-fade2d52c105" containerName="payload-pre-processing" Nov 03 02:29:26.506654 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:26.506617 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e429518f-695c-46c8-839b-fade2d52c105" containerName="payload-pre-processing" Nov 03 02:29:26.506748 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:26.506727 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="e429518f-695c-46c8-839b-fade2d52c105" containerName="payload-pre-processing" Nov 03 02:29:26.506748 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:26.506739 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0e676c5-fdda-4636-86de-941149320c96" containerName="payload-processing" Nov 03 02:29:26.510552 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:26.510531 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e013133c-f9hgs" Nov 03 02:29:26.564382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:26.561799 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:29:26.576385 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:26.569073 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13192b33-c9a3-422f-a2cd-79c52dd55715-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e013133c-f9hgs\" (UID: \"13192b33-c9a3-422f-a2cd-79c52dd55715\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e013133c-f9hgs" Nov 03 02:29:26.576385 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:26.569209 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9jg6\" (UniqueName: \"kubernetes.io/projected/13192b33-c9a3-422f-a2cd-79c52dd55715-kube-api-access-r9jg6\") pod \"maas-api-revoke-sub-e013133c-f9hgs\" (UID: \"13192b33-c9a3-422f-a2cd-79c52dd55715\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e013133c-f9hgs" Nov 03 02:29:26.576385 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:26.569269 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13192b33-c9a3-422f-a2cd-79c52dd55715-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e013133c-f9hgs\" (UID: \"13192b33-c9a3-422f-a2cd-79c52dd55715\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e013133c-f9hgs" Nov 03 02:29:26.608607 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:26.608572 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:29:26.608607 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:26.608578 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:29:26.608893 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:26.608632 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:29:26.610846 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:26.610817 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e013133c-f9hgs"] Nov 03 02:29:26.671380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:26.670416 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9jg6\" (UniqueName: \"kubernetes.io/projected/13192b33-c9a3-422f-a2cd-79c52dd55715-kube-api-access-r9jg6\") pod \"maas-api-revoke-sub-e013133c-f9hgs\" (UID: \"13192b33-c9a3-422f-a2cd-79c52dd55715\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e013133c-f9hgs" Nov 03 02:29:26.671380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:26.670496 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13192b33-c9a3-422f-a2cd-79c52dd55715-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e013133c-f9hgs\" (UID: \"13192b33-c9a3-422f-a2cd-79c52dd55715\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e013133c-f9hgs" Nov 03 02:29:26.671380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:26.670606 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13192b33-c9a3-422f-a2cd-79c52dd55715-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e013133c-f9hgs\" (UID: \"13192b33-c9a3-422f-a2cd-79c52dd55715\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e013133c-f9hgs" Nov 03 02:29:26.680387 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:26.672130 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13192b33-c9a3-422f-a2cd-79c52dd55715-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e013133c-f9hgs\" (UID: \"13192b33-c9a3-422f-a2cd-79c52dd55715\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e013133c-f9hgs" Nov 03 02:29:26.680387 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:26.674793 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13192b33-c9a3-422f-a2cd-79c52dd55715-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e013133c-f9hgs\" (UID: \"13192b33-c9a3-422f-a2cd-79c52dd55715\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e013133c-f9hgs" Nov 03 02:29:26.723827 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:26.723795 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9jg6\" (UniqueName: \"kubernetes.io/projected/13192b33-c9a3-422f-a2cd-79c52dd55715-kube-api-access-r9jg6\") pod \"maas-api-revoke-sub-e013133c-f9hgs\" (UID: \"13192b33-c9a3-422f-a2cd-79c52dd55715\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e013133c-f9hgs" Nov 03 02:29:26.824421 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:26.824382 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e013133c-f9hgs" Nov 03 02:29:27.216050 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:27.215951 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e013133c-f9hgs"] Nov 03 02:29:27.222242 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:29:27.222210 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13192b33_c9a3_422f_a2cd_79c52dd55715.slice/crio-3c010ebeeb7ad5bc989f0ece81cd06204124e1392b73b847af4372ac97a36920 WatchSource:0}: Error finding container 3c010ebeeb7ad5bc989f0ece81cd06204124e1392b73b847af4372ac97a36920: Status 404 returned error can't find the container with id 3c010ebeeb7ad5bc989f0ece81cd06204124e1392b73b847af4372ac97a36920 Nov 03 02:29:27.738833 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:27.738792 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e013133c-f9hgs" event={"ID":"13192b33-c9a3-422f-a2cd-79c52dd55715","Type":"ContainerStarted","Data":"98dcc5b6508cb9db19f18809b31532088bc00726d8e2ceaa49c41ebea04ebf56"} Nov 03 02:29:27.739292 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:27.738849 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e013133c-f9hgs" event={"ID":"13192b33-c9a3-422f-a2cd-79c52dd55715","Type":"ContainerStarted","Data":"3c010ebeeb7ad5bc989f0ece81cd06204124e1392b73b847af4372ac97a36920"} Nov 03 02:29:28.746552 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:28.746446 5605 generic.go:356] "Generic (PLEG): container finished" podID="13192b33-c9a3-422f-a2cd-79c52dd55715" containerID="98dcc5b6508cb9db19f18809b31532088bc00726d8e2ceaa49c41ebea04ebf56" exitCode=0 Nov 03 02:29:28.746552 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:28.746538 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e013133c-f9hgs" event={"ID":"13192b33-c9a3-422f-a2cd-79c52dd55715","Type":"ContainerDied","Data":"98dcc5b6508cb9db19f18809b31532088bc00726d8e2ceaa49c41ebea04ebf56"} Nov 03 02:29:29.917807 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:29.917777 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e013133c-f9hgs" Nov 03 02:29:30.015796 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:30.015758 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13192b33-c9a3-422f-a2cd-79c52dd55715-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13192b33-c9a3-422f-a2cd-79c52dd55715-maas-api-openshift-ca\") pod \"13192b33-c9a3-422f-a2cd-79c52dd55715\" (UID: \"13192b33-c9a3-422f-a2cd-79c52dd55715\") " Nov 03 02:29:30.015983 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:30.015828 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13192b33-c9a3-422f-a2cd-79c52dd55715-kube-api-access-r9jg6\" (UniqueName: \"kubernetes.io/projected/13192b33-c9a3-422f-a2cd-79c52dd55715-kube-api-access-r9jg6\") pod \"13192b33-c9a3-422f-a2cd-79c52dd55715\" (UID: \"13192b33-c9a3-422f-a2cd-79c52dd55715\") " Nov 03 02:29:30.015983 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:30.015898 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13192b33-c9a3-422f-a2cd-79c52dd55715-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13192b33-c9a3-422f-a2cd-79c52dd55715-maas-api-xks-ca\") pod \"13192b33-c9a3-422f-a2cd-79c52dd55715\" (UID: \"13192b33-c9a3-422f-a2cd-79c52dd55715\") " Nov 03 02:29:30.016184 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:30.016154 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13192b33-c9a3-422f-a2cd-79c52dd55715-maas-api-openshift-ca" pod "13192b33-c9a3-422f-a2cd-79c52dd55715" (UID: "13192b33-c9a3-422f-a2cd-79c52dd55715"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:29:30.016290 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:30.016269 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13192b33-c9a3-422f-a2cd-79c52dd55715-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:29:30.018490 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:30.018460 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13192b33-c9a3-422f-a2cd-79c52dd55715-maas-api-xks-ca" pod "13192b33-c9a3-422f-a2cd-79c52dd55715" (UID: "13192b33-c9a3-422f-a2cd-79c52dd55715"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:29:30.019100 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:30.019080 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13192b33-c9a3-422f-a2cd-79c52dd55715-kube-api-access-r9jg6" pod "13192b33-c9a3-422f-a2cd-79c52dd55715" (UID: "13192b33-c9a3-422f-a2cd-79c52dd55715"). InnerVolumeSpecName "kube-api-access-r9jg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:29:30.117294 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:30.117255 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9jg6\" (UniqueName: \"kubernetes.io/projected/13192b33-c9a3-422f-a2cd-79c52dd55715-kube-api-access-r9jg6\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:29:30.117294 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:30.117287 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13192b33-c9a3-422f-a2cd-79c52dd55715-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:29:30.760329 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:30.760290 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e013133c-f9hgs" Nov 03 02:29:30.760571 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:30.760333 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e013133c-f9hgs" event={"ID":"13192b33-c9a3-422f-a2cd-79c52dd55715","Type":"ContainerDied","Data":"3c010ebeeb7ad5bc989f0ece81cd06204124e1392b73b847af4372ac97a36920"} Nov 03 02:29:30.760571 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:30.760438 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c010ebeeb7ad5bc989f0ece81cd06204124e1392b73b847af4372ac97a36920" Nov 03 02:29:32.773616 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:32.773523 5605 generic.go:356] "Generic (PLEG): container finished" podID="fb2bd860-6aa6-4f34-a90a-d599c74c4cba" containerID="5f3cb2962cf3a2c311567b52bdf4f52f4217e88dceb011d141514e2b6096f540" exitCode=0 Nov 03 02:29:32.773616 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:32.773591 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn" event={"ID":"fb2bd860-6aa6-4f34-a90a-d599c74c4cba","Type":"ContainerDied","Data":"5f3cb2962cf3a2c311567b52bdf4f52f4217e88dceb011d141514e2b6096f540"} Nov 03 02:29:32.774056 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:32.773976 5605 scope.go:122] "RemoveContainer" containerID="5f3cb2962cf3a2c311567b52bdf4f52f4217e88dceb011d141514e2b6096f540" Nov 03 02:29:32.775532 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:32.775478 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:29:32.775642 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:32.775576 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:29:32.775642 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:32.775609 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:29:33.786207 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:33.786172 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn" event={"ID":"fb2bd860-6aa6-4f34-a90a-d599c74c4cba","Type":"ContainerStarted","Data":"dc5399d0c525d8ee88da7da8052ae368ccf311b045adc2048badb9d3596b978d"} Nov 03 02:29:34.170799 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:34.170760 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn" Nov 03 02:29:34.178011 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:34.177970 5605 patch_prober.go:29] interesting pod/e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 03 02:29:34.178309 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:34.178279 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn" podUID="fb2bd860-6aa6-4f34-a90a-d599c74c4cba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 03 02:29:35.169384 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:35.159786 5605 patch_prober.go:29] interesting pod/e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 03 02:29:35.169384 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:35.159869 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn" podUID="fb2bd860-6aa6-4f34-a90a-d599c74c4cba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 03 02:29:36.165839 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:36.165796 5605 patch_prober.go:29] interesting pod/e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 03 02:29:36.166158 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:36.166125 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn" podUID="fb2bd860-6aa6-4f34-a90a-d599c74c4cba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 03 02:29:36.175487 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:36.175453 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn" Nov 03 02:29:37.174384 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:37.170823 5605 patch_prober.go:29] interesting pod/e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 03 02:29:37.174384 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:37.170892 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn" podUID="fb2bd860-6aa6-4f34-a90a-d599c74c4cba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 03 02:29:38.174383 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:38.168828 5605 patch_prober.go:29] interesting pod/e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 03 02:29:38.174383 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:38.168896 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn" podUID="fb2bd860-6aa6-4f34-a90a-d599c74c4cba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 03 02:29:39.164061 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:39.164020 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn" Nov 03 02:29:39.164957 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:39.164935 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn" Nov 03 02:29:43.839397 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:43.839283 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1db7f710-z7ksr"] Nov 03 02:29:43.839797 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:43.839773 5605 cpu_manager.go:414] "containerMap: removing container" podUID="13192b33-c9a3-422f-a2cd-79c52dd55715" containerName="revoke-keys" Nov 03 02:29:43.839797 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:43.839788 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13192b33-c9a3-422f-a2cd-79c52dd55715" containerName="revoke-keys" Nov 03 02:29:43.839906 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:43.839889 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="13192b33-c9a3-422f-a2cd-79c52dd55715" containerName="revoke-keys" Nov 03 02:29:43.843399 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:43.843342 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1db7f710-z7ksr" Nov 03 02:29:43.896141 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:43.896109 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1db7f710-z7ksr"] Nov 03 02:29:43.940791 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:43.940758 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:29:43.940791 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:43.940787 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:29:43.940993 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:43.940794 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:29:43.941141 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:43.941120 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:29:43.995177 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:43.995149 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d2a3dba-562a-4ee3-b9a1-d6845a4d93b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1db7f710-z7ksr\" (UID: \"2d2a3dba-562a-4ee3-b9a1-d6845a4d93b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1db7f710-z7ksr" Nov 03 02:29:43.995345 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:43.995181 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d2a3dba-562a-4ee3-b9a1-d6845a4d93b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1db7f710-z7ksr\" (UID: \"2d2a3dba-562a-4ee3-b9a1-d6845a4d93b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1db7f710-z7ksr" Nov 03 02:29:43.995345 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:43.995294 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mtxp\" (UniqueName: \"kubernetes.io/projected/2d2a3dba-562a-4ee3-b9a1-d6845a4d93b3-kube-api-access-2mtxp\") pod \"maas-api-revoke-sub-1db7f710-z7ksr\" (UID: \"2d2a3dba-562a-4ee3-b9a1-d6845a4d93b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1db7f710-z7ksr" Nov 03 02:29:44.096486 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:44.096378 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d2a3dba-562a-4ee3-b9a1-d6845a4d93b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1db7f710-z7ksr\" (UID: \"2d2a3dba-562a-4ee3-b9a1-d6845a4d93b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1db7f710-z7ksr" Nov 03 02:29:44.096486 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:44.096437 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d2a3dba-562a-4ee3-b9a1-d6845a4d93b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1db7f710-z7ksr\" (UID: \"2d2a3dba-562a-4ee3-b9a1-d6845a4d93b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1db7f710-z7ksr" Nov 03 02:29:44.096789 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:44.096489 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mtxp\" (UniqueName: \"kubernetes.io/projected/2d2a3dba-562a-4ee3-b9a1-d6845a4d93b3-kube-api-access-2mtxp\") pod \"maas-api-revoke-sub-1db7f710-z7ksr\" (UID: \"2d2a3dba-562a-4ee3-b9a1-d6845a4d93b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1db7f710-z7ksr" Nov 03 02:29:44.098450 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:44.097646 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d2a3dba-562a-4ee3-b9a1-d6845a4d93b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1db7f710-z7ksr\" (UID: \"2d2a3dba-562a-4ee3-b9a1-d6845a4d93b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1db7f710-z7ksr" Nov 03 02:29:44.099465 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:44.099433 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d2a3dba-562a-4ee3-b9a1-d6845a4d93b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1db7f710-z7ksr\" (UID: \"2d2a3dba-562a-4ee3-b9a1-d6845a4d93b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1db7f710-z7ksr" Nov 03 02:29:44.149778 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:44.149750 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mtxp\" (UniqueName: \"kubernetes.io/projected/2d2a3dba-562a-4ee3-b9a1-d6845a4d93b3-kube-api-access-2mtxp\") pod \"maas-api-revoke-sub-1db7f710-z7ksr\" (UID: \"2d2a3dba-562a-4ee3-b9a1-d6845a4d93b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1db7f710-z7ksr" Nov 03 02:29:44.156370 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:44.156334 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1db7f710-z7ksr" Nov 03 02:29:44.526536 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:44.526453 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1db7f710-z7ksr"] Nov 03 02:29:44.533091 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:29:44.533063 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d2a3dba_562a_4ee3_b9a1_d6845a4d93b3.slice/crio-dcd3063a034a3c2ada04294f0ad4f52d3e276b72479ac6531f12133d079a6671 WatchSource:0}: Error finding container dcd3063a034a3c2ada04294f0ad4f52d3e276b72479ac6531f12133d079a6671: Status 404 returned error can't find the container with id dcd3063a034a3c2ada04294f0ad4f52d3e276b72479ac6531f12133d079a6671 Nov 03 02:29:44.767368 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:44.767325 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7a55cf2-pb6hs"] Nov 03 02:29:44.772445 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:44.772425 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a55cf2-pb6hs" Nov 03 02:29:44.824376 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:44.824336 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7a55cf2-pb6hs"] Nov 03 02:29:44.850823 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:44.850783 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1db7f710-z7ksr" event={"ID":"2d2a3dba-562a-4ee3-b9a1-d6845a4d93b3","Type":"ContainerStarted","Data":"dcd3063a034a3c2ada04294f0ad4f52d3e276b72479ac6531f12133d079a6671"} Nov 03 02:29:44.916930 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:44.916896 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2cd0a0a-e029-426f-b25a-8fdb92bd10ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7a55cf2-pb6hs\" (UID: \"c2cd0a0a-e029-426f-b25a-8fdb92bd10ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a55cf2-pb6hs" Nov 03 02:29:44.917102 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:44.917068 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzbcp\" (UniqueName: \"kubernetes.io/projected/c2cd0a0a-e029-426f-b25a-8fdb92bd10ea-kube-api-access-xzbcp\") pod \"maas-api-revoke-sub-a7a55cf2-pb6hs\" (UID: \"c2cd0a0a-e029-426f-b25a-8fdb92bd10ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a55cf2-pb6hs" Nov 03 02:29:44.917156 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:44.917113 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2cd0a0a-e029-426f-b25a-8fdb92bd10ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7a55cf2-pb6hs\" (UID: \"c2cd0a0a-e029-426f-b25a-8fdb92bd10ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a55cf2-pb6hs" Nov 03 02:29:45.018653 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:45.018617 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2cd0a0a-e029-426f-b25a-8fdb92bd10ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7a55cf2-pb6hs\" (UID: \"c2cd0a0a-e029-426f-b25a-8fdb92bd10ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a55cf2-pb6hs" Nov 03 02:29:45.018816 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:45.018681 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2cd0a0a-e029-426f-b25a-8fdb92bd10ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7a55cf2-pb6hs\" (UID: \"c2cd0a0a-e029-426f-b25a-8fdb92bd10ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a55cf2-pb6hs" Nov 03 02:29:45.018880 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:45.018824 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzbcp\" (UniqueName: \"kubernetes.io/projected/c2cd0a0a-e029-426f-b25a-8fdb92bd10ea-kube-api-access-xzbcp\") pod \"maas-api-revoke-sub-a7a55cf2-pb6hs\" (UID: \"c2cd0a0a-e029-426f-b25a-8fdb92bd10ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a55cf2-pb6hs" Nov 03 02:29:45.019405 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:45.019376 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2cd0a0a-e029-426f-b25a-8fdb92bd10ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7a55cf2-pb6hs\" (UID: \"c2cd0a0a-e029-426f-b25a-8fdb92bd10ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a55cf2-pb6hs" Nov 03 02:29:45.023442 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:45.023414 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2cd0a0a-e029-426f-b25a-8fdb92bd10ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7a55cf2-pb6hs\" (UID: \"c2cd0a0a-e029-426f-b25a-8fdb92bd10ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a55cf2-pb6hs" Nov 03 02:29:45.071742 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:45.071706 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzbcp\" (UniqueName: \"kubernetes.io/projected/c2cd0a0a-e029-426f-b25a-8fdb92bd10ea-kube-api-access-xzbcp\") pod \"maas-api-revoke-sub-a7a55cf2-pb6hs\" (UID: \"c2cd0a0a-e029-426f-b25a-8fdb92bd10ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a55cf2-pb6hs" Nov 03 02:29:45.090192 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:45.090155 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a55cf2-pb6hs" Nov 03 02:29:45.463307 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:45.463276 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7a55cf2-pb6hs"] Nov 03 02:29:45.517346 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:29:45.517306 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2cd0a0a_e029_426f_b25a_8fdb92bd10ea.slice/crio-33001ac7c1c5e1c5bc98a6cc335575f5e9ef3dc832bd221635301f8a5f9d0eea WatchSource:0}: Error finding container 33001ac7c1c5e1c5bc98a6cc335575f5e9ef3dc832bd221635301f8a5f9d0eea: Status 404 returned error can't find the container with id 33001ac7c1c5e1c5bc98a6cc335575f5e9ef3dc832bd221635301f8a5f9d0eea Nov 03 02:29:45.856303 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:45.856270 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a55cf2-pb6hs" event={"ID":"c2cd0a0a-e029-426f-b25a-8fdb92bd10ea","Type":"ContainerStarted","Data":"33001ac7c1c5e1c5bc98a6cc335575f5e9ef3dc832bd221635301f8a5f9d0eea"} Nov 03 02:29:45.857989 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:45.857964 5605 generic.go:356] "Generic (PLEG): container finished" podID="2d2a3dba-562a-4ee3-b9a1-d6845a4d93b3" containerID="38a5a3375eddf1d2d3b677b9b5e36be53bea8cc0fc7fd2498336f03c7658c04c" exitCode=0 Nov 03 02:29:45.858110 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:45.858009 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1db7f710-z7ksr" event={"ID":"2d2a3dba-562a-4ee3-b9a1-d6845a4d93b3","Type":"ContainerDied","Data":"38a5a3375eddf1d2d3b677b9b5e36be53bea8cc0fc7fd2498336f03c7658c04c"} Nov 03 02:29:46.864762 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:46.864728 5605 generic.go:356] "Generic (PLEG): container finished" podID="c2cd0a0a-e029-426f-b25a-8fdb92bd10ea" containerID="c53f58cbde646c8d93835f6ea2e451b0a6a32cf8bfd1d9c4e8cfc45b78f08bf2" exitCode=0 Nov 03 02:29:46.865183 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:46.864815 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a55cf2-pb6hs" event={"ID":"c2cd0a0a-e029-426f-b25a-8fdb92bd10ea","Type":"ContainerDied","Data":"c53f58cbde646c8d93835f6ea2e451b0a6a32cf8bfd1d9c4e8cfc45b78f08bf2"} Nov 03 02:29:47.011864 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:47.011837 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1db7f710-z7ksr" Nov 03 02:29:47.042219 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:47.042194 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d2a3dba-562a-4ee3-b9a1-d6845a4d93b3-kube-api-access-2mtxp\" (UniqueName: \"kubernetes.io/projected/2d2a3dba-562a-4ee3-b9a1-d6845a4d93b3-kube-api-access-2mtxp\") pod \"2d2a3dba-562a-4ee3-b9a1-d6845a4d93b3\" (UID: \"2d2a3dba-562a-4ee3-b9a1-d6845a4d93b3\") " Nov 03 02:29:47.042388 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:47.042343 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d2a3dba-562a-4ee3-b9a1-d6845a4d93b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d2a3dba-562a-4ee3-b9a1-d6845a4d93b3-maas-api-xks-ca\") pod \"2d2a3dba-562a-4ee3-b9a1-d6845a4d93b3\" (UID: \"2d2a3dba-562a-4ee3-b9a1-d6845a4d93b3\") " Nov 03 02:29:47.042497 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:47.042483 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d2a3dba-562a-4ee3-b9a1-d6845a4d93b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d2a3dba-562a-4ee3-b9a1-d6845a4d93b3-maas-api-openshift-ca\") pod \"2d2a3dba-562a-4ee3-b9a1-d6845a4d93b3\" (UID: \"2d2a3dba-562a-4ee3-b9a1-d6845a4d93b3\") " Nov 03 02:29:47.042800 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:47.042777 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d2a3dba-562a-4ee3-b9a1-d6845a4d93b3-maas-api-openshift-ca" pod "2d2a3dba-562a-4ee3-b9a1-d6845a4d93b3" (UID: "2d2a3dba-562a-4ee3-b9a1-d6845a4d93b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:29:47.044189 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:47.044169 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d2a3dba-562a-4ee3-b9a1-d6845a4d93b3-maas-api-xks-ca" pod "2d2a3dba-562a-4ee3-b9a1-d6845a4d93b3" (UID: "2d2a3dba-562a-4ee3-b9a1-d6845a4d93b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:29:47.044279 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:47.044263 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d2a3dba-562a-4ee3-b9a1-d6845a4d93b3-kube-api-access-2mtxp" pod "2d2a3dba-562a-4ee3-b9a1-d6845a4d93b3" (UID: "2d2a3dba-562a-4ee3-b9a1-d6845a4d93b3"). InnerVolumeSpecName "kube-api-access-2mtxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:29:47.144081 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:47.144005 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d2a3dba-562a-4ee3-b9a1-d6845a4d93b3-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:29:47.144081 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:47.144037 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mtxp\" (UniqueName: \"kubernetes.io/projected/2d2a3dba-562a-4ee3-b9a1-d6845a4d93b3-kube-api-access-2mtxp\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:29:47.144081 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:47.144047 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d2a3dba-562a-4ee3-b9a1-d6845a4d93b3-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:29:47.870719 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:47.870691 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1db7f710-z7ksr" Nov 03 02:29:47.871175 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:47.870729 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1db7f710-z7ksr" event={"ID":"2d2a3dba-562a-4ee3-b9a1-d6845a4d93b3","Type":"ContainerDied","Data":"dcd3063a034a3c2ada04294f0ad4f52d3e276b72479ac6531f12133d079a6671"} Nov 03 02:29:47.871175 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:47.870766 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dcd3063a034a3c2ada04294f0ad4f52d3e276b72479ac6531f12133d079a6671" Nov 03 02:29:48.013474 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:48.013446 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a55cf2-pb6hs" Nov 03 02:29:48.053076 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:48.053047 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2cd0a0a-e029-426f-b25a-8fdb92bd10ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2cd0a0a-e029-426f-b25a-8fdb92bd10ea-maas-api-xks-ca\") pod \"c2cd0a0a-e029-426f-b25a-8fdb92bd10ea\" (UID: \"c2cd0a0a-e029-426f-b25a-8fdb92bd10ea\") " Nov 03 02:29:48.053219 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:48.053135 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2cd0a0a-e029-426f-b25a-8fdb92bd10ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2cd0a0a-e029-426f-b25a-8fdb92bd10ea-maas-api-openshift-ca\") pod \"c2cd0a0a-e029-426f-b25a-8fdb92bd10ea\" (UID: \"c2cd0a0a-e029-426f-b25a-8fdb92bd10ea\") " Nov 03 02:29:48.053283 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:48.053222 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2cd0a0a-e029-426f-b25a-8fdb92bd10ea-kube-api-access-xzbcp\" (UniqueName: \"kubernetes.io/projected/c2cd0a0a-e029-426f-b25a-8fdb92bd10ea-kube-api-access-xzbcp\") pod \"c2cd0a0a-e029-426f-b25a-8fdb92bd10ea\" (UID: \"c2cd0a0a-e029-426f-b25a-8fdb92bd10ea\") " Nov 03 02:29:48.053566 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:48.053470 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2cd0a0a-e029-426f-b25a-8fdb92bd10ea-maas-api-openshift-ca" pod "c2cd0a0a-e029-426f-b25a-8fdb92bd10ea" (UID: "c2cd0a0a-e029-426f-b25a-8fdb92bd10ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:29:48.053748 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:48.053730 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2cd0a0a-e029-426f-b25a-8fdb92bd10ea-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:29:48.055183 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:48.055153 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2cd0a0a-e029-426f-b25a-8fdb92bd10ea-maas-api-xks-ca" pod "c2cd0a0a-e029-426f-b25a-8fdb92bd10ea" (UID: "c2cd0a0a-e029-426f-b25a-8fdb92bd10ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:29:48.055274 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:48.055234 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2cd0a0a-e029-426f-b25a-8fdb92bd10ea-kube-api-access-xzbcp" pod "c2cd0a0a-e029-426f-b25a-8fdb92bd10ea" (UID: "c2cd0a0a-e029-426f-b25a-8fdb92bd10ea"). InnerVolumeSpecName "kube-api-access-xzbcp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:29:48.155296 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:48.155221 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzbcp\" (UniqueName: \"kubernetes.io/projected/c2cd0a0a-e029-426f-b25a-8fdb92bd10ea-kube-api-access-xzbcp\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:29:48.155296 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:48.155250 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2cd0a0a-e029-426f-b25a-8fdb92bd10ea-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:29:48.877181 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:48.877145 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a55cf2-pb6hs" event={"ID":"c2cd0a0a-e029-426f-b25a-8fdb92bd10ea","Type":"ContainerDied","Data":"33001ac7c1c5e1c5bc98a6cc335575f5e9ef3dc832bd221635301f8a5f9d0eea"} Nov 03 02:29:48.877578 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:48.877201 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33001ac7c1c5e1c5bc98a6cc335575f5e9ef3dc832bd221635301f8a5f9d0eea" Nov 03 02:29:48.877578 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:29:48.877166 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a55cf2-pb6hs" Nov 03 02:29:53.915511 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:29:53.915477 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:30:00.155042 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.154997 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894550-dx9j4"] Nov 03 02:30:00.155697 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.155675 5605 cpu_manager.go:414] "containerMap: removing container" podUID="2d2a3dba-562a-4ee3-b9a1-d6845a4d93b3" containerName="revoke-keys" Nov 03 02:30:00.155781 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.155698 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d2a3dba-562a-4ee3-b9a1-d6845a4d93b3" containerName="revoke-keys" Nov 03 02:30:00.155781 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.155731 5605 cpu_manager.go:414] "containerMap: removing container" podUID="c2cd0a0a-e029-426f-b25a-8fdb92bd10ea" containerName="revoke-keys" Nov 03 02:30:00.155781 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.155739 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2cd0a0a-e029-426f-b25a-8fdb92bd10ea" containerName="revoke-keys" Nov 03 02:30:00.155938 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.155881 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2cd0a0a-e029-426f-b25a-8fdb92bd10ea" containerName="revoke-keys" Nov 03 02:30:00.155938 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.155903 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d2a3dba-562a-4ee3-b9a1-d6845a4d93b3" containerName="revoke-keys" Nov 03 02:30:00.159077 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.159051 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-f760d3-29894550-8kxmb"] Nov 03 02:30:00.159238 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.159197 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894550-dx9j4" Nov 03 02:30:00.162598 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.162579 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894550-n7jzq"] Nov 03 02:30:00.162718 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.162697 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-f760d3-29894550-8kxmb" Nov 03 02:30:00.165859 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.165839 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894550-n7jzq" Nov 03 02:30:00.201933 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.201906 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-496177-29894550-h27bc"] Nov 03 02:30:00.205351 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.205331 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894550-jhpt8"] Nov 03 02:30:00.205478 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.205464 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-496177-29894550-h27bc" Nov 03 02:30:00.212119 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.208822 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fb1940-29894550-jw8ct"] Nov 03 02:30:00.212119 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.209946 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894550-jhpt8" Nov 03 02:30:00.217219 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.216630 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894550-2pn5m"] Nov 03 02:30:00.217219 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.216976 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fb1940-29894550-jw8ct" Nov 03 02:30:00.221686 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.221668 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-711318-29894550-g884l"] Nov 03 02:30:00.221792 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.221764 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894550-2pn5m" Nov 03 02:30:00.225099 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.225081 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-711318-29894550-g884l" Nov 03 02:30:00.246941 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.246656 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:30:00.246941 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.246771 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:30:00.246941 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.246847 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:30:00.249770 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.249746 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894550-dx9j4"] Nov 03 02:30:00.258912 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.253629 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-f760d3-29894550-8kxmb"] Nov 03 02:30:00.258912 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.256418 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894550-n7jzq"] Nov 03 02:30:00.261042 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.261020 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-496177-29894550-h27bc"] Nov 03 02:30:00.312509 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.312426 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bb55cd1-e726-45c4-b1fd-b03229ec0378-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-f760d3-29894550-8kxmb\" (UID: \"3bb55cd1-e726-45c4-b1fd-b03229ec0378\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-f760d3-29894550-8kxmb" Nov 03 02:30:00.312509 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.312488 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eff793f-67d6-4044-87fb-ca5e46a5e6ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894550-dx9j4\" (UID: \"6eff793f-67d6-4044-87fb-ca5e46a5e6ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894550-dx9j4" Nov 03 02:30:00.312743 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.312556 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/433647e9-f97a-470d-bd48-cf2c6f9d6064-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-fb1940-29894550-jw8ct\" (UID: \"433647e9-f97a-470d-bd48-cf2c6f9d6064\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fb1940-29894550-jw8ct" Nov 03 02:30:00.312743 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.312589 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvh82\" (UniqueName: \"kubernetes.io/projected/a45e73e4-5176-408e-bd3c-b0412e7a3f88-kube-api-access-zvh82\") pod \"maas-api-key-cleanup-e2e-praxis-b-711318-29894550-g884l\" (UID: \"a45e73e4-5176-408e-bd3c-b0412e7a3f88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-711318-29894550-g884l" Nov 03 02:30:00.312743 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.312619 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfplg\" (UniqueName: \"kubernetes.io/projected/c5a086a9-767c-4878-9b57-de00c92df859-kube-api-access-jfplg\") pod \"maas-api-key-cleanup-e2e-worker-w2-521f1f-29894550-2pn5m\" (UID: \"c5a086a9-767c-4878-9b57-de00c92df859\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894550-2pn5m" Nov 03 02:30:00.312743 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.312643 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlsrs\" (UniqueName: \"kubernetes.io/projected/433647e9-f97a-470d-bd48-cf2c6f9d6064-kube-api-access-jlsrs\") pod \"maas-api-key-cleanup-e2e-ipp-a-fb1940-29894550-jw8ct\" (UID: \"433647e9-f97a-470d-bd48-cf2c6f9d6064\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fb1940-29894550-jw8ct" Nov 03 02:30:00.312743 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.312713 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/963f49af-b49b-4664-a9a5-57937449e0df-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894550-jhpt8\" (UID: \"963f49af-b49b-4664-a9a5-57937449e0df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894550-jhpt8" Nov 03 02:30:00.313020 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.312766 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9cd06a2-1beb-4d2b-9561-4e4a3b788e4d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-496177-29894550-h27bc\" (UID: \"a9cd06a2-1beb-4d2b-9561-4e4a3b788e4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-496177-29894550-h27bc" Nov 03 02:30:00.313020 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.312838 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25790141-f19b-4e73-8b61-4c82f33a1825-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c7d758-29894550-n7jzq\" (UID: \"25790141-f19b-4e73-8b61-4c82f33a1825\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894550-n7jzq" Nov 03 02:30:00.313020 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.312867 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x76nr\" (UniqueName: \"kubernetes.io/projected/a9cd06a2-1beb-4d2b-9561-4e4a3b788e4d-kube-api-access-x76nr\") pod \"maas-api-key-cleanup-e2e-praxis-b-496177-29894550-h27bc\" (UID: \"a9cd06a2-1beb-4d2b-9561-4e4a3b788e4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-496177-29894550-h27bc" Nov 03 02:30:00.313020 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.312902 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5a086a9-767c-4878-9b57-de00c92df859-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-521f1f-29894550-2pn5m\" (UID: \"c5a086a9-767c-4878-9b57-de00c92df859\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894550-2pn5m" Nov 03 02:30:00.313020 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.312937 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt6sq\" (UniqueName: \"kubernetes.io/projected/6eff793f-67d6-4044-87fb-ca5e46a5e6ac-kube-api-access-gt6sq\") pod \"maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894550-dx9j4\" (UID: \"6eff793f-67d6-4044-87fb-ca5e46a5e6ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894550-dx9j4" Nov 03 02:30:00.313020 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.312962 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdjgz\" (UniqueName: \"kubernetes.io/projected/25790141-f19b-4e73-8b61-4c82f33a1825-kube-api-access-gdjgz\") pod \"maas-api-key-cleanup-e2e-worker-w4-c7d758-29894550-n7jzq\" (UID: \"25790141-f19b-4e73-8b61-4c82f33a1825\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894550-n7jzq" Nov 03 02:30:00.313020 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.312993 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a45e73e4-5176-408e-bd3c-b0412e7a3f88-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-711318-29894550-g884l\" (UID: \"a45e73e4-5176-408e-bd3c-b0412e7a3f88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-711318-29894550-g884l" Nov 03 02:30:00.313465 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.313040 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dht8s\" (UniqueName: \"kubernetes.io/projected/963f49af-b49b-4664-a9a5-57937449e0df-kube-api-access-dht8s\") pod \"maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894550-jhpt8\" (UID: \"963f49af-b49b-4664-a9a5-57937449e0df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894550-jhpt8" Nov 03 02:30:00.313465 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.313087 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zddrf\" (UniqueName: \"kubernetes.io/projected/3bb55cd1-e726-45c4-b1fd-b03229ec0378-kube-api-access-zddrf\") pod \"maas-api-key-cleanup-e2e-praxis-a-f760d3-29894550-8kxmb\" (UID: \"3bb55cd1-e726-45c4-b1fd-b03229ec0378\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-f760d3-29894550-8kxmb" Nov 03 02:30:00.315870 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.315843 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-711318-29894550-g884l"] Nov 03 02:30:00.323801 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.323770 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894550-2pn5m"] Nov 03 02:30:00.323960 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.323843 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894550-jhpt8"] Nov 03 02:30:00.326865 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.326843 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fb1940-29894550-jw8ct"] Nov 03 02:30:00.414601 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.414520 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9cd06a2-1beb-4d2b-9561-4e4a3b788e4d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-496177-29894550-h27bc\" (UID: \"a9cd06a2-1beb-4d2b-9561-4e4a3b788e4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-496177-29894550-h27bc" Nov 03 02:30:00.414601 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.414574 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25790141-f19b-4e73-8b61-4c82f33a1825-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c7d758-29894550-n7jzq\" (UID: \"25790141-f19b-4e73-8b61-4c82f33a1825\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894550-n7jzq" Nov 03 02:30:00.414601 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.414593 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x76nr\" (UniqueName: \"kubernetes.io/projected/a9cd06a2-1beb-4d2b-9561-4e4a3b788e4d-kube-api-access-x76nr\") pod \"maas-api-key-cleanup-e2e-praxis-b-496177-29894550-h27bc\" (UID: \"a9cd06a2-1beb-4d2b-9561-4e4a3b788e4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-496177-29894550-h27bc" Nov 03 02:30:00.414890 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.414614 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5a086a9-767c-4878-9b57-de00c92df859-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-521f1f-29894550-2pn5m\" (UID: \"c5a086a9-767c-4878-9b57-de00c92df859\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894550-2pn5m" Nov 03 02:30:00.414890 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.414636 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt6sq\" (UniqueName: \"kubernetes.io/projected/6eff793f-67d6-4044-87fb-ca5e46a5e6ac-kube-api-access-gt6sq\") pod \"maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894550-dx9j4\" (UID: \"6eff793f-67d6-4044-87fb-ca5e46a5e6ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894550-dx9j4" Nov 03 02:30:00.414890 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.414652 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdjgz\" (UniqueName: \"kubernetes.io/projected/25790141-f19b-4e73-8b61-4c82f33a1825-kube-api-access-gdjgz\") pod \"maas-api-key-cleanup-e2e-worker-w4-c7d758-29894550-n7jzq\" (UID: \"25790141-f19b-4e73-8b61-4c82f33a1825\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894550-n7jzq" Nov 03 02:30:00.414890 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.414670 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a45e73e4-5176-408e-bd3c-b0412e7a3f88-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-711318-29894550-g884l\" (UID: \"a45e73e4-5176-408e-bd3c-b0412e7a3f88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-711318-29894550-g884l" Nov 03 02:30:00.414890 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.414701 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dht8s\" (UniqueName: \"kubernetes.io/projected/963f49af-b49b-4664-a9a5-57937449e0df-kube-api-access-dht8s\") pod \"maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894550-jhpt8\" (UID: \"963f49af-b49b-4664-a9a5-57937449e0df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894550-jhpt8" Nov 03 02:30:00.414890 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.414739 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zddrf\" (UniqueName: \"kubernetes.io/projected/3bb55cd1-e726-45c4-b1fd-b03229ec0378-kube-api-access-zddrf\") pod \"maas-api-key-cleanup-e2e-praxis-a-f760d3-29894550-8kxmb\" (UID: \"3bb55cd1-e726-45c4-b1fd-b03229ec0378\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-f760d3-29894550-8kxmb" Nov 03 02:30:00.414890 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.414772 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bb55cd1-e726-45c4-b1fd-b03229ec0378-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-f760d3-29894550-8kxmb\" (UID: \"3bb55cd1-e726-45c4-b1fd-b03229ec0378\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-f760d3-29894550-8kxmb" Nov 03 02:30:00.414890 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.414814 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eff793f-67d6-4044-87fb-ca5e46a5e6ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894550-dx9j4\" (UID: \"6eff793f-67d6-4044-87fb-ca5e46a5e6ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894550-dx9j4" Nov 03 02:30:00.414890 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.414882 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/433647e9-f97a-470d-bd48-cf2c6f9d6064-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-fb1940-29894550-jw8ct\" (UID: \"433647e9-f97a-470d-bd48-cf2c6f9d6064\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fb1940-29894550-jw8ct" Nov 03 02:30:00.415412 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.414918 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvh82\" (UniqueName: \"kubernetes.io/projected/a45e73e4-5176-408e-bd3c-b0412e7a3f88-kube-api-access-zvh82\") pod \"maas-api-key-cleanup-e2e-praxis-b-711318-29894550-g884l\" (UID: \"a45e73e4-5176-408e-bd3c-b0412e7a3f88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-711318-29894550-g884l" Nov 03 02:30:00.415412 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.415039 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfplg\" (UniqueName: \"kubernetes.io/projected/c5a086a9-767c-4878-9b57-de00c92df859-kube-api-access-jfplg\") pod \"maas-api-key-cleanup-e2e-worker-w2-521f1f-29894550-2pn5m\" (UID: \"c5a086a9-767c-4878-9b57-de00c92df859\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894550-2pn5m" Nov 03 02:30:00.415412 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.415085 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlsrs\" (UniqueName: \"kubernetes.io/projected/433647e9-f97a-470d-bd48-cf2c6f9d6064-kube-api-access-jlsrs\") pod \"maas-api-key-cleanup-e2e-ipp-a-fb1940-29894550-jw8ct\" (UID: \"433647e9-f97a-470d-bd48-cf2c6f9d6064\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fb1940-29894550-jw8ct" Nov 03 02:30:00.415412 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.415173 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/963f49af-b49b-4664-a9a5-57937449e0df-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894550-jhpt8\" (UID: \"963f49af-b49b-4664-a9a5-57937449e0df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894550-jhpt8" Nov 03 02:30:00.415412 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.415349 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9cd06a2-1beb-4d2b-9561-4e4a3b788e4d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-496177-29894550-h27bc\" (UID: \"a9cd06a2-1beb-4d2b-9561-4e4a3b788e4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-496177-29894550-h27bc" Nov 03 02:30:00.415669 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.415503 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5a086a9-767c-4878-9b57-de00c92df859-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-521f1f-29894550-2pn5m\" (UID: \"c5a086a9-767c-4878-9b57-de00c92df859\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894550-2pn5m" Nov 03 02:30:00.415669 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.415527 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25790141-f19b-4e73-8b61-4c82f33a1825-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c7d758-29894550-n7jzq\" (UID: \"25790141-f19b-4e73-8b61-4c82f33a1825\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894550-n7jzq" Nov 03 02:30:00.415669 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.415552 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a45e73e4-5176-408e-bd3c-b0412e7a3f88-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-711318-29894550-g884l\" (UID: \"a45e73e4-5176-408e-bd3c-b0412e7a3f88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-711318-29894550-g884l" Nov 03 02:30:00.415931 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.415913 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/963f49af-b49b-4664-a9a5-57937449e0df-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894550-jhpt8\" (UID: \"963f49af-b49b-4664-a9a5-57937449e0df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894550-jhpt8" Nov 03 02:30:00.415969 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.415926 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eff793f-67d6-4044-87fb-ca5e46a5e6ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894550-dx9j4\" (UID: \"6eff793f-67d6-4044-87fb-ca5e46a5e6ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894550-dx9j4" Nov 03 02:30:00.415969 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.415928 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/433647e9-f97a-470d-bd48-cf2c6f9d6064-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-fb1940-29894550-jw8ct\" (UID: \"433647e9-f97a-470d-bd48-cf2c6f9d6064\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fb1940-29894550-jw8ct" Nov 03 02:30:00.416163 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.416147 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bb55cd1-e726-45c4-b1fd-b03229ec0378-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-f760d3-29894550-8kxmb\" (UID: \"3bb55cd1-e726-45c4-b1fd-b03229ec0378\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-f760d3-29894550-8kxmb" Nov 03 02:30:00.469008 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.468981 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dht8s\" (UniqueName: \"kubernetes.io/projected/963f49af-b49b-4664-a9a5-57937449e0df-kube-api-access-dht8s\") pod \"maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894550-jhpt8\" (UID: \"963f49af-b49b-4664-a9a5-57937449e0df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894550-jhpt8" Nov 03 02:30:00.469172 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.469156 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvh82\" (UniqueName: \"kubernetes.io/projected/a45e73e4-5176-408e-bd3c-b0412e7a3f88-kube-api-access-zvh82\") pod \"maas-api-key-cleanup-e2e-praxis-b-711318-29894550-g884l\" (UID: \"a45e73e4-5176-408e-bd3c-b0412e7a3f88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-711318-29894550-g884l" Nov 03 02:30:00.469646 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.469611 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x76nr\" (UniqueName: \"kubernetes.io/projected/a9cd06a2-1beb-4d2b-9561-4e4a3b788e4d-kube-api-access-x76nr\") pod \"maas-api-key-cleanup-e2e-praxis-b-496177-29894550-h27bc\" (UID: \"a9cd06a2-1beb-4d2b-9561-4e4a3b788e4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-496177-29894550-h27bc" Nov 03 02:30:00.469761 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.469675 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdjgz\" (UniqueName: \"kubernetes.io/projected/25790141-f19b-4e73-8b61-4c82f33a1825-kube-api-access-gdjgz\") pod \"maas-api-key-cleanup-e2e-worker-w4-c7d758-29894550-n7jzq\" (UID: \"25790141-f19b-4e73-8b61-4c82f33a1825\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894550-n7jzq" Nov 03 02:30:00.469761 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.469684 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfplg\" (UniqueName: \"kubernetes.io/projected/c5a086a9-767c-4878-9b57-de00c92df859-kube-api-access-jfplg\") pod \"maas-api-key-cleanup-e2e-worker-w2-521f1f-29894550-2pn5m\" (UID: \"c5a086a9-767c-4878-9b57-de00c92df859\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894550-2pn5m" Nov 03 02:30:00.469761 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.469692 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlsrs\" (UniqueName: \"kubernetes.io/projected/433647e9-f97a-470d-bd48-cf2c6f9d6064-kube-api-access-jlsrs\") pod \"maas-api-key-cleanup-e2e-ipp-a-fb1940-29894550-jw8ct\" (UID: \"433647e9-f97a-470d-bd48-cf2c6f9d6064\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fb1940-29894550-jw8ct" Nov 03 02:30:00.469761 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.469692 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zddrf\" (UniqueName: \"kubernetes.io/projected/3bb55cd1-e726-45c4-b1fd-b03229ec0378-kube-api-access-zddrf\") pod \"maas-api-key-cleanup-e2e-praxis-a-f760d3-29894550-8kxmb\" (UID: \"3bb55cd1-e726-45c4-b1fd-b03229ec0378\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-f760d3-29894550-8kxmb" Nov 03 02:30:00.469958 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.469938 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt6sq\" (UniqueName: \"kubernetes.io/projected/6eff793f-67d6-4044-87fb-ca5e46a5e6ac-kube-api-access-gt6sq\") pod \"maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894550-dx9j4\" (UID: \"6eff793f-67d6-4044-87fb-ca5e46a5e6ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894550-dx9j4" Nov 03 02:30:00.473505 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.473486 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894550-dx9j4" Nov 03 02:30:00.480900 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.480880 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-f760d3-29894550-8kxmb" Nov 03 02:30:00.487150 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.487121 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894550-n7jzq" Nov 03 02:30:00.526160 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.525199 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-496177-29894550-h27bc" Nov 03 02:30:00.531896 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.531862 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894550-jhpt8" Nov 03 02:30:00.539707 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.539682 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fb1940-29894550-jw8ct" Nov 03 02:30:00.549539 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.549228 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894550-2pn5m" Nov 03 02:30:00.561268 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.561237 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-711318-29894550-g884l" Nov 03 02:30:00.855313 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.855277 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894550-dx9j4"] Nov 03 02:30:00.860118 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:30:00.860062 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eff793f_67d6_4044_87fb_ca5e46a5e6ac.slice/crio-19a94d75cd11778b8a88246a9b62916aae28862feb7237583020915a5e9c3d0e WatchSource:0}: Error finding container 19a94d75cd11778b8a88246a9b62916aae28862feb7237583020915a5e9c3d0e: Status 404 returned error can't find the container with id 19a94d75cd11778b8a88246a9b62916aae28862feb7237583020915a5e9c3d0e Nov 03 02:30:00.957142 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:00.957051 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894550-dx9j4" event={"ID":"6eff793f-67d6-4044-87fb-ca5e46a5e6ac","Type":"ContainerStarted","Data":"19a94d75cd11778b8a88246a9b62916aae28862feb7237583020915a5e9c3d0e"} Nov 03 02:30:01.002500 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:01.002462 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-711318-29894550-g884l"] Nov 03 02:30:01.006920 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:30:01.006894 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda45e73e4_5176_408e_bd3c_b0412e7a3f88.slice/crio-5e51cc21abe0c3b0d8f876270b78fade9239d523494dad02583c782a14bec5cc WatchSource:0}: Error finding container 5e51cc21abe0c3b0d8f876270b78fade9239d523494dad02583c782a14bec5cc: Status 404 returned error can't find the container with id 5e51cc21abe0c3b0d8f876270b78fade9239d523494dad02583c782a14bec5cc Nov 03 02:30:01.020010 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:01.019979 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894550-2pn5m"] Nov 03 02:30:01.025051 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:30:01.025026 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5a086a9_767c_4878_9b57_de00c92df859.slice/crio-560935f6678ea9c285678c271f9f06339cb4a4aca10be2d1723bb7d1c2d3c93d WatchSource:0}: Error finding container 560935f6678ea9c285678c271f9f06339cb4a4aca10be2d1723bb7d1c2d3c93d: Status 404 returned error can't find the container with id 560935f6678ea9c285678c271f9f06339cb4a4aca10be2d1723bb7d1c2d3c93d Nov 03 02:30:01.052232 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:01.052202 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fb1940-29894550-jw8ct"] Nov 03 02:30:01.057197 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:30:01.057170 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod433647e9_f97a_470d_bd48_cf2c6f9d6064.slice/crio-904850f3da329af43476624771b5339f40e513f2e7801ab0f6e9ef9d75588336 WatchSource:0}: Error finding container 904850f3da329af43476624771b5339f40e513f2e7801ab0f6e9ef9d75588336: Status 404 returned error can't find the container with id 904850f3da329af43476624771b5339f40e513f2e7801ab0f6e9ef9d75588336 Nov 03 02:30:01.062682 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:01.062658 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-f760d3-29894550-8kxmb"] Nov 03 02:30:01.068751 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:30:01.068724 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bb55cd1_e726_45c4_b1fd_b03229ec0378.slice/crio-f01273b124bff2ba60377051a629cf82bab38e4d516f864f6ed105acefd4bd8f WatchSource:0}: Error finding container f01273b124bff2ba60377051a629cf82bab38e4d516f864f6ed105acefd4bd8f: Status 404 returned error can't find the container with id f01273b124bff2ba60377051a629cf82bab38e4d516f864f6ed105acefd4bd8f Nov 03 02:30:01.074125 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:01.074106 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894550-n7jzq"] Nov 03 02:30:01.152942 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:01.152907 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894550-jhpt8"] Nov 03 02:30:01.159615 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:30:01.159587 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod963f49af_b49b_4664_a9a5_57937449e0df.slice/crio-48af6c53151cde1cd57e98130351f28d9caf3497e4ed57796d2d758e7263ecd2 WatchSource:0}: Error finding container 48af6c53151cde1cd57e98130351f28d9caf3497e4ed57796d2d758e7263ecd2: Status 404 returned error can't find the container with id 48af6c53151cde1cd57e98130351f28d9caf3497e4ed57796d2d758e7263ecd2 Nov 03 02:30:01.162583 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:01.162557 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-496177-29894550-h27bc"] Nov 03 02:30:01.259508 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:30:01.259428 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9cd06a2_1beb_4d2b_9561_4e4a3b788e4d.slice/crio-29dc5f2d4d42465b626bc03ab1d31e33ca81d43afe9f2c187b0424cf38bcc0b1 WatchSource:0}: Error finding container 29dc5f2d4d42465b626bc03ab1d31e33ca81d43afe9f2c187b0424cf38bcc0b1: Status 404 returned error can't find the container with id 29dc5f2d4d42465b626bc03ab1d31e33ca81d43afe9f2c187b0424cf38bcc0b1 Nov 03 02:30:01.970385 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:01.968879 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fb1940-29894550-jw8ct" event={"ID":"433647e9-f97a-470d-bd48-cf2c6f9d6064","Type":"ContainerStarted","Data":"fef6158bf6966da0f422d935dde07ce9051c4799433367471811f84ace6b868d"} Nov 03 02:30:01.970385 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:01.968981 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fb1940-29894550-jw8ct" event={"ID":"433647e9-f97a-470d-bd48-cf2c6f9d6064","Type":"ContainerStarted","Data":"904850f3da329af43476624771b5339f40e513f2e7801ab0f6e9ef9d75588336"} Nov 03 02:30:01.977346 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:01.977305 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894550-2pn5m" event={"ID":"c5a086a9-767c-4878-9b57-de00c92df859","Type":"ContainerStarted","Data":"789a4c1aec4f05397e5868742c838b6b05cf4d2af00f3554b04ca8826ffac3c9"} Nov 03 02:30:01.977507 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:01.977405 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894550-2pn5m" event={"ID":"c5a086a9-767c-4878-9b57-de00c92df859","Type":"ContainerStarted","Data":"560935f6678ea9c285678c271f9f06339cb4a4aca10be2d1723bb7d1c2d3c93d"} Nov 03 02:30:01.979238 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:01.979208 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-496177-29894550-h27bc" event={"ID":"a9cd06a2-1beb-4d2b-9561-4e4a3b788e4d","Type":"ContainerStarted","Data":"d3029199015ac21f0e312f0ca2ce282f349ef028da1bbce55e45c19fd18784a0"} Nov 03 02:30:01.979370 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:01.979261 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-496177-29894550-h27bc" event={"ID":"a9cd06a2-1beb-4d2b-9561-4e4a3b788e4d","Type":"ContainerStarted","Data":"29dc5f2d4d42465b626bc03ab1d31e33ca81d43afe9f2c187b0424cf38bcc0b1"} Nov 03 02:30:01.980681 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:01.980658 5605 generic.go:356] "Generic (PLEG): container finished" podID="a45e73e4-5176-408e-bd3c-b0412e7a3f88" containerID="5eaea23b77773db12f3f9428a90f2451cc175b4ebcbca65ad6368f73373dac11" exitCode=7 Nov 03 02:30:01.980784 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:01.980682 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-711318-29894550-g884l" event={"ID":"a45e73e4-5176-408e-bd3c-b0412e7a3f88","Type":"ContainerDied","Data":"5eaea23b77773db12f3f9428a90f2451cc175b4ebcbca65ad6368f73373dac11"} Nov 03 02:30:01.980784 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:01.980727 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-711318-29894550-g884l" event={"ID":"a45e73e4-5176-408e-bd3c-b0412e7a3f88","Type":"ContainerStarted","Data":"5e51cc21abe0c3b0d8f876270b78fade9239d523494dad02583c782a14bec5cc"} Nov 03 02:30:01.980893 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:01.980879 5605 scope.go:122] "RemoveContainer" containerID="5eaea23b77773db12f3f9428a90f2451cc175b4ebcbca65ad6368f73373dac11" Nov 03 02:30:01.982526 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:01.982379 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894550-dx9j4" event={"ID":"6eff793f-67d6-4044-87fb-ca5e46a5e6ac","Type":"ContainerStarted","Data":"9a229f69d6091770894fbdae9869260d0841396229f98073ae825366b4201c93"} Nov 03 02:30:01.982600 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:01.982577 5605 scope.go:122] "RemoveContainer" containerID="9a229f69d6091770894fbdae9869260d0841396229f98073ae825366b4201c93" Nov 03 02:30:01.984428 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:01.984407 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894550-jhpt8" event={"ID":"963f49af-b49b-4664-a9a5-57937449e0df","Type":"ContainerStarted","Data":"669423c7fe322e69e521dce14fde51e6dccd8f35f039f3528fe5301834f9f4b7"} Nov 03 02:30:01.984495 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:01.984449 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894550-jhpt8" event={"ID":"963f49af-b49b-4664-a9a5-57937449e0df","Type":"ContainerStarted","Data":"48af6c53151cde1cd57e98130351f28d9caf3497e4ed57796d2d758e7263ecd2"} Nov 03 02:30:01.986421 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:01.986399 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894550-n7jzq" event={"ID":"25790141-f19b-4e73-8b61-4c82f33a1825","Type":"ContainerStarted","Data":"850da39706cfe082e1f210556e6d4d7665476f8c93d91ed96c9d056507e9595c"} Nov 03 02:30:01.986529 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:01.986444 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894550-n7jzq" event={"ID":"25790141-f19b-4e73-8b61-4c82f33a1825","Type":"ContainerStarted","Data":"3eb248a18a54d37acf4a3c1bb883af54494a1dcb7f8137e0048fecc89e2f2e02"} Nov 03 02:30:01.987932 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:01.987912 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-f760d3-29894550-8kxmb" event={"ID":"3bb55cd1-e726-45c4-b1fd-b03229ec0378","Type":"ContainerStarted","Data":"fbe1e7dbb3e3b0302fb15c88a83901b8b20f5016163471b3fa15b5d7aa8f1c73"} Nov 03 02:30:01.988007 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:01.987950 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-f760d3-29894550-8kxmb" event={"ID":"3bb55cd1-e726-45c4-b1fd-b03229ec0378","Type":"ContainerStarted","Data":"f01273b124bff2ba60377051a629cf82bab38e4d516f864f6ed105acefd4bd8f"} Nov 03 02:30:02.077026 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:02.076976 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fb1940-29894550-jw8ct" podStartSLOduration=2.076960853 podStartE2EDuration="2.076960853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:30:02.074984181 +0000 UTC m=+2279.334182338" watchObservedRunningTime="2026-11-03 02:30:02.076960853 +0000 UTC m=+2279.336158998" Nov 03 02:30:02.181064 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:02.181014 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894550-n7jzq" podStartSLOduration=2.18100008 podStartE2EDuration="2.18100008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:30:02.179962244 +0000 UTC m=+2279.439160396" watchObservedRunningTime="2026-11-03 02:30:02.18100008 +0000 UTC m=+2279.440198233" Nov 03 02:30:02.285198 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:02.285145 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894550-2pn5m" podStartSLOduration=2.285130856 podStartE2EDuration="2.285130856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:30:02.284046107 +0000 UTC m=+2279.543244262" watchObservedRunningTime="2026-11-03 02:30:02.285130856 +0000 UTC m=+2279.544329010" Nov 03 02:30:02.496467 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:02.496406 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-496177-29894550-h27bc" podStartSLOduration=2.496390925 podStartE2EDuration="2.496390925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:30:02.494719651 +0000 UTC m=+2279.753917806" watchObservedRunningTime="2026-11-03 02:30:02.496390925 +0000 UTC m=+2279.755589073" Nov 03 02:30:02.706061 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:02.705956 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894550-jhpt8" podStartSLOduration=2.705939945 podStartE2EDuration="2.705939945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:30:02.704204294 +0000 UTC m=+2279.963402442" watchObservedRunningTime="2026-11-03 02:30:02.705939945 +0000 UTC m=+2279.965138099" Nov 03 02:30:02.812733 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:02.812681 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-f760d3-29894550-8kxmb" podStartSLOduration=2.812667991 podStartE2EDuration="2.812667991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:30:02.810516452 +0000 UTC m=+2280.069714607" watchObservedRunningTime="2026-11-03 02:30:02.812667991 +0000 UTC m=+2280.071866146" Nov 03 02:30:02.996566 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:02.996525 5605 generic.go:356] "Generic (PLEG): container finished" podID="25790141-f19b-4e73-8b61-4c82f33a1825" containerID="850da39706cfe082e1f210556e6d4d7665476f8c93d91ed96c9d056507e9595c" exitCode=60 Nov 03 02:30:02.996728 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:02.996587 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894550-n7jzq" event={"ID":"25790141-f19b-4e73-8b61-4c82f33a1825","Type":"ContainerDied","Data":"850da39706cfe082e1f210556e6d4d7665476f8c93d91ed96c9d056507e9595c"} Nov 03 02:30:02.996854 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:02.996838 5605 scope.go:122] "RemoveContainer" containerID="850da39706cfe082e1f210556e6d4d7665476f8c93d91ed96c9d056507e9595c" Nov 03 02:30:02.998239 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:02.998215 5605 generic.go:356] "Generic (PLEG): container finished" podID="3bb55cd1-e726-45c4-b1fd-b03229ec0378" containerID="fbe1e7dbb3e3b0302fb15c88a83901b8b20f5016163471b3fa15b5d7aa8f1c73" exitCode=60 Nov 03 02:30:02.998376 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:02.998292 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-f760d3-29894550-8kxmb" event={"ID":"3bb55cd1-e726-45c4-b1fd-b03229ec0378","Type":"ContainerDied","Data":"fbe1e7dbb3e3b0302fb15c88a83901b8b20f5016163471b3fa15b5d7aa8f1c73"} Nov 03 02:30:02.998544 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:02.998525 5605 scope.go:122] "RemoveContainer" containerID="fbe1e7dbb3e3b0302fb15c88a83901b8b20f5016163471b3fa15b5d7aa8f1c73" Nov 03 02:30:03.000036 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:02.999882 5605 generic.go:356] "Generic (PLEG): container finished" podID="433647e9-f97a-470d-bd48-cf2c6f9d6064" containerID="fef6158bf6966da0f422d935dde07ce9051c4799433367471811f84ace6b868d" exitCode=60 Nov 03 02:30:03.000036 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:02.999948 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fb1940-29894550-jw8ct" event={"ID":"433647e9-f97a-470d-bd48-cf2c6f9d6064","Type":"ContainerDied","Data":"fef6158bf6966da0f422d935dde07ce9051c4799433367471811f84ace6b868d"} Nov 03 02:30:03.000257 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:03.000166 5605 scope.go:122] "RemoveContainer" containerID="fef6158bf6966da0f422d935dde07ce9051c4799433367471811f84ace6b868d" Nov 03 02:30:03.001719 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:03.001702 5605 generic.go:356] "Generic (PLEG): container finished" podID="c5a086a9-767c-4878-9b57-de00c92df859" containerID="789a4c1aec4f05397e5868742c838b6b05cf4d2af00f3554b04ca8826ffac3c9" exitCode=60 Nov 03 02:30:03.001806 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:03.001763 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894550-2pn5m" event={"ID":"c5a086a9-767c-4878-9b57-de00c92df859","Type":"ContainerDied","Data":"789a4c1aec4f05397e5868742c838b6b05cf4d2af00f3554b04ca8826ffac3c9"} Nov 03 02:30:03.001998 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:03.001981 5605 scope.go:122] "RemoveContainer" containerID="789a4c1aec4f05397e5868742c838b6b05cf4d2af00f3554b04ca8826ffac3c9" Nov 03 02:30:03.003384 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:03.003345 5605 generic.go:356] "Generic (PLEG): container finished" podID="a9cd06a2-1beb-4d2b-9561-4e4a3b788e4d" containerID="d3029199015ac21f0e312f0ca2ce282f349ef028da1bbce55e45c19fd18784a0" exitCode=60 Nov 03 02:30:03.003460 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:03.003386 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-496177-29894550-h27bc" event={"ID":"a9cd06a2-1beb-4d2b-9561-4e4a3b788e4d","Type":"ContainerDied","Data":"d3029199015ac21f0e312f0ca2ce282f349ef028da1bbce55e45c19fd18784a0"} Nov 03 02:30:03.003679 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:03.003663 5605 scope.go:122] "RemoveContainer" containerID="d3029199015ac21f0e312f0ca2ce282f349ef028da1bbce55e45c19fd18784a0" Nov 03 02:30:03.005177 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:03.005139 5605 generic.go:356] "Generic (PLEG): container finished" podID="a45e73e4-5176-408e-bd3c-b0412e7a3f88" containerID="43f38364147173104c1da0668c520b25822555629a75b6339efecc4f7e52c67c" exitCode=7 Nov 03 02:30:03.005177 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:03.005163 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-711318-29894550-g884l" event={"ID":"a45e73e4-5176-408e-bd3c-b0412e7a3f88","Type":"ContainerDied","Data":"43f38364147173104c1da0668c520b25822555629a75b6339efecc4f7e52c67c"} Nov 03 02:30:03.005280 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:03.005209 5605 scope.go:122] "RemoveContainer" containerID="5eaea23b77773db12f3f9428a90f2451cc175b4ebcbca65ad6368f73373dac11" Nov 03 02:30:03.005480 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:03.005462 5605 scope.go:122] "RemoveContainer" containerID="43f38364147173104c1da0668c520b25822555629a75b6339efecc4f7e52c67c" Nov 03 02:30:03.005647 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:30:03.005628 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-711318-29894550-g884l_odh-ai-gateway-infra(a45e73e4-5176-408e-bd3c-b0412e7a3f88)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-711318-29894550-g884l" podUID="a45e73e4-5176-408e-bd3c-b0412e7a3f88" Nov 03 02:30:03.007537 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:03.007425 5605 generic.go:356] "Generic (PLEG): container finished" podID="6eff793f-67d6-4044-87fb-ca5e46a5e6ac" containerID="9a229f69d6091770894fbdae9869260d0841396229f98073ae825366b4201c93" exitCode=60 Nov 03 02:30:03.007537 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:03.007445 5605 generic.go:356] "Generic (PLEG): container finished" podID="6eff793f-67d6-4044-87fb-ca5e46a5e6ac" containerID="001026d31fa4eeb1d72d8e82ab4afef6c685f24c83f32181aa4d96ad2e83de16" exitCode=60 Nov 03 02:30:03.007537 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:03.007518 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894550-dx9j4" event={"ID":"6eff793f-67d6-4044-87fb-ca5e46a5e6ac","Type":"ContainerDied","Data":"9a229f69d6091770894fbdae9869260d0841396229f98073ae825366b4201c93"} Nov 03 02:30:03.007693 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:03.007560 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894550-dx9j4" event={"ID":"6eff793f-67d6-4044-87fb-ca5e46a5e6ac","Type":"ContainerDied","Data":"001026d31fa4eeb1d72d8e82ab4afef6c685f24c83f32181aa4d96ad2e83de16"} Nov 03 02:30:03.007693 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:03.007690 5605 scope.go:122] "RemoveContainer" containerID="001026d31fa4eeb1d72d8e82ab4afef6c685f24c83f32181aa4d96ad2e83de16" Nov 03 02:30:03.007868 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:30:03.007851 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894550-dx9j4_odh-ai-gateway-infra(6eff793f-67d6-4044-87fb-ca5e46a5e6ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894550-dx9j4" podUID="6eff793f-67d6-4044-87fb-ca5e46a5e6ac" Nov 03 02:30:03.009166 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:03.009148 5605 generic.go:356] "Generic (PLEG): container finished" podID="963f49af-b49b-4664-a9a5-57937449e0df" containerID="669423c7fe322e69e521dce14fde51e6dccd8f35f039f3528fe5301834f9f4b7" exitCode=60 Nov 03 02:30:03.009249 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:03.009174 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894550-jhpt8" event={"ID":"963f49af-b49b-4664-a9a5-57937449e0df","Type":"ContainerDied","Data":"669423c7fe322e69e521dce14fde51e6dccd8f35f039f3528fe5301834f9f4b7"} Nov 03 02:30:03.009411 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:03.009352 5605 scope.go:122] "RemoveContainer" containerID="669423c7fe322e69e521dce14fde51e6dccd8f35f039f3528fe5301834f9f4b7" Nov 03 02:30:03.086998 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:03.086979 5605 scope.go:122] "RemoveContainer" containerID="9a229f69d6091770894fbdae9869260d0841396229f98073ae825366b4201c93" Nov 03 02:30:03.102687 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:03.102657 5605 scope.go:122] "RemoveContainer" containerID="9a229f69d6091770894fbdae9869260d0841396229f98073ae825366b4201c93" Nov 03 02:30:03.103007 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:30:03.102980 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a229f69d6091770894fbdae9869260d0841396229f98073ae825366b4201c93\": container with ID starting with 9a229f69d6091770894fbdae9869260d0841396229f98073ae825366b4201c93 not found: ID does not exist" containerID="9a229f69d6091770894fbdae9869260d0841396229f98073ae825366b4201c93" Nov 03 02:30:03.103106 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:03.103016 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a229f69d6091770894fbdae9869260d0841396229f98073ae825366b4201c93"} err="failed to get container status \"9a229f69d6091770894fbdae9869260d0841396229f98073ae825366b4201c93\": rpc error: code = NotFound desc = could not find container \"9a229f69d6091770894fbdae9869260d0841396229f98073ae825366b4201c93\": container with ID starting with 9a229f69d6091770894fbdae9869260d0841396229f98073ae825366b4201c93 not found: ID does not exist" Nov 03 02:30:04.016511 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:04.016478 5605 generic.go:356] "Generic (PLEG): container finished" podID="433647e9-f97a-470d-bd48-cf2c6f9d6064" containerID="ab505a1a2652408280ec1314153348ac4217217780e6ccc3b1571f4f63cdfa30" exitCode=60 Nov 03 02:30:04.016996 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:04.016548 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fb1940-29894550-jw8ct" event={"ID":"433647e9-f97a-470d-bd48-cf2c6f9d6064","Type":"ContainerDied","Data":"ab505a1a2652408280ec1314153348ac4217217780e6ccc3b1571f4f63cdfa30"} Nov 03 02:30:04.016996 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:04.016599 5605 scope.go:122] "RemoveContainer" containerID="fef6158bf6966da0f422d935dde07ce9051c4799433367471811f84ace6b868d" Nov 03 02:30:04.016996 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:04.016940 5605 scope.go:122] "RemoveContainer" containerID="ab505a1a2652408280ec1314153348ac4217217780e6ccc3b1571f4f63cdfa30" Nov 03 02:30:04.017270 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:30:04.017246 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-fb1940-29894550-jw8ct_odh-ai-gateway-infra(433647e9-f97a-470d-bd48-cf2c6f9d6064)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fb1940-29894550-jw8ct" podUID="433647e9-f97a-470d-bd48-cf2c6f9d6064" Nov 03 02:30:04.018819 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:04.018796 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894550-2pn5m" event={"ID":"c5a086a9-767c-4878-9b57-de00c92df859","Type":"ContainerStarted","Data":"c779789c0f6a5842c579a8b7f4bb6a6dff40ca9f82812a65edad54ee995359c1"} Nov 03 02:30:04.020634 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:04.020609 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-496177-29894550-h27bc" event={"ID":"a9cd06a2-1beb-4d2b-9561-4e4a3b788e4d","Type":"ContainerStarted","Data":"44fb9305cf489283b8cd896217b2b7396282d8899f4c73c2673c1553d1916d3f"} Nov 03 02:30:04.022861 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:04.022838 5605 scope.go:122] "RemoveContainer" containerID="43f38364147173104c1da0668c520b25822555629a75b6339efecc4f7e52c67c" Nov 03 02:30:04.023086 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:30:04.023064 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-711318-29894550-g884l_odh-ai-gateway-infra(a45e73e4-5176-408e-bd3c-b0412e7a3f88)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-711318-29894550-g884l" podUID="a45e73e4-5176-408e-bd3c-b0412e7a3f88" Nov 03 02:30:04.025097 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:04.025076 5605 scope.go:122] "RemoveContainer" containerID="001026d31fa4eeb1d72d8e82ab4afef6c685f24c83f32181aa4d96ad2e83de16" Nov 03 02:30:04.025350 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:30:04.025327 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894550-dx9j4_odh-ai-gateway-infra(6eff793f-67d6-4044-87fb-ca5e46a5e6ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894550-dx9j4" podUID="6eff793f-67d6-4044-87fb-ca5e46a5e6ac" Nov 03 02:30:04.026918 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:04.026866 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894550-jhpt8" event={"ID":"963f49af-b49b-4664-a9a5-57937449e0df","Type":"ContainerStarted","Data":"a9b2ee3ee6486e8d9a64280301ef874d2d9c4c307fc98103253c4db17cf89381"} Nov 03 02:30:04.029055 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:04.029026 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894550-n7jzq" event={"ID":"25790141-f19b-4e73-8b61-4c82f33a1825","Type":"ContainerStarted","Data":"f224a6fcc1f9600c4ba1a164557bd02e6d26801b38c6fbfb199066a2b258547e"} Nov 03 02:30:04.031219 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:04.031201 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-f760d3-29894550-8kxmb" event={"ID":"3bb55cd1-e726-45c4-b1fd-b03229ec0378","Type":"ContainerStarted","Data":"3c028dbb31cb21997e8804d0d01b2ecae05f58a8086d64db3c0594e87121ab3d"} Nov 03 02:30:05.039153 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:05.039116 5605 generic.go:356] "Generic (PLEG): container finished" podID="963f49af-b49b-4664-a9a5-57937449e0df" containerID="a9b2ee3ee6486e8d9a64280301ef874d2d9c4c307fc98103253c4db17cf89381" exitCode=60 Nov 03 02:30:05.039635 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:05.039193 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894550-jhpt8" event={"ID":"963f49af-b49b-4664-a9a5-57937449e0df","Type":"ContainerDied","Data":"a9b2ee3ee6486e8d9a64280301ef874d2d9c4c307fc98103253c4db17cf89381"} Nov 03 02:30:05.039635 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:05.039254 5605 scope.go:122] "RemoveContainer" containerID="669423c7fe322e69e521dce14fde51e6dccd8f35f039f3528fe5301834f9f4b7" Nov 03 02:30:05.039635 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:05.039462 5605 scope.go:122] "RemoveContainer" containerID="a9b2ee3ee6486e8d9a64280301ef874d2d9c4c307fc98103253c4db17cf89381" Nov 03 02:30:05.039792 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:30:05.039764 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894550-jhpt8_odh-ai-gateway-infra(963f49af-b49b-4664-a9a5-57937449e0df)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894550-jhpt8" podUID="963f49af-b49b-4664-a9a5-57937449e0df" Nov 03 02:30:05.041228 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:05.041202 5605 generic.go:356] "Generic (PLEG): container finished" podID="25790141-f19b-4e73-8b61-4c82f33a1825" containerID="f224a6fcc1f9600c4ba1a164557bd02e6d26801b38c6fbfb199066a2b258547e" exitCode=60 Nov 03 02:30:05.041311 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:05.041242 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894550-n7jzq" event={"ID":"25790141-f19b-4e73-8b61-4c82f33a1825","Type":"ContainerDied","Data":"f224a6fcc1f9600c4ba1a164557bd02e6d26801b38c6fbfb199066a2b258547e"} Nov 03 02:30:05.041536 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:05.041521 5605 scope.go:122] "RemoveContainer" containerID="f224a6fcc1f9600c4ba1a164557bd02e6d26801b38c6fbfb199066a2b258547e" Nov 03 02:30:05.041807 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:30:05.041788 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-c7d758-29894550-n7jzq_odh-ai-gateway-infra(25790141-f19b-4e73-8b61-4c82f33a1825)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894550-n7jzq" podUID="25790141-f19b-4e73-8b61-4c82f33a1825" Nov 03 02:30:05.043020 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:05.042997 5605 generic.go:356] "Generic (PLEG): container finished" podID="3bb55cd1-e726-45c4-b1fd-b03229ec0378" containerID="3c028dbb31cb21997e8804d0d01b2ecae05f58a8086d64db3c0594e87121ab3d" exitCode=60 Nov 03 02:30:05.043116 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:05.043070 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-f760d3-29894550-8kxmb" event={"ID":"3bb55cd1-e726-45c4-b1fd-b03229ec0378","Type":"ContainerDied","Data":"3c028dbb31cb21997e8804d0d01b2ecae05f58a8086d64db3c0594e87121ab3d"} Nov 03 02:30:05.043287 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:05.043273 5605 scope.go:122] "RemoveContainer" containerID="3c028dbb31cb21997e8804d0d01b2ecae05f58a8086d64db3c0594e87121ab3d" Nov 03 02:30:05.043551 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:30:05.043530 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-f760d3-29894550-8kxmb_odh-ai-gateway-infra(3bb55cd1-e726-45c4-b1fd-b03229ec0378)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-f760d3-29894550-8kxmb" podUID="3bb55cd1-e726-45c4-b1fd-b03229ec0378" Nov 03 02:30:05.044917 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:05.044899 5605 scope.go:122] "RemoveContainer" containerID="ab505a1a2652408280ec1314153348ac4217217780e6ccc3b1571f4f63cdfa30" Nov 03 02:30:05.045165 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:30:05.045147 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-fb1940-29894550-jw8ct_odh-ai-gateway-infra(433647e9-f97a-470d-bd48-cf2c6f9d6064)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fb1940-29894550-jw8ct" podUID="433647e9-f97a-470d-bd48-cf2c6f9d6064" Nov 03 02:30:05.046494 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:05.046472 5605 generic.go:356] "Generic (PLEG): container finished" podID="c5a086a9-767c-4878-9b57-de00c92df859" containerID="c779789c0f6a5842c579a8b7f4bb6a6dff40ca9f82812a65edad54ee995359c1" exitCode=60 Nov 03 02:30:05.046579 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:05.046548 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894550-2pn5m" event={"ID":"c5a086a9-767c-4878-9b57-de00c92df859","Type":"ContainerDied","Data":"c779789c0f6a5842c579a8b7f4bb6a6dff40ca9f82812a65edad54ee995359c1"} Nov 03 02:30:05.046791 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:05.046774 5605 scope.go:122] "RemoveContainer" containerID="c779789c0f6a5842c579a8b7f4bb6a6dff40ca9f82812a65edad54ee995359c1" Nov 03 02:30:05.047023 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:30:05.047000 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-521f1f-29894550-2pn5m_odh-ai-gateway-infra(c5a086a9-767c-4878-9b57-de00c92df859)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894550-2pn5m" podUID="c5a086a9-767c-4878-9b57-de00c92df859" Nov 03 02:30:05.048421 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:05.048400 5605 generic.go:356] "Generic (PLEG): container finished" podID="a9cd06a2-1beb-4d2b-9561-4e4a3b788e4d" containerID="44fb9305cf489283b8cd896217b2b7396282d8899f4c73c2673c1553d1916d3f" exitCode=60 Nov 03 02:30:05.048526 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:05.048490 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-496177-29894550-h27bc" event={"ID":"a9cd06a2-1beb-4d2b-9561-4e4a3b788e4d","Type":"ContainerDied","Data":"44fb9305cf489283b8cd896217b2b7396282d8899f4c73c2673c1553d1916d3f"} Nov 03 02:30:05.048648 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:05.048635 5605 scope.go:122] "RemoveContainer" containerID="44fb9305cf489283b8cd896217b2b7396282d8899f4c73c2673c1553d1916d3f" Nov 03 02:30:05.048825 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:30:05.048803 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-496177-29894550-h27bc_odh-ai-gateway-infra(a9cd06a2-1beb-4d2b-9561-4e4a3b788e4d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-496177-29894550-h27bc" podUID="a9cd06a2-1beb-4d2b-9561-4e4a3b788e4d" Nov 03 02:30:05.051647 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:05.051629 5605 scope.go:122] "RemoveContainer" containerID="850da39706cfe082e1f210556e6d4d7665476f8c93d91ed96c9d056507e9595c" Nov 03 02:30:05.063763 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:05.063738 5605 scope.go:122] "RemoveContainer" containerID="fbe1e7dbb3e3b0302fb15c88a83901b8b20f5016163471b3fa15b5d7aa8f1c73" Nov 03 02:30:05.076263 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:05.076246 5605 scope.go:122] "RemoveContainer" containerID="789a4c1aec4f05397e5868742c838b6b05cf4d2af00f3554b04ca8826ffac3c9" Nov 03 02:30:05.088995 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:05.088973 5605 scope.go:122] "RemoveContainer" containerID="d3029199015ac21f0e312f0ca2ce282f349ef028da1bbce55e45c19fd18784a0" Nov 03 02:30:06.055915 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:06.055885 5605 scope.go:122] "RemoveContainer" containerID="a9b2ee3ee6486e8d9a64280301ef874d2d9c4c307fc98103253c4db17cf89381" Nov 03 02:30:06.056376 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:30:06.056223 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894550-jhpt8_odh-ai-gateway-infra(963f49af-b49b-4664-a9a5-57937449e0df)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894550-jhpt8" podUID="963f49af-b49b-4664-a9a5-57937449e0df" Nov 03 02:30:06.057487 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:06.057462 5605 scope.go:122] "RemoveContainer" containerID="f224a6fcc1f9600c4ba1a164557bd02e6d26801b38c6fbfb199066a2b258547e" Nov 03 02:30:06.057753 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:30:06.057732 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-c7d758-29894550-n7jzq_odh-ai-gateway-infra(25790141-f19b-4e73-8b61-4c82f33a1825)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894550-n7jzq" podUID="25790141-f19b-4e73-8b61-4c82f33a1825" Nov 03 02:30:06.058851 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:06.058830 5605 scope.go:122] "RemoveContainer" containerID="3c028dbb31cb21997e8804d0d01b2ecae05f58a8086d64db3c0594e87121ab3d" Nov 03 02:30:06.059068 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:30:06.059047 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-f760d3-29894550-8kxmb_odh-ai-gateway-infra(3bb55cd1-e726-45c4-b1fd-b03229ec0378)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-f760d3-29894550-8kxmb" podUID="3bb55cd1-e726-45c4-b1fd-b03229ec0378" Nov 03 02:30:06.060262 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:06.060244 5605 scope.go:122] "RemoveContainer" containerID="c779789c0f6a5842c579a8b7f4bb6a6dff40ca9f82812a65edad54ee995359c1" Nov 03 02:30:06.060474 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:30:06.060451 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-521f1f-29894550-2pn5m_odh-ai-gateway-infra(c5a086a9-767c-4878-9b57-de00c92df859)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894550-2pn5m" podUID="c5a086a9-767c-4878-9b57-de00c92df859" Nov 03 02:30:06.061644 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:06.061629 5605 scope.go:122] "RemoveContainer" containerID="44fb9305cf489283b8cd896217b2b7396282d8899f4c73c2673c1553d1916d3f" Nov 03 02:30:06.061822 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:30:06.061805 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-496177-29894550-h27bc_odh-ai-gateway-infra(a9cd06a2-1beb-4d2b-9561-4e4a3b788e4d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-496177-29894550-h27bc" podUID="a9cd06a2-1beb-4d2b-9561-4e4a3b788e4d" Nov 03 02:30:14.844106 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:14.844068 5605 scope.go:122] "RemoveContainer" containerID="3c028dbb31cb21997e8804d0d01b2ecae05f58a8086d64db3c0594e87121ab3d" Nov 03 02:30:14.844624 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:14.844122 5605 scope.go:122] "RemoveContainer" containerID="ab505a1a2652408280ec1314153348ac4217217780e6ccc3b1571f4f63cdfa30" Nov 03 02:30:15.667163 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:15.667131 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:30:15.843938 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:15.843912 5605 scope.go:122] "RemoveContainer" containerID="c779789c0f6a5842c579a8b7f4bb6a6dff40ca9f82812a65edad54ee995359c1" Nov 03 02:30:16.129710 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:16.129672 5605 generic.go:356] "Generic (PLEG): container finished" podID="433647e9-f97a-470d-bd48-cf2c6f9d6064" containerID="17d0639621ead60cde71cf2092a3efe8f8bfb4e6ed7ff3b386ef7a78870e32c3" exitCode=60 Nov 03 02:30:16.130133 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:16.129739 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fb1940-29894550-jw8ct" event={"ID":"433647e9-f97a-470d-bd48-cf2c6f9d6064","Type":"ContainerDied","Data":"17d0639621ead60cde71cf2092a3efe8f8bfb4e6ed7ff3b386ef7a78870e32c3"} Nov 03 02:30:16.130133 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:16.129796 5605 scope.go:122] "RemoveContainer" containerID="ab505a1a2652408280ec1314153348ac4217217780e6ccc3b1571f4f63cdfa30" Nov 03 02:30:16.130223 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:16.130179 5605 scope.go:122] "RemoveContainer" containerID="17d0639621ead60cde71cf2092a3efe8f8bfb4e6ed7ff3b386ef7a78870e32c3" Nov 03 02:30:16.130531 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:30:16.130497 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-fb1940-29894550-jw8ct_odh-ai-gateway-infra(433647e9-f97a-470d-bd48-cf2c6f9d6064)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fb1940-29894550-jw8ct" podUID="433647e9-f97a-470d-bd48-cf2c6f9d6064" Nov 03 02:30:16.132292 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:16.132272 5605 generic.go:356] "Generic (PLEG): container finished" podID="3bb55cd1-e726-45c4-b1fd-b03229ec0378" containerID="270dc00040bcc5b19edf27a02beeddfc5cb568757bd9c26677387d4adbd201a8" exitCode=60 Nov 03 02:30:16.132419 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:16.132298 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-f760d3-29894550-8kxmb" event={"ID":"3bb55cd1-e726-45c4-b1fd-b03229ec0378","Type":"ContainerDied","Data":"270dc00040bcc5b19edf27a02beeddfc5cb568757bd9c26677387d4adbd201a8"} Nov 03 02:30:16.132635 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:16.132617 5605 scope.go:122] "RemoveContainer" containerID="270dc00040bcc5b19edf27a02beeddfc5cb568757bd9c26677387d4adbd201a8" Nov 03 02:30:16.132862 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:30:16.132845 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-f760d3-29894550-8kxmb_odh-ai-gateway-infra(3bb55cd1-e726-45c4-b1fd-b03229ec0378)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-f760d3-29894550-8kxmb" podUID="3bb55cd1-e726-45c4-b1fd-b03229ec0378" Nov 03 02:30:16.143148 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:16.143132 5605 scope.go:122] "RemoveContainer" containerID="3c028dbb31cb21997e8804d0d01b2ecae05f58a8086d64db3c0594e87121ab3d" Nov 03 02:30:16.847011 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:16.846975 5605 scope.go:122] "RemoveContainer" containerID="a9b2ee3ee6486e8d9a64280301ef874d2d9c4c307fc98103253c4db17cf89381" Nov 03 02:30:16.847886 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:16.847857 5605 scope.go:122] "RemoveContainer" containerID="44fb9305cf489283b8cd896217b2b7396282d8899f4c73c2673c1553d1916d3f" Nov 03 02:30:16.851232 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:16.849632 5605 scope.go:122] "RemoveContainer" containerID="001026d31fa4eeb1d72d8e82ab4afef6c685f24c83f32181aa4d96ad2e83de16" Nov 03 02:30:17.144346 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:17.144316 5605 generic.go:356] "Generic (PLEG): container finished" podID="c5a086a9-767c-4878-9b57-de00c92df859" containerID="611d3e66e588b190a3af244a4eedf7d7568696284fb36587ecd64c0bee2a6327" exitCode=60 Nov 03 02:30:17.144816 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:17.144401 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894550-2pn5m" event={"ID":"c5a086a9-767c-4878-9b57-de00c92df859","Type":"ContainerDied","Data":"611d3e66e588b190a3af244a4eedf7d7568696284fb36587ecd64c0bee2a6327"} Nov 03 02:30:17.144816 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:17.144472 5605 scope.go:122] "RemoveContainer" containerID="c779789c0f6a5842c579a8b7f4bb6a6dff40ca9f82812a65edad54ee995359c1" Nov 03 02:30:17.144946 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:17.144874 5605 scope.go:122] "RemoveContainer" containerID="611d3e66e588b190a3af244a4eedf7d7568696284fb36587ecd64c0bee2a6327" Nov 03 02:30:17.145192 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:30:17.145161 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-521f1f-29894550-2pn5m_odh-ai-gateway-infra(c5a086a9-767c-4878-9b57-de00c92df859)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894550-2pn5m" podUID="c5a086a9-767c-4878-9b57-de00c92df859" Nov 03 02:30:17.300062 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:17.300030 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fb1940-29894550-jw8ct"] Nov 03 02:30:17.404496 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:17.394373 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-f760d3-29894550-8kxmb"] Nov 03 02:30:17.532232 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:17.532202 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fb1940-29894550-jw8ct" Nov 03 02:30:17.630513 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:17.629282 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/433647e9-f97a-470d-bd48-cf2c6f9d6064-kube-api-access-jlsrs\" (UniqueName: \"kubernetes.io/projected/433647e9-f97a-470d-bd48-cf2c6f9d6064-kube-api-access-jlsrs\") pod \"433647e9-f97a-470d-bd48-cf2c6f9d6064\" (UID: \"433647e9-f97a-470d-bd48-cf2c6f9d6064\") " Nov 03 02:30:17.630513 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:17.629341 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/433647e9-f97a-470d-bd48-cf2c6f9d6064-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/433647e9-f97a-470d-bd48-cf2c6f9d6064-maas-api-service-ca\") pod \"433647e9-f97a-470d-bd48-cf2c6f9d6064\" (UID: \"433647e9-f97a-470d-bd48-cf2c6f9d6064\") " Nov 03 02:30:17.631849 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:17.630887 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/433647e9-f97a-470d-bd48-cf2c6f9d6064-maas-api-service-ca" pod "433647e9-f97a-470d-bd48-cf2c6f9d6064" (UID: "433647e9-f97a-470d-bd48-cf2c6f9d6064"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:30:17.638956 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:17.638645 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/433647e9-f97a-470d-bd48-cf2c6f9d6064-kube-api-access-jlsrs" pod "433647e9-f97a-470d-bd48-cf2c6f9d6064" (UID: "433647e9-f97a-470d-bd48-cf2c6f9d6064"). InnerVolumeSpecName "kube-api-access-jlsrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:30:17.728374 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:17.728322 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-f760d3-29894550-8kxmb" Nov 03 02:30:17.731521 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:17.731490 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlsrs\" (UniqueName: \"kubernetes.io/projected/433647e9-f97a-470d-bd48-cf2c6f9d6064-kube-api-access-jlsrs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:17.731521 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:17.731516 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/433647e9-f97a-470d-bd48-cf2c6f9d6064-maas-api-service-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:17.832814 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:17.832774 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bb55cd1-e726-45c4-b1fd-b03229ec0378-kube-api-access-zddrf\" (UniqueName: \"kubernetes.io/projected/3bb55cd1-e726-45c4-b1fd-b03229ec0378-kube-api-access-zddrf\") pod \"3bb55cd1-e726-45c4-b1fd-b03229ec0378\" (UID: \"3bb55cd1-e726-45c4-b1fd-b03229ec0378\") " Nov 03 02:30:17.832964 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:17.832864 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bb55cd1-e726-45c4-b1fd-b03229ec0378-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bb55cd1-e726-45c4-b1fd-b03229ec0378-maas-api-service-ca\") pod \"3bb55cd1-e726-45c4-b1fd-b03229ec0378\" (UID: \"3bb55cd1-e726-45c4-b1fd-b03229ec0378\") " Nov 03 02:30:17.833448 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:17.833418 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bb55cd1-e726-45c4-b1fd-b03229ec0378-maas-api-service-ca" pod "3bb55cd1-e726-45c4-b1fd-b03229ec0378" (UID: "3bb55cd1-e726-45c4-b1fd-b03229ec0378"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:30:17.835447 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:17.835419 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bb55cd1-e726-45c4-b1fd-b03229ec0378-kube-api-access-zddrf" pod "3bb55cd1-e726-45c4-b1fd-b03229ec0378" (UID: "3bb55cd1-e726-45c4-b1fd-b03229ec0378"). InnerVolumeSpecName "kube-api-access-zddrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:30:17.843709 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:17.843685 5605 scope.go:122] "RemoveContainer" containerID="43f38364147173104c1da0668c520b25822555629a75b6339efecc4f7e52c67c" Nov 03 02:30:17.934799 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:17.934768 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zddrf\" (UniqueName: \"kubernetes.io/projected/3bb55cd1-e726-45c4-b1fd-b03229ec0378-kube-api-access-zddrf\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:17.934945 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:17.934802 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bb55cd1-e726-45c4-b1fd-b03229ec0378-maas-api-service-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:18.151593 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:18.151502 5605 generic.go:356] "Generic (PLEG): container finished" podID="963f49af-b49b-4664-a9a5-57937449e0df" containerID="343c73322f1b99ad59d827f3d58a71b908410c8cfad428655f6010a5168599be" exitCode=60 Nov 03 02:30:18.152014 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:18.151581 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894550-jhpt8" event={"ID":"963f49af-b49b-4664-a9a5-57937449e0df","Type":"ContainerDied","Data":"343c73322f1b99ad59d827f3d58a71b908410c8cfad428655f6010a5168599be"} Nov 03 02:30:18.152014 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:18.151653 5605 scope.go:122] "RemoveContainer" containerID="a9b2ee3ee6486e8d9a64280301ef874d2d9c4c307fc98103253c4db17cf89381" Nov 03 02:30:18.152014 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:18.151972 5605 scope.go:122] "RemoveContainer" containerID="343c73322f1b99ad59d827f3d58a71b908410c8cfad428655f6010a5168599be" Nov 03 02:30:18.152461 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:30:18.152248 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894550-jhpt8_odh-ai-gateway-infra(963f49af-b49b-4664-a9a5-57937449e0df)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894550-jhpt8" podUID="963f49af-b49b-4664-a9a5-57937449e0df" Nov 03 02:30:18.153036 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:18.153018 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-f760d3-29894550-8kxmb" Nov 03 02:30:18.153147 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:18.153045 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-f760d3-29894550-8kxmb" event={"ID":"3bb55cd1-e726-45c4-b1fd-b03229ec0378","Type":"ContainerDied","Data":"f01273b124bff2ba60377051a629cf82bab38e4d516f864f6ed105acefd4bd8f"} Nov 03 02:30:18.164421 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:18.154348 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fb1940-29894550-jw8ct" Nov 03 02:30:18.164421 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:18.154345 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fb1940-29894550-jw8ct" event={"ID":"433647e9-f97a-470d-bd48-cf2c6f9d6064","Type":"ContainerDied","Data":"904850f3da329af43476624771b5339f40e513f2e7801ab0f6e9ef9d75588336"} Nov 03 02:30:18.179114 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:18.178176 5605 generic.go:356] "Generic (PLEG): container finished" podID="a9cd06a2-1beb-4d2b-9561-4e4a3b788e4d" containerID="8d6087b22404ffeba4835e3a0bdd588628bcabe33bc436ee58e0df0fb37287c2" exitCode=60 Nov 03 02:30:18.179114 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:18.178298 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-496177-29894550-h27bc" event={"ID":"a9cd06a2-1beb-4d2b-9561-4e4a3b788e4d","Type":"ContainerDied","Data":"8d6087b22404ffeba4835e3a0bdd588628bcabe33bc436ee58e0df0fb37287c2"} Nov 03 02:30:18.179114 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:18.178736 5605 scope.go:122] "RemoveContainer" containerID="8d6087b22404ffeba4835e3a0bdd588628bcabe33bc436ee58e0df0fb37287c2" Nov 03 02:30:18.179114 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:30:18.179043 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-496177-29894550-h27bc_odh-ai-gateway-infra(a9cd06a2-1beb-4d2b-9561-4e4a3b788e4d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-496177-29894550-h27bc" podUID="a9cd06a2-1beb-4d2b-9561-4e4a3b788e4d" Nov 03 02:30:18.182380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:18.181691 5605 generic.go:356] "Generic (PLEG): container finished" podID="6eff793f-67d6-4044-87fb-ca5e46a5e6ac" containerID="aa2795226bbedc033b3a84378149f69e3a5dadad2462454be8391a80876731ec" exitCode=60 Nov 03 02:30:18.182380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:18.181728 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894550-dx9j4" event={"ID":"6eff793f-67d6-4044-87fb-ca5e46a5e6ac","Type":"ContainerDied","Data":"aa2795226bbedc033b3a84378149f69e3a5dadad2462454be8391a80876731ec"} Nov 03 02:30:18.182380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:18.182082 5605 scope.go:122] "RemoveContainer" containerID="aa2795226bbedc033b3a84378149f69e3a5dadad2462454be8391a80876731ec" Nov 03 02:30:18.182676 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:30:18.182351 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894550-dx9j4_odh-ai-gateway-infra(6eff793f-67d6-4044-87fb-ca5e46a5e6ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894550-dx9j4" podUID="6eff793f-67d6-4044-87fb-ca5e46a5e6ac" Nov 03 02:30:18.213155 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:18.213133 5605 scope.go:122] "RemoveContainer" containerID="270dc00040bcc5b19edf27a02beeddfc5cb568757bd9c26677387d4adbd201a8" Nov 03 02:30:18.223818 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:18.223797 5605 scope.go:122] "RemoveContainer" containerID="17d0639621ead60cde71cf2092a3efe8f8bfb4e6ed7ff3b386ef7a78870e32c3" Nov 03 02:30:18.234298 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:18.234274 5605 scope.go:122] "RemoveContainer" containerID="44fb9305cf489283b8cd896217b2b7396282d8899f4c73c2673c1553d1916d3f" Nov 03 02:30:18.246535 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:18.246516 5605 scope.go:122] "RemoveContainer" containerID="001026d31fa4eeb1d72d8e82ab4afef6c685f24c83f32181aa4d96ad2e83de16" Nov 03 02:30:18.300191 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:18.299271 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894550-2pn5m"] Nov 03 02:30:18.477603 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:18.477565 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894550-2pn5m" Nov 03 02:30:18.485203 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:18.485159 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fb1940-29894550-jw8ct"] Nov 03 02:30:18.488720 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:18.488690 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fb1940-29894550-jw8ct"] Nov 03 02:30:18.541856 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:18.541815 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5a086a9-767c-4878-9b57-de00c92df859-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5a086a9-767c-4878-9b57-de00c92df859-maas-api-service-ca\") pod \"c5a086a9-767c-4878-9b57-de00c92df859\" (UID: \"c5a086a9-767c-4878-9b57-de00c92df859\") " Nov 03 02:30:18.542035 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:18.541941 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5a086a9-767c-4878-9b57-de00c92df859-kube-api-access-jfplg\" (UniqueName: \"kubernetes.io/projected/c5a086a9-767c-4878-9b57-de00c92df859-kube-api-access-jfplg\") pod \"c5a086a9-767c-4878-9b57-de00c92df859\" (UID: \"c5a086a9-767c-4878-9b57-de00c92df859\") " Nov 03 02:30:18.542153 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:18.542132 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5a086a9-767c-4878-9b57-de00c92df859-maas-api-service-ca" pod "c5a086a9-767c-4878-9b57-de00c92df859" (UID: "c5a086a9-767c-4878-9b57-de00c92df859"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:30:18.542345 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:18.542323 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5a086a9-767c-4878-9b57-de00c92df859-maas-api-service-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:18.543943 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:18.543926 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5a086a9-767c-4878-9b57-de00c92df859-kube-api-access-jfplg" pod "c5a086a9-767c-4878-9b57-de00c92df859" (UID: "c5a086a9-767c-4878-9b57-de00c92df859"). InnerVolumeSpecName "kube-api-access-jfplg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:30:18.593022 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:18.592980 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-f760d3-29894550-8kxmb"] Nov 03 02:30:18.600135 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:18.600108 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-f760d3-29894550-8kxmb"] Nov 03 02:30:18.643317 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:18.643285 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfplg\" (UniqueName: \"kubernetes.io/projected/c5a086a9-767c-4878-9b57-de00c92df859-kube-api-access-jfplg\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:18.847168 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:18.845900 5605 scope.go:122] "RemoveContainer" containerID="f224a6fcc1f9600c4ba1a164557bd02e6d26801b38c6fbfb199066a2b258547e" Nov 03 02:30:18.858626 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:18.853572 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bb55cd1-e726-45c4-b1fd-b03229ec0378" path="/var/lib/kubelet/pods/3bb55cd1-e726-45c4-b1fd-b03229ec0378/volumes" Nov 03 02:30:18.858626 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:18.854086 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="433647e9-f97a-470d-bd48-cf2c6f9d6064" path="/var/lib/kubelet/pods/433647e9-f97a-470d-bd48-cf2c6f9d6064/volumes" Nov 03 02:30:19.192569 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:19.192544 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894550-2pn5m" Nov 03 02:30:19.192934 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:19.192580 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894550-2pn5m" event={"ID":"c5a086a9-767c-4878-9b57-de00c92df859","Type":"ContainerDied","Data":"560935f6678ea9c285678c271f9f06339cb4a4aca10be2d1723bb7d1c2d3c93d"} Nov 03 02:30:19.192934 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:19.192651 5605 scope.go:122] "RemoveContainer" containerID="611d3e66e588b190a3af244a4eedf7d7568696284fb36587ecd64c0bee2a6327" Nov 03 02:30:19.196530 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:19.196506 5605 generic.go:356] "Generic (PLEG): container finished" podID="a45e73e4-5176-408e-bd3c-b0412e7a3f88" containerID="3e139a5b290a58e7cc06370a8c6cfd33836a566e9b2316241e4e1bfe9232a9a7" exitCode=60 Nov 03 02:30:19.196643 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:19.196578 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-711318-29894550-g884l" event={"ID":"a45e73e4-5176-408e-bd3c-b0412e7a3f88","Type":"ContainerDied","Data":"3e139a5b290a58e7cc06370a8c6cfd33836a566e9b2316241e4e1bfe9232a9a7"} Nov 03 02:30:19.197021 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:19.197002 5605 scope.go:122] "RemoveContainer" containerID="3e139a5b290a58e7cc06370a8c6cfd33836a566e9b2316241e4e1bfe9232a9a7" Nov 03 02:30:19.197351 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:30:19.197306 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-711318-29894550-g884l_odh-ai-gateway-infra(a45e73e4-5176-408e-bd3c-b0412e7a3f88)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-711318-29894550-g884l" podUID="a45e73e4-5176-408e-bd3c-b0412e7a3f88" Nov 03 02:30:19.201824 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:19.201808 5605 scope.go:122] "RemoveContainer" containerID="43f38364147173104c1da0668c520b25822555629a75b6339efecc4f7e52c67c" Nov 03 02:30:19.280840 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:19.280336 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894550-jhpt8"] Nov 03 02:30:19.347260 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:19.347229 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894550-2pn5m"] Nov 03 02:30:19.351569 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:19.351541 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894550-2pn5m"] Nov 03 02:30:19.390737 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:19.390701 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894550-dx9j4"] Nov 03 02:30:19.488586 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:19.488552 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894550-jhpt8" Nov 03 02:30:19.532870 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:19.532839 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894550-dx9j4" Nov 03 02:30:19.554105 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:19.554081 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6eff793f-67d6-4044-87fb-ca5e46a5e6ac-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eff793f-67d6-4044-87fb-ca5e46a5e6ac-maas-api-service-ca\") pod \"6eff793f-67d6-4044-87fb-ca5e46a5e6ac\" (UID: \"6eff793f-67d6-4044-87fb-ca5e46a5e6ac\") " Nov 03 02:30:19.554241 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:19.554212 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6eff793f-67d6-4044-87fb-ca5e46a5e6ac-kube-api-access-gt6sq\" (UniqueName: \"kubernetes.io/projected/6eff793f-67d6-4044-87fb-ca5e46a5e6ac-kube-api-access-gt6sq\") pod \"6eff793f-67d6-4044-87fb-ca5e46a5e6ac\" (UID: \"6eff793f-67d6-4044-87fb-ca5e46a5e6ac\") " Nov 03 02:30:19.554321 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:19.554244 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/963f49af-b49b-4664-a9a5-57937449e0df-kube-api-access-dht8s\" (UniqueName: \"kubernetes.io/projected/963f49af-b49b-4664-a9a5-57937449e0df-kube-api-access-dht8s\") pod \"963f49af-b49b-4664-a9a5-57937449e0df\" (UID: \"963f49af-b49b-4664-a9a5-57937449e0df\") " Nov 03 02:30:19.554321 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:19.554282 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/963f49af-b49b-4664-a9a5-57937449e0df-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/963f49af-b49b-4664-a9a5-57937449e0df-maas-api-service-ca\") pod \"963f49af-b49b-4664-a9a5-57937449e0df\" (UID: \"963f49af-b49b-4664-a9a5-57937449e0df\") " Nov 03 02:30:19.574381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:19.554495 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6eff793f-67d6-4044-87fb-ca5e46a5e6ac-maas-api-service-ca" pod "6eff793f-67d6-4044-87fb-ca5e46a5e6ac" (UID: "6eff793f-67d6-4044-87fb-ca5e46a5e6ac"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:30:19.574381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:19.554745 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eff793f-67d6-4044-87fb-ca5e46a5e6ac-maas-api-service-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:19.575927 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:19.575165 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/963f49af-b49b-4664-a9a5-57937449e0df-maas-api-service-ca" pod "963f49af-b49b-4664-a9a5-57937449e0df" (UID: "963f49af-b49b-4664-a9a5-57937449e0df"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:30:19.585917 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:19.579559 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/963f49af-b49b-4664-a9a5-57937449e0df-kube-api-access-dht8s" pod "963f49af-b49b-4664-a9a5-57937449e0df" (UID: "963f49af-b49b-4664-a9a5-57937449e0df"). InnerVolumeSpecName "kube-api-access-dht8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:30:19.586845 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:19.586817 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eff793f-67d6-4044-87fb-ca5e46a5e6ac-kube-api-access-gt6sq" pod "6eff793f-67d6-4044-87fb-ca5e46a5e6ac" (UID: "6eff793f-67d6-4044-87fb-ca5e46a5e6ac"). InnerVolumeSpecName "kube-api-access-gt6sq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:30:19.680239 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:19.680162 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt6sq\" (UniqueName: \"kubernetes.io/projected/6eff793f-67d6-4044-87fb-ca5e46a5e6ac-kube-api-access-gt6sq\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:19.680239 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:19.680194 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dht8s\" (UniqueName: \"kubernetes.io/projected/963f49af-b49b-4664-a9a5-57937449e0df-kube-api-access-dht8s\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:19.680239 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:19.680214 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/963f49af-b49b-4664-a9a5-57937449e0df-maas-api-service-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:19.714120 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:19.713756 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-496177-29894550-h27bc"] Nov 03 02:30:19.911223 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:19.911192 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-496177-29894550-h27bc" Nov 03 02:30:19.991791 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:19.987775 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9cd06a2-1beb-4d2b-9561-4e4a3b788e4d-kube-api-access-x76nr\" (UniqueName: \"kubernetes.io/projected/a9cd06a2-1beb-4d2b-9561-4e4a3b788e4d-kube-api-access-x76nr\") pod \"a9cd06a2-1beb-4d2b-9561-4e4a3b788e4d\" (UID: \"a9cd06a2-1beb-4d2b-9561-4e4a3b788e4d\") " Nov 03 02:30:19.991791 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:19.987940 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9cd06a2-1beb-4d2b-9561-4e4a3b788e4d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9cd06a2-1beb-4d2b-9561-4e4a3b788e4d-maas-api-service-ca\") pod \"a9cd06a2-1beb-4d2b-9561-4e4a3b788e4d\" (UID: \"a9cd06a2-1beb-4d2b-9561-4e4a3b788e4d\") " Nov 03 02:30:20.002606 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:20.002571 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9cd06a2-1beb-4d2b-9561-4e4a3b788e4d-maas-api-service-ca" pod "a9cd06a2-1beb-4d2b-9561-4e4a3b788e4d" (UID: "a9cd06a2-1beb-4d2b-9561-4e4a3b788e4d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:30:20.006556 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:20.006524 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9cd06a2-1beb-4d2b-9561-4e4a3b788e4d-kube-api-access-x76nr" pod "a9cd06a2-1beb-4d2b-9561-4e4a3b788e4d" (UID: "a9cd06a2-1beb-4d2b-9561-4e4a3b788e4d"). InnerVolumeSpecName "kube-api-access-x76nr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:30:20.094685 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:20.094647 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x76nr\" (UniqueName: \"kubernetes.io/projected/a9cd06a2-1beb-4d2b-9561-4e4a3b788e4d-kube-api-access-x76nr\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:20.094685 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:20.094678 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9cd06a2-1beb-4d2b-9561-4e4a3b788e4d-maas-api-service-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:20.206828 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:20.206785 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-496177-29894550-h27bc" event={"ID":"a9cd06a2-1beb-4d2b-9561-4e4a3b788e4d","Type":"ContainerDied","Data":"29dc5f2d4d42465b626bc03ab1d31e33ca81d43afe9f2c187b0424cf38bcc0b1"} Nov 03 02:30:20.206828 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:20.206772 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-496177-29894550-h27bc" Nov 03 02:30:20.207407 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:20.206888 5605 scope.go:122] "RemoveContainer" containerID="8d6087b22404ffeba4835e3a0bdd588628bcabe33bc436ee58e0df0fb37287c2" Nov 03 02:30:20.210743 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:20.210707 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894550-dx9j4" event={"ID":"6eff793f-67d6-4044-87fb-ca5e46a5e6ac","Type":"ContainerDied","Data":"19a94d75cd11778b8a88246a9b62916aae28862feb7237583020915a5e9c3d0e"} Nov 03 02:30:20.211099 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:20.211019 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894550-dx9j4" Nov 03 02:30:20.212560 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:20.212543 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894550-jhpt8" Nov 03 02:30:20.212627 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:20.212555 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894550-jhpt8" event={"ID":"963f49af-b49b-4664-a9a5-57937449e0df","Type":"ContainerDied","Data":"48af6c53151cde1cd57e98130351f28d9caf3497e4ed57796d2d758e7263ecd2"} Nov 03 02:30:20.217847 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:20.217773 5605 scope.go:122] "RemoveContainer" containerID="aa2795226bbedc033b3a84378149f69e3a5dadad2462454be8391a80876731ec" Nov 03 02:30:20.217958 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:20.217942 5605 generic.go:356] "Generic (PLEG): container finished" podID="25790141-f19b-4e73-8b61-4c82f33a1825" containerID="85335277935a1b0abe1a478c77754c0ca95eefb8cd80b881e4ebff727308e4cc" exitCode=60 Nov 03 02:30:20.218019 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:20.217980 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894550-n7jzq" event={"ID":"25790141-f19b-4e73-8b61-4c82f33a1825","Type":"ContainerDied","Data":"85335277935a1b0abe1a478c77754c0ca95eefb8cd80b881e4ebff727308e4cc"} Nov 03 02:30:20.218522 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:20.218412 5605 scope.go:122] "RemoveContainer" containerID="85335277935a1b0abe1a478c77754c0ca95eefb8cd80b881e4ebff727308e4cc" Nov 03 02:30:20.218919 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:30:20.218683 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-c7d758-29894550-n7jzq_odh-ai-gateway-infra(25790141-f19b-4e73-8b61-4c82f33a1825)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894550-n7jzq" podUID="25790141-f19b-4e73-8b61-4c82f33a1825" Nov 03 02:30:20.229308 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:20.229289 5605 scope.go:122] "RemoveContainer" containerID="343c73322f1b99ad59d827f3d58a71b908410c8cfad428655f6010a5168599be" Nov 03 02:30:20.237461 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:20.237439 5605 scope.go:122] "RemoveContainer" containerID="f224a6fcc1f9600c4ba1a164557bd02e6d26801b38c6fbfb199066a2b258547e" Nov 03 02:30:20.382030 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:20.381995 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-496177-29894550-h27bc"] Nov 03 02:30:20.384373 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:20.384335 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-496177-29894550-h27bc"] Nov 03 02:30:20.488863 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:20.488767 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-711318-29894550-g884l"] Nov 03 02:30:20.492043 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:20.492011 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894550-dx9j4"] Nov 03 02:30:20.494882 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:20.494807 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894550-dx9j4"] Nov 03 02:30:20.589308 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:20.589278 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894550-jhpt8"] Nov 03 02:30:20.593972 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:20.593453 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894550-jhpt8"] Nov 03 02:30:20.631339 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:20.631314 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-711318-29894550-g884l" Nov 03 02:30:20.704091 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:20.704060 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a45e73e4-5176-408e-bd3c-b0412e7a3f88-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a45e73e4-5176-408e-bd3c-b0412e7a3f88-maas-api-service-ca\") pod \"a45e73e4-5176-408e-bd3c-b0412e7a3f88\" (UID: \"a45e73e4-5176-408e-bd3c-b0412e7a3f88\") " Nov 03 02:30:20.704244 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:20.704141 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a45e73e4-5176-408e-bd3c-b0412e7a3f88-kube-api-access-zvh82\" (UniqueName: \"kubernetes.io/projected/a45e73e4-5176-408e-bd3c-b0412e7a3f88-kube-api-access-zvh82\") pod \"a45e73e4-5176-408e-bd3c-b0412e7a3f88\" (UID: \"a45e73e4-5176-408e-bd3c-b0412e7a3f88\") " Nov 03 02:30:20.704441 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:20.704417 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a45e73e4-5176-408e-bd3c-b0412e7a3f88-maas-api-service-ca" pod "a45e73e4-5176-408e-bd3c-b0412e7a3f88" (UID: "a45e73e4-5176-408e-bd3c-b0412e7a3f88"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:30:20.704574 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:20.704557 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a45e73e4-5176-408e-bd3c-b0412e7a3f88-maas-api-service-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:20.706292 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:20.706267 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a45e73e4-5176-408e-bd3c-b0412e7a3f88-kube-api-access-zvh82" pod "a45e73e4-5176-408e-bd3c-b0412e7a3f88" (UID: "a45e73e4-5176-408e-bd3c-b0412e7a3f88"). InnerVolumeSpecName "kube-api-access-zvh82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:30:20.805481 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:20.805450 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvh82\" (UniqueName: \"kubernetes.io/projected/a45e73e4-5176-408e-bd3c-b0412e7a3f88-kube-api-access-zvh82\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:20.850848 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:20.850809 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6eff793f-67d6-4044-87fb-ca5e46a5e6ac" path="/var/lib/kubelet/pods/6eff793f-67d6-4044-87fb-ca5e46a5e6ac/volumes" Nov 03 02:30:20.851251 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:20.851237 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="963f49af-b49b-4664-a9a5-57937449e0df" path="/var/lib/kubelet/pods/963f49af-b49b-4664-a9a5-57937449e0df/volumes" Nov 03 02:30:20.851632 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:20.851619 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9cd06a2-1beb-4d2b-9561-4e4a3b788e4d" path="/var/lib/kubelet/pods/a9cd06a2-1beb-4d2b-9561-4e4a3b788e4d/volumes" Nov 03 02:30:20.852292 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:20.852278 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5a086a9-767c-4878-9b57-de00c92df859" path="/var/lib/kubelet/pods/c5a086a9-767c-4878-9b57-de00c92df859/volumes" Nov 03 02:30:21.228197 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:21.228124 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-711318-29894550-g884l" Nov 03 02:30:21.228649 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:21.228121 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-711318-29894550-g884l" event={"ID":"a45e73e4-5176-408e-bd3c-b0412e7a3f88","Type":"ContainerDied","Data":"5e51cc21abe0c3b0d8f876270b78fade9239d523494dad02583c782a14bec5cc"} Nov 03 02:30:21.228649 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:21.228251 5605 scope.go:122] "RemoveContainer" containerID="3e139a5b290a58e7cc06370a8c6cfd33836a566e9b2316241e4e1bfe9232a9a7" Nov 03 02:30:21.400496 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:21.400452 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-711318-29894550-g884l"] Nov 03 02:30:21.402628 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:21.402592 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-711318-29894550-g884l"] Nov 03 02:30:21.707095 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:21.707064 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894550-n7jzq"] Nov 03 02:30:22.011389 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:22.004695 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894550-n7jzq" Nov 03 02:30:22.122510 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:22.122480 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25790141-f19b-4e73-8b61-4c82f33a1825-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25790141-f19b-4e73-8b61-4c82f33a1825-maas-api-service-ca\") pod \"25790141-f19b-4e73-8b61-4c82f33a1825\" (UID: \"25790141-f19b-4e73-8b61-4c82f33a1825\") " Nov 03 02:30:22.122690 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:22.122653 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25790141-f19b-4e73-8b61-4c82f33a1825-kube-api-access-gdjgz\" (UniqueName: \"kubernetes.io/projected/25790141-f19b-4e73-8b61-4c82f33a1825-kube-api-access-gdjgz\") pod \"25790141-f19b-4e73-8b61-4c82f33a1825\" (UID: \"25790141-f19b-4e73-8b61-4c82f33a1825\") " Nov 03 02:30:22.122928 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:22.122893 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25790141-f19b-4e73-8b61-4c82f33a1825-maas-api-service-ca" pod "25790141-f19b-4e73-8b61-4c82f33a1825" (UID: "25790141-f19b-4e73-8b61-4c82f33a1825"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:30:22.124735 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:22.124712 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25790141-f19b-4e73-8b61-4c82f33a1825-kube-api-access-gdjgz" pod "25790141-f19b-4e73-8b61-4c82f33a1825" (UID: "25790141-f19b-4e73-8b61-4c82f33a1825"). InnerVolumeSpecName "kube-api-access-gdjgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:30:22.224947 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:22.224854 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25790141-f19b-4e73-8b61-4c82f33a1825-maas-api-service-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:22.224947 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:22.224893 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdjgz\" (UniqueName: \"kubernetes.io/projected/25790141-f19b-4e73-8b61-4c82f33a1825-kube-api-access-gdjgz\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:22.236832 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:22.236803 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894550-n7jzq" Nov 03 02:30:22.236832 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:22.236819 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894550-n7jzq" event={"ID":"25790141-f19b-4e73-8b61-4c82f33a1825","Type":"ContainerDied","Data":"3eb248a18a54d37acf4a3c1bb883af54494a1dcb7f8137e0048fecc89e2f2e02"} Nov 03 02:30:22.237266 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:22.236875 5605 scope.go:122] "RemoveContainer" containerID="85335277935a1b0abe1a478c77754c0ca95eefb8cd80b881e4ebff727308e4cc" Nov 03 02:30:22.391119 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:22.391084 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894550-n7jzq"] Nov 03 02:30:22.396230 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:22.396200 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894550-n7jzq"] Nov 03 02:30:22.850577 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:22.850543 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25790141-f19b-4e73-8b61-4c82f33a1825" path="/var/lib/kubelet/pods/25790141-f19b-4e73-8b61-4c82f33a1825/volumes" Nov 03 02:30:22.850945 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:22.850931 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a45e73e4-5176-408e-bd3c-b0412e7a3f88" path="/var/lib/kubelet/pods/a45e73e4-5176-408e-bd3c-b0412e7a3f88/volumes" Nov 03 02:30:23.565558 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:23.565522 5605 patch_prober.go:29] interesting pod/e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:30:23.566045 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:23.565610 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5" podUID="6e8485b2-ec6f-4344-bee7-37495006a34c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:30:23.918535 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:30:23.918451 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:30:25.500593 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:25.500558 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f760d3-984756bb7-5v2b6"] Nov 03 02:30:26.263500 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:26.263462 5605 generic.go:356] "Generic (PLEG): container finished" podID="6e8485b2-ec6f-4344-bee7-37495006a34c" containerID="d0a29f070628364b4e12e88048780351f44c00f620f53883a89bebec92ab2128" exitCode=0 Nov 03 02:30:26.263711 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:26.263550 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5" event={"ID":"6e8485b2-ec6f-4344-bee7-37495006a34c","Type":"ContainerDied","Data":"d0a29f070628364b4e12e88048780351f44c00f620f53883a89bebec92ab2128"} Nov 03 02:30:26.264117 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:26.264100 5605 scope.go:122] "RemoveContainer" containerID="d0a29f070628364b4e12e88048780351f44c00f620f53883a89bebec92ab2128" Nov 03 02:30:26.265755 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:26.265728 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:30:26.265823 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:26.265796 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:30:26.265861 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:26.265828 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:30:27.298407 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:27.298352 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5" event={"ID":"6e8485b2-ec6f-4344-bee7-37495006a34c","Type":"ContainerStarted","Data":"4fb8e5c48c374a4d444dac5503aa9b95e1f55b06ae604b8d66138b8c2f355d38"} Nov 03 02:30:27.771022 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:27.770396 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76"] Nov 03 02:30:27.771022 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:27.770692 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76" podUID="4bd56b15-0fc7-4a81-a569-4eba9ffaf645" containerName="istio-proxy" containerID="cri-o://b3fcbcb6fb615e86c873dacad12dec42505fcfae737f0f08a8b91503ece900b5" gracePeriod=30 Nov 03 02:30:27.852973 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:27.852945 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5" Nov 03 02:30:27.853136 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:27.853084 5605 patch_prober.go:29] interesting pod/e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 02:30:27.853196 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:27.853136 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5" podUID="6e8485b2-ec6f-4344-bee7-37495006a34c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 02:30:28.853879 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:28.853843 5605 patch_prober.go:29] interesting pod/e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 02:30:28.854262 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:28.853897 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5" podUID="6e8485b2-ec6f-4344-bee7-37495006a34c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 02:30:28.881674 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:28.881637 5605 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 02:30:28.881820 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:28.881693 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76" podUID="4bd56b15-0fc7-4a81-a569-4eba9ffaf645" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 02:30:29.853967 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:29.853928 5605 patch_prober.go:29] interesting pod/e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 02:30:29.854503 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:29.853985 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5" podUID="6e8485b2-ec6f-4344-bee7-37495006a34c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 02:30:30.501728 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:30.501663 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-f760d3-984756bb7-5v2b6" podUID="d8afe8b7-91ea-4e07-a283-1ce1f81a814a" containerName="payload-processing" containerID="cri-o://84631ae50a4b0e3cf6b72ed63288761e0c45b9342ee873bc8e77b4af79b65d1a" gracePeriod=25 Nov 03 02:30:30.792827 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:30.792796 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-f760d3-984756bb7-5v2b6" Nov 03 02:30:30.834529 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:30.834495 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8afe8b7-91ea-4e07-a283-1ce1f81a814a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8afe8b7-91ea-4e07-a283-1ce1f81a814a-plugins-config-volume\") pod \"d8afe8b7-91ea-4e07-a283-1ce1f81a814a\" (UID: \"d8afe8b7-91ea-4e07-a283-1ce1f81a814a\") " Nov 03 02:30:30.834710 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:30.834593 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8afe8b7-91ea-4e07-a283-1ce1f81a814a-kube-api-access-xspqg\" (UniqueName: \"kubernetes.io/projected/d8afe8b7-91ea-4e07-a283-1ce1f81a814a-kube-api-access-xspqg\") pod \"d8afe8b7-91ea-4e07-a283-1ce1f81a814a\" (UID: \"d8afe8b7-91ea-4e07-a283-1ce1f81a814a\") " Nov 03 02:30:30.834910 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:30.834886 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8afe8b7-91ea-4e07-a283-1ce1f81a814a-plugins-config-volume" pod "d8afe8b7-91ea-4e07-a283-1ce1f81a814a" (UID: "d8afe8b7-91ea-4e07-a283-1ce1f81a814a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:30:30.836611 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:30.836590 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8afe8b7-91ea-4e07-a283-1ce1f81a814a-kube-api-access-xspqg" pod "d8afe8b7-91ea-4e07-a283-1ce1f81a814a" (UID: "d8afe8b7-91ea-4e07-a283-1ce1f81a814a"). InnerVolumeSpecName "kube-api-access-xspqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:30:30.853747 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:30.853720 5605 patch_prober.go:29] interesting pod/e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 02:30:30.853848 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:30.853782 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5" podUID="6e8485b2-ec6f-4344-bee7-37495006a34c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 02:30:30.935817 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:30.935767 5605 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8afe8b7-91ea-4e07-a283-1ce1f81a814a-plugins-config-volume\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:30.935817 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:30.935817 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xspqg\" (UniqueName: \"kubernetes.io/projected/d8afe8b7-91ea-4e07-a283-1ce1f81a814a-kube-api-access-xspqg\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:31.322649 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:31.322613 5605 generic.go:356] "Generic (PLEG): container finished" podID="d8afe8b7-91ea-4e07-a283-1ce1f81a814a" containerID="84631ae50a4b0e3cf6b72ed63288761e0c45b9342ee873bc8e77b4af79b65d1a" exitCode=0 Nov 03 02:30:31.322830 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:31.322658 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-f760d3-984756bb7-5v2b6" Nov 03 02:30:31.322830 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:31.322693 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-f760d3-984756bb7-5v2b6" event={"ID":"d8afe8b7-91ea-4e07-a283-1ce1f81a814a","Type":"ContainerDied","Data":"84631ae50a4b0e3cf6b72ed63288761e0c45b9342ee873bc8e77b4af79b65d1a"} Nov 03 02:30:31.322830 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:31.322751 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-f760d3-984756bb7-5v2b6" event={"ID":"d8afe8b7-91ea-4e07-a283-1ce1f81a814a","Type":"ContainerDied","Data":"d93a7604d4f7a2b045f47f9a7d0250c1acf5fa329377f55ca7e14b46d1ff7f52"} Nov 03 02:30:31.322830 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:31.322776 5605 scope.go:122] "RemoveContainer" containerID="84631ae50a4b0e3cf6b72ed63288761e0c45b9342ee873bc8e77b4af79b65d1a" Nov 03 02:30:31.334966 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:31.334943 5605 scope.go:122] "RemoveContainer" containerID="84631ae50a4b0e3cf6b72ed63288761e0c45b9342ee873bc8e77b4af79b65d1a" Nov 03 02:30:31.335246 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:30:31.335226 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84631ae50a4b0e3cf6b72ed63288761e0c45b9342ee873bc8e77b4af79b65d1a\": container with ID starting with 84631ae50a4b0e3cf6b72ed63288761e0c45b9342ee873bc8e77b4af79b65d1a not found: ID does not exist" containerID="84631ae50a4b0e3cf6b72ed63288761e0c45b9342ee873bc8e77b4af79b65d1a" Nov 03 02:30:31.335314 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:31.335253 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84631ae50a4b0e3cf6b72ed63288761e0c45b9342ee873bc8e77b4af79b65d1a"} err="failed to get container status \"84631ae50a4b0e3cf6b72ed63288761e0c45b9342ee873bc8e77b4af79b65d1a\": rpc error: code = NotFound desc = could not find container \"84631ae50a4b0e3cf6b72ed63288761e0c45b9342ee873bc8e77b4af79b65d1a\": container with ID starting with 84631ae50a4b0e3cf6b72ed63288761e0c45b9342ee873bc8e77b4af79b65d1a not found: ID does not exist" Nov 03 02:30:31.480292 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:31.480251 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f760d3-984756bb7-5v2b6"] Nov 03 02:30:31.525559 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:31.525529 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f760d3-984756bb7-5v2b6"] Nov 03 02:30:31.853224 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:31.853192 5605 patch_prober.go:29] interesting pod/e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 02:30:31.853436 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:31.853260 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5" podUID="6e8485b2-ec6f-4344-bee7-37495006a34c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 02:30:32.850650 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:32.850621 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8afe8b7-91ea-4e07-a283-1ce1f81a814a" path="/var/lib/kubelet/pods/d8afe8b7-91ea-4e07-a283-1ce1f81a814a/volumes" Nov 03 02:30:32.857568 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:32.857545 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5" Nov 03 02:30:32.857944 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:32.857927 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5" Nov 03 02:30:32.858675 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:32.858653 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5" Nov 03 02:30:33.037931 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:33.037907 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76" Nov 03 02:30:33.181236 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:33.181203 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bd56b15-0fc7-4a81-a569-4eba9ffaf645-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bd56b15-0fc7-4a81-a569-4eba9ffaf645-istio-envoy\") pod \"4bd56b15-0fc7-4a81-a569-4eba9ffaf645\" (UID: \"4bd56b15-0fc7-4a81-a569-4eba9ffaf645\") " Nov 03 02:30:33.181453 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:33.181286 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bd56b15-0fc7-4a81-a569-4eba9ffaf645-istio-token\" (UniqueName: \"kubernetes.io/projected/4bd56b15-0fc7-4a81-a569-4eba9ffaf645-istio-token\") pod \"4bd56b15-0fc7-4a81-a569-4eba9ffaf645\" (UID: \"4bd56b15-0fc7-4a81-a569-4eba9ffaf645\") " Nov 03 02:30:33.181453 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:33.181380 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bd56b15-0fc7-4a81-a569-4eba9ffaf645-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bd56b15-0fc7-4a81-a569-4eba9ffaf645-istiod-ca-cert\") pod \"4bd56b15-0fc7-4a81-a569-4eba9ffaf645\" (UID: \"4bd56b15-0fc7-4a81-a569-4eba9ffaf645\") " Nov 03 02:30:33.181453 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:33.181429 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bd56b15-0fc7-4a81-a569-4eba9ffaf645-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bd56b15-0fc7-4a81-a569-4eba9ffaf645-istio-data\") pod \"4bd56b15-0fc7-4a81-a569-4eba9ffaf645\" (UID: \"4bd56b15-0fc7-4a81-a569-4eba9ffaf645\") " Nov 03 02:30:33.181639 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:33.181464 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bd56b15-0fc7-4a81-a569-4eba9ffaf645-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bd56b15-0fc7-4a81-a569-4eba9ffaf645-credential-socket\") pod \"4bd56b15-0fc7-4a81-a569-4eba9ffaf645\" (UID: \"4bd56b15-0fc7-4a81-a569-4eba9ffaf645\") " Nov 03 02:30:33.181639 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:33.181543 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4bd56b15-0fc7-4a81-a569-4eba9ffaf645-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bd56b15-0fc7-4a81-a569-4eba9ffaf645-istio-podinfo\") pod \"4bd56b15-0fc7-4a81-a569-4eba9ffaf645\" (UID: \"4bd56b15-0fc7-4a81-a569-4eba9ffaf645\") " Nov 03 02:30:33.181639 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:33.181580 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bd56b15-0fc7-4a81-a569-4eba9ffaf645-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bd56b15-0fc7-4a81-a569-4eba9ffaf645-workload-certs\") pod \"4bd56b15-0fc7-4a81-a569-4eba9ffaf645\" (UID: \"4bd56b15-0fc7-4a81-a569-4eba9ffaf645\") " Nov 03 02:30:33.181639 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:33.181621 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bd56b15-0fc7-4a81-a569-4eba9ffaf645-kube-api-access-dqqjm\" (UniqueName: \"kubernetes.io/projected/4bd56b15-0fc7-4a81-a569-4eba9ffaf645-kube-api-access-dqqjm\") pod \"4bd56b15-0fc7-4a81-a569-4eba9ffaf645\" (UID: \"4bd56b15-0fc7-4a81-a569-4eba9ffaf645\") " Nov 03 02:30:33.181881 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:33.181649 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bd56b15-0fc7-4a81-a569-4eba9ffaf645-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bd56b15-0fc7-4a81-a569-4eba9ffaf645-workload-socket\") pod \"4bd56b15-0fc7-4a81-a569-4eba9ffaf645\" (UID: \"4bd56b15-0fc7-4a81-a569-4eba9ffaf645\") " Nov 03 02:30:33.181881 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:33.181678 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bd56b15-0fc7-4a81-a569-4eba9ffaf645-istio-data" pod "4bd56b15-0fc7-4a81-a569-4eba9ffaf645" (UID: "4bd56b15-0fc7-4a81-a569-4eba9ffaf645"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:30:33.181881 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:33.181846 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bd56b15-0fc7-4a81-a569-4eba9ffaf645-workload-certs" pod "4bd56b15-0fc7-4a81-a569-4eba9ffaf645" (UID: "4bd56b15-0fc7-4a81-a569-4eba9ffaf645"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:30:33.182094 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:33.182067 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bd56b15-0fc7-4a81-a569-4eba9ffaf645-istiod-ca-cert" pod "4bd56b15-0fc7-4a81-a569-4eba9ffaf645" (UID: "4bd56b15-0fc7-4a81-a569-4eba9ffaf645"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:30:33.182094 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:33.182082 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bd56b15-0fc7-4a81-a569-4eba9ffaf645-credential-socket" pod "4bd56b15-0fc7-4a81-a569-4eba9ffaf645" (UID: "4bd56b15-0fc7-4a81-a569-4eba9ffaf645"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:30:33.182248 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:33.182071 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bd56b15-0fc7-4a81-a569-4eba9ffaf645-workload-socket" pod "4bd56b15-0fc7-4a81-a569-4eba9ffaf645" (UID: "4bd56b15-0fc7-4a81-a569-4eba9ffaf645"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:30:33.182307 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:33.182243 5605 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bd56b15-0fc7-4a81-a569-4eba9ffaf645-workload-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:33.182307 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:33.182268 5605 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bd56b15-0fc7-4a81-a569-4eba9ffaf645-workload-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:33.182307 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:33.182283 5605 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bd56b15-0fc7-4a81-a569-4eba9ffaf645-istiod-ca-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:33.182307 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:33.182296 5605 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bd56b15-0fc7-4a81-a569-4eba9ffaf645-istio-data\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:33.182552 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:33.182309 5605 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bd56b15-0fc7-4a81-a569-4eba9ffaf645-credential-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:33.183911 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:33.183887 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bd56b15-0fc7-4a81-a569-4eba9ffaf645-kube-api-access-dqqjm" pod "4bd56b15-0fc7-4a81-a569-4eba9ffaf645" (UID: "4bd56b15-0fc7-4a81-a569-4eba9ffaf645"). InnerVolumeSpecName "kube-api-access-dqqjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:30:33.184053 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:33.183919 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4bd56b15-0fc7-4a81-a569-4eba9ffaf645-istio-podinfo" pod "4bd56b15-0fc7-4a81-a569-4eba9ffaf645" (UID: "4bd56b15-0fc7-4a81-a569-4eba9ffaf645"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:30:33.184053 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:33.184003 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bd56b15-0fc7-4a81-a569-4eba9ffaf645-istio-token" pod "4bd56b15-0fc7-4a81-a569-4eba9ffaf645" (UID: "4bd56b15-0fc7-4a81-a569-4eba9ffaf645"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:30:33.184169 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:33.184087 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bd56b15-0fc7-4a81-a569-4eba9ffaf645-istio-envoy" pod "4bd56b15-0fc7-4a81-a569-4eba9ffaf645" (UID: "4bd56b15-0fc7-4a81-a569-4eba9ffaf645"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:30:33.283681 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:33.283647 5605 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bd56b15-0fc7-4a81-a569-4eba9ffaf645-istio-podinfo\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:33.283681 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:33.283682 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqqjm\" (UniqueName: \"kubernetes.io/projected/4bd56b15-0fc7-4a81-a569-4eba9ffaf645-kube-api-access-dqqjm\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:33.283887 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:33.283699 5605 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bd56b15-0fc7-4a81-a569-4eba9ffaf645-istio-envoy\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:33.283887 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:33.283714 5605 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bd56b15-0fc7-4a81-a569-4eba9ffaf645-istio-token\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:33.335966 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:33.335932 5605 generic.go:356] "Generic (PLEG): container finished" podID="4bd56b15-0fc7-4a81-a569-4eba9ffaf645" containerID="b3fcbcb6fb615e86c873dacad12dec42505fcfae737f0f08a8b91503ece900b5" exitCode=0 Nov 03 02:30:33.336156 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:33.335982 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76" Nov 03 02:30:33.336156 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:33.336011 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76" event={"ID":"4bd56b15-0fc7-4a81-a569-4eba9ffaf645","Type":"ContainerDied","Data":"b3fcbcb6fb615e86c873dacad12dec42505fcfae737f0f08a8b91503ece900b5"} Nov 03 02:30:33.336156 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:33.336068 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76" event={"ID":"4bd56b15-0fc7-4a81-a569-4eba9ffaf645","Type":"ContainerDied","Data":"c65707b594fb5f4346bd4491ca654467845ce8c7cd6aa7caa9f9fb38d6a0213a"} Nov 03 02:30:33.336156 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:33.336093 5605 scope.go:122] "RemoveContainer" containerID="b3fcbcb6fb615e86c873dacad12dec42505fcfae737f0f08a8b91503ece900b5" Nov 03 02:30:33.349058 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:33.349041 5605 scope.go:122] "RemoveContainer" containerID="b3fcbcb6fb615e86c873dacad12dec42505fcfae737f0f08a8b91503ece900b5" Nov 03 02:30:33.349439 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:30:33.349339 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3fcbcb6fb615e86c873dacad12dec42505fcfae737f0f08a8b91503ece900b5\": container with ID starting with b3fcbcb6fb615e86c873dacad12dec42505fcfae737f0f08a8b91503ece900b5 not found: ID does not exist" containerID="b3fcbcb6fb615e86c873dacad12dec42505fcfae737f0f08a8b91503ece900b5" Nov 03 02:30:33.349439 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:33.349388 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3fcbcb6fb615e86c873dacad12dec42505fcfae737f0f08a8b91503ece900b5"} err="failed to get container status \"b3fcbcb6fb615e86c873dacad12dec42505fcfae737f0f08a8b91503ece900b5\": rpc error: code = NotFound desc = could not find container \"b3fcbcb6fb615e86c873dacad12dec42505fcfae737f0f08a8b91503ece900b5\": container with ID starting with b3fcbcb6fb615e86c873dacad12dec42505fcfae737f0f08a8b91503ece900b5 not found: ID does not exist" Nov 03 02:30:33.494939 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:33.494851 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76"] Nov 03 02:30:33.496742 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:33.496713 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-mpr76"] Nov 03 02:30:34.627137 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.627103 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7ba433a-b5pwq"] Nov 03 02:30:34.627603 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.627545 5605 cpu_manager.go:414] "containerMap: removing container" podUID="a45e73e4-5176-408e-bd3c-b0412e7a3f88" containerName="cleanup" Nov 03 02:30:34.627603 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.627557 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a45e73e4-5176-408e-bd3c-b0412e7a3f88" containerName="cleanup" Nov 03 02:30:34.627603 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.627567 5605 cpu_manager.go:414] "containerMap: removing container" podUID="963f49af-b49b-4664-a9a5-57937449e0df" containerName="cleanup" Nov 03 02:30:34.627603 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.627572 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="963f49af-b49b-4664-a9a5-57937449e0df" containerName="cleanup" Nov 03 02:30:34.627603 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.627584 5605 cpu_manager.go:414] "containerMap: removing container" podUID="a9cd06a2-1beb-4d2b-9561-4e4a3b788e4d" containerName="cleanup" Nov 03 02:30:34.627603 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.627589 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9cd06a2-1beb-4d2b-9561-4e4a3b788e4d" containerName="cleanup" Nov 03 02:30:34.627603 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.627596 5605 cpu_manager.go:414] "containerMap: removing container" podUID="a45e73e4-5176-408e-bd3c-b0412e7a3f88" containerName="cleanup" Nov 03 02:30:34.627603 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.627601 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a45e73e4-5176-408e-bd3c-b0412e7a3f88" containerName="cleanup" Nov 03 02:30:34.627603 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.627607 5605 cpu_manager.go:414] "containerMap: removing container" podUID="433647e9-f97a-470d-bd48-cf2c6f9d6064" containerName="cleanup" Nov 03 02:30:34.627603 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.627613 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="433647e9-f97a-470d-bd48-cf2c6f9d6064" containerName="cleanup" Nov 03 02:30:34.627942 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.627626 5605 cpu_manager.go:414] "containerMap: removing container" podUID="433647e9-f97a-470d-bd48-cf2c6f9d6064" containerName="cleanup" Nov 03 02:30:34.627942 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.627631 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="433647e9-f97a-470d-bd48-cf2c6f9d6064" containerName="cleanup" Nov 03 02:30:34.627942 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.627637 5605 cpu_manager.go:414] "containerMap: removing container" podUID="a9cd06a2-1beb-4d2b-9561-4e4a3b788e4d" containerName="cleanup" Nov 03 02:30:34.627942 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.627641 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9cd06a2-1beb-4d2b-9561-4e4a3b788e4d" containerName="cleanup" Nov 03 02:30:34.627942 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.627650 5605 cpu_manager.go:414] "containerMap: removing container" podUID="c5a086a9-767c-4878-9b57-de00c92df859" containerName="cleanup" Nov 03 02:30:34.627942 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.627654 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5a086a9-767c-4878-9b57-de00c92df859" containerName="cleanup" Nov 03 02:30:34.627942 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.627665 5605 cpu_manager.go:414] "containerMap: removing container" podUID="6eff793f-67d6-4044-87fb-ca5e46a5e6ac" containerName="cleanup" Nov 03 02:30:34.627942 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.627672 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6eff793f-67d6-4044-87fb-ca5e46a5e6ac" containerName="cleanup" Nov 03 02:30:34.627942 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.627681 5605 cpu_manager.go:414] "containerMap: removing container" podUID="c5a086a9-767c-4878-9b57-de00c92df859" containerName="cleanup" Nov 03 02:30:34.627942 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.627688 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5a086a9-767c-4878-9b57-de00c92df859" containerName="cleanup" Nov 03 02:30:34.627942 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.627704 5605 cpu_manager.go:414] "containerMap: removing container" podUID="d8afe8b7-91ea-4e07-a283-1ce1f81a814a" containerName="payload-processing" Nov 03 02:30:34.627942 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.627710 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8afe8b7-91ea-4e07-a283-1ce1f81a814a" containerName="payload-processing" Nov 03 02:30:34.627942 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.627717 5605 cpu_manager.go:414] "containerMap: removing container" podUID="25790141-f19b-4e73-8b61-4c82f33a1825" containerName="cleanup" Nov 03 02:30:34.627942 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.627721 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25790141-f19b-4e73-8b61-4c82f33a1825" containerName="cleanup" Nov 03 02:30:34.627942 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.627729 5605 cpu_manager.go:414] "containerMap: removing container" podUID="3bb55cd1-e726-45c4-b1fd-b03229ec0378" containerName="cleanup" Nov 03 02:30:34.627942 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.627733 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bb55cd1-e726-45c4-b1fd-b03229ec0378" containerName="cleanup" Nov 03 02:30:34.627942 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.627739 5605 cpu_manager.go:414] "containerMap: removing container" podUID="4bd56b15-0fc7-4a81-a569-4eba9ffaf645" containerName="istio-proxy" Nov 03 02:30:34.627942 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.627743 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bd56b15-0fc7-4a81-a569-4eba9ffaf645" containerName="istio-proxy" Nov 03 02:30:34.627942 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.627748 5605 cpu_manager.go:414] "containerMap: removing container" podUID="c5a086a9-767c-4878-9b57-de00c92df859" containerName="cleanup" Nov 03 02:30:34.627942 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.627752 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5a086a9-767c-4878-9b57-de00c92df859" containerName="cleanup" Nov 03 02:30:34.627942 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.627759 5605 cpu_manager.go:414] "containerMap: removing container" podUID="25790141-f19b-4e73-8b61-4c82f33a1825" containerName="cleanup" Nov 03 02:30:34.627942 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.627763 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25790141-f19b-4e73-8b61-4c82f33a1825" containerName="cleanup" Nov 03 02:30:34.627942 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.627770 5605 cpu_manager.go:414] "containerMap: removing container" podUID="a45e73e4-5176-408e-bd3c-b0412e7a3f88" containerName="cleanup" Nov 03 02:30:34.627942 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.627774 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a45e73e4-5176-408e-bd3c-b0412e7a3f88" containerName="cleanup" Nov 03 02:30:34.627942 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.627781 5605 cpu_manager.go:414] "containerMap: removing container" podUID="6eff793f-67d6-4044-87fb-ca5e46a5e6ac" containerName="cleanup" Nov 03 02:30:34.627942 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.627786 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6eff793f-67d6-4044-87fb-ca5e46a5e6ac" containerName="cleanup" Nov 03 02:30:34.627942 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.627791 5605 cpu_manager.go:414] "containerMap: removing container" podUID="3bb55cd1-e726-45c4-b1fd-b03229ec0378" containerName="cleanup" Nov 03 02:30:34.627942 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.627795 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bb55cd1-e726-45c4-b1fd-b03229ec0378" containerName="cleanup" Nov 03 02:30:34.627942 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.627805 5605 cpu_manager.go:414] "containerMap: removing container" podUID="6eff793f-67d6-4044-87fb-ca5e46a5e6ac" containerName="cleanup" Nov 03 02:30:34.627942 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.627811 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6eff793f-67d6-4044-87fb-ca5e46a5e6ac" containerName="cleanup" Nov 03 02:30:34.627942 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.627819 5605 cpu_manager.go:414] "containerMap: removing container" podUID="25790141-f19b-4e73-8b61-4c82f33a1825" containerName="cleanup" Nov 03 02:30:34.627942 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.627823 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25790141-f19b-4e73-8b61-4c82f33a1825" containerName="cleanup" Nov 03 02:30:34.627942 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.627834 5605 cpu_manager.go:414] "containerMap: removing container" podUID="963f49af-b49b-4664-a9a5-57937449e0df" containerName="cleanup" Nov 03 02:30:34.627942 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.627838 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="963f49af-b49b-4664-a9a5-57937449e0df" containerName="cleanup" Nov 03 02:30:34.627942 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.627899 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="6eff793f-67d6-4044-87fb-ca5e46a5e6ac" containerName="cleanup" Nov 03 02:30:34.627942 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.627906 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5a086a9-767c-4878-9b57-de00c92df859" containerName="cleanup" Nov 03 02:30:34.627942 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.627912 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bb55cd1-e726-45c4-b1fd-b03229ec0378" containerName="cleanup" Nov 03 02:30:34.627942 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.627917 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="963f49af-b49b-4664-a9a5-57937449e0df" containerName="cleanup" Nov 03 02:30:34.627942 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.627923 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8afe8b7-91ea-4e07-a283-1ce1f81a814a" containerName="payload-processing" Nov 03 02:30:34.627942 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.627930 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="25790141-f19b-4e73-8b61-4c82f33a1825" containerName="cleanup" Nov 03 02:30:34.627942 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.627935 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="a45e73e4-5176-408e-bd3c-b0412e7a3f88" containerName="cleanup" Nov 03 02:30:34.627942 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.627940 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bd56b15-0fc7-4a81-a569-4eba9ffaf645" containerName="istio-proxy" Nov 03 02:30:34.627942 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.627947 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bb55cd1-e726-45c4-b1fd-b03229ec0378" containerName="cleanup" Nov 03 02:30:34.627942 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.627952 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5a086a9-767c-4878-9b57-de00c92df859" containerName="cleanup" Nov 03 02:30:34.627942 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.627958 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9cd06a2-1beb-4d2b-9561-4e4a3b788e4d" containerName="cleanup" Nov 03 02:30:34.627942 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.627963 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="a45e73e4-5176-408e-bd3c-b0412e7a3f88" containerName="cleanup" Nov 03 02:30:34.629506 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.627969 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bb55cd1-e726-45c4-b1fd-b03229ec0378" containerName="cleanup" Nov 03 02:30:34.629506 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.627974 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="433647e9-f97a-470d-bd48-cf2c6f9d6064" containerName="cleanup" Nov 03 02:30:34.629506 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.627979 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="963f49af-b49b-4664-a9a5-57937449e0df" containerName="cleanup" Nov 03 02:30:34.629506 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.627983 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="25790141-f19b-4e73-8b61-4c82f33a1825" containerName="cleanup" Nov 03 02:30:34.629506 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.627988 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="a45e73e4-5176-408e-bd3c-b0412e7a3f88" containerName="cleanup" Nov 03 02:30:34.629506 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.627994 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9cd06a2-1beb-4d2b-9561-4e4a3b788e4d" containerName="cleanup" Nov 03 02:30:34.629506 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.627999 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="6eff793f-67d6-4044-87fb-ca5e46a5e6ac" containerName="cleanup" Nov 03 02:30:34.629506 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.628005 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5a086a9-767c-4878-9b57-de00c92df859" containerName="cleanup" Nov 03 02:30:34.629506 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.628010 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9cd06a2-1beb-4d2b-9561-4e4a3b788e4d" containerName="cleanup" Nov 03 02:30:34.629506 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.628015 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="433647e9-f97a-470d-bd48-cf2c6f9d6064" containerName="cleanup" Nov 03 02:30:34.632547 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.632529 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7ba433a-b5pwq" Nov 03 02:30:34.682564 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.682098 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:30:34.682564 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.682162 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:30:34.682564 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.682220 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:30:34.683632 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.683607 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7ba433a-b5pwq"] Nov 03 02:30:34.698188 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.698160 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3807ac44-ce6e-407f-a37f-77f9afb60819-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7ba433a-b5pwq\" (UID: \"3807ac44-ce6e-407f-a37f-77f9afb60819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7ba433a-b5pwq" Nov 03 02:30:34.698310 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.698219 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3807ac44-ce6e-407f-a37f-77f9afb60819-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7ba433a-b5pwq\" (UID: \"3807ac44-ce6e-407f-a37f-77f9afb60819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7ba433a-b5pwq" Nov 03 02:30:34.698367 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.698316 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnsdg\" (UniqueName: \"kubernetes.io/projected/3807ac44-ce6e-407f-a37f-77f9afb60819-kube-api-access-jnsdg\") pod \"maas-api-revoke-sub-d7ba433a-b5pwq\" (UID: \"3807ac44-ce6e-407f-a37f-77f9afb60819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7ba433a-b5pwq" Nov 03 02:30:34.727575 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.727551 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:30:34.799415 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.799387 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnsdg\" (UniqueName: \"kubernetes.io/projected/3807ac44-ce6e-407f-a37f-77f9afb60819-kube-api-access-jnsdg\") pod \"maas-api-revoke-sub-d7ba433a-b5pwq\" (UID: \"3807ac44-ce6e-407f-a37f-77f9afb60819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7ba433a-b5pwq" Nov 03 02:30:34.799566 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.799495 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3807ac44-ce6e-407f-a37f-77f9afb60819-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7ba433a-b5pwq\" (UID: \"3807ac44-ce6e-407f-a37f-77f9afb60819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7ba433a-b5pwq" Nov 03 02:30:34.799566 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.799531 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3807ac44-ce6e-407f-a37f-77f9afb60819-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7ba433a-b5pwq\" (UID: \"3807ac44-ce6e-407f-a37f-77f9afb60819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7ba433a-b5pwq" Nov 03 02:30:34.800161 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.800136 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3807ac44-ce6e-407f-a37f-77f9afb60819-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7ba433a-b5pwq\" (UID: \"3807ac44-ce6e-407f-a37f-77f9afb60819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7ba433a-b5pwq" Nov 03 02:30:34.801821 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.801804 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3807ac44-ce6e-407f-a37f-77f9afb60819-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7ba433a-b5pwq\" (UID: \"3807ac44-ce6e-407f-a37f-77f9afb60819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7ba433a-b5pwq" Nov 03 02:30:34.850080 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.850042 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bd56b15-0fc7-4a81-a569-4eba9ffaf645" path="/var/lib/kubelet/pods/4bd56b15-0fc7-4a81-a569-4eba9ffaf645/volumes" Nov 03 02:30:34.852520 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.852490 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnsdg\" (UniqueName: \"kubernetes.io/projected/3807ac44-ce6e-407f-a37f-77f9afb60819-kube-api-access-jnsdg\") pod \"maas-api-revoke-sub-d7ba433a-b5pwq\" (UID: \"3807ac44-ce6e-407f-a37f-77f9afb60819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7ba433a-b5pwq" Nov 03 02:30:34.945548 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.945460 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7ba433a-b5pwq" Nov 03 02:30:34.954698 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.954668 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-plhfr"] Nov 03 02:30:34.954896 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:34.954871 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-plhfr" podUID="462c5bfd-f256-4620-87b3-20ed3c1fd9e4" containerName="discovery" containerID="cri-o://12831d5535f775f63c8058daa29e761266df8b3de540ebc9ae6805d8c76d855f" gracePeriod=30 Nov 03 02:30:35.038483 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:35.038448 5605 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-plhfr container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.25:8080/ready\": dial tcp 10.134.0.25:8080: connect: connection refused" start-of-body= Nov 03 02:30:35.038644 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:35.038507 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-plhfr" podUID="462c5bfd-f256-4620-87b3-20ed3c1fd9e4" containerName="discovery" probeResult="failure" output="Get \"http://10.134.0.25:8080/ready\": dial tcp 10.134.0.25:8080: connect: connection refused" Nov 03 02:30:35.200406 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:35.200332 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-plhfr" Nov 03 02:30:35.305058 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:35.305030 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/462c5bfd-f256-4620-87b3-20ed3c1fd9e4-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/462c5bfd-f256-4620-87b3-20ed3c1fd9e4-istio-kubeconfig\") pod \"462c5bfd-f256-4620-87b3-20ed3c1fd9e4\" (UID: \"462c5bfd-f256-4620-87b3-20ed3c1fd9e4\") " Nov 03 02:30:35.305229 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:35.305092 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/462c5bfd-f256-4620-87b3-20ed3c1fd9e4-kube-api-access-zb8tt\" (UniqueName: \"kubernetes.io/projected/462c5bfd-f256-4620-87b3-20ed3c1fd9e4-kube-api-access-zb8tt\") pod \"462c5bfd-f256-4620-87b3-20ed3c1fd9e4\" (UID: \"462c5bfd-f256-4620-87b3-20ed3c1fd9e4\") " Nov 03 02:30:35.305229 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:35.305134 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/462c5bfd-f256-4620-87b3-20ed3c1fd9e4-istio-token\" (UniqueName: \"kubernetes.io/projected/462c5bfd-f256-4620-87b3-20ed3c1fd9e4-istio-token\") pod \"462c5bfd-f256-4620-87b3-20ed3c1fd9e4\" (UID: \"462c5bfd-f256-4620-87b3-20ed3c1fd9e4\") " Nov 03 02:30:35.305229 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:35.305182 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/462c5bfd-f256-4620-87b3-20ed3c1fd9e4-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/462c5bfd-f256-4620-87b3-20ed3c1fd9e4-istio-csr-ca-configmap\") pod \"462c5bfd-f256-4620-87b3-20ed3c1fd9e4\" (UID: \"462c5bfd-f256-4620-87b3-20ed3c1fd9e4\") " Nov 03 02:30:35.305229 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:35.305208 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/462c5bfd-f256-4620-87b3-20ed3c1fd9e4-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/462c5bfd-f256-4620-87b3-20ed3c1fd9e4-istio-csr-dns-cert\") pod \"462c5bfd-f256-4620-87b3-20ed3c1fd9e4\" (UID: \"462c5bfd-f256-4620-87b3-20ed3c1fd9e4\") " Nov 03 02:30:35.305471 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:35.305239 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/462c5bfd-f256-4620-87b3-20ed3c1fd9e4-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/462c5bfd-f256-4620-87b3-20ed3c1fd9e4-local-certs\") pod \"462c5bfd-f256-4620-87b3-20ed3c1fd9e4\" (UID: \"462c5bfd-f256-4620-87b3-20ed3c1fd9e4\") " Nov 03 02:30:35.305471 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:35.305312 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/462c5bfd-f256-4620-87b3-20ed3c1fd9e4-cacerts\" (UniqueName: \"kubernetes.io/secret/462c5bfd-f256-4620-87b3-20ed3c1fd9e4-cacerts\") pod \"462c5bfd-f256-4620-87b3-20ed3c1fd9e4\" (UID: \"462c5bfd-f256-4620-87b3-20ed3c1fd9e4\") " Nov 03 02:30:35.305841 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:35.305812 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/462c5bfd-f256-4620-87b3-20ed3c1fd9e4-istio-csr-ca-configmap" pod "462c5bfd-f256-4620-87b3-20ed3c1fd9e4" (UID: "462c5bfd-f256-4620-87b3-20ed3c1fd9e4"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:30:35.307441 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:35.307409 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/462c5bfd-f256-4620-87b3-20ed3c1fd9e4-local-certs" pod "462c5bfd-f256-4620-87b3-20ed3c1fd9e4" (UID: "462c5bfd-f256-4620-87b3-20ed3c1fd9e4"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:30:35.307776 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:35.307754 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/462c5bfd-f256-4620-87b3-20ed3c1fd9e4-istio-kubeconfig" pod "462c5bfd-f256-4620-87b3-20ed3c1fd9e4" (UID: "462c5bfd-f256-4620-87b3-20ed3c1fd9e4"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:30:35.307843 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:35.307819 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/462c5bfd-f256-4620-87b3-20ed3c1fd9e4-kube-api-access-zb8tt" pod "462c5bfd-f256-4620-87b3-20ed3c1fd9e4" (UID: "462c5bfd-f256-4620-87b3-20ed3c1fd9e4"). InnerVolumeSpecName "kube-api-access-zb8tt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:30:35.307941 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:35.307911 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/462c5bfd-f256-4620-87b3-20ed3c1fd9e4-istio-token" pod "462c5bfd-f256-4620-87b3-20ed3c1fd9e4" (UID: "462c5bfd-f256-4620-87b3-20ed3c1fd9e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:30:35.308003 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:35.307982 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/462c5bfd-f256-4620-87b3-20ed3c1fd9e4-istio-csr-dns-cert" pod "462c5bfd-f256-4620-87b3-20ed3c1fd9e4" (UID: "462c5bfd-f256-4620-87b3-20ed3c1fd9e4"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:30:35.308003 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:35.307995 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/462c5bfd-f256-4620-87b3-20ed3c1fd9e4-cacerts" pod "462c5bfd-f256-4620-87b3-20ed3c1fd9e4" (UID: "462c5bfd-f256-4620-87b3-20ed3c1fd9e4"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:30:35.323712 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:35.323684 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7ba433a-b5pwq"] Nov 03 02:30:35.331468 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:30:35.331426 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3807ac44_ce6e_407f_a37f_77f9afb60819.slice/crio-e5d8209aa0f6ca6b44f1a9322481d01c49e7538c3790ad1ebd043f0bbcf5825f WatchSource:0}: Error finding container e5d8209aa0f6ca6b44f1a9322481d01c49e7538c3790ad1ebd043f0bbcf5825f: Status 404 returned error can't find the container with id e5d8209aa0f6ca6b44f1a9322481d01c49e7538c3790ad1ebd043f0bbcf5825f Nov 03 02:30:35.350321 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:35.350296 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7ba433a-b5pwq" event={"ID":"3807ac44-ce6e-407f-a37f-77f9afb60819","Type":"ContainerStarted","Data":"e5d8209aa0f6ca6b44f1a9322481d01c49e7538c3790ad1ebd043f0bbcf5825f"} Nov 03 02:30:35.351925 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:35.351902 5605 generic.go:356] "Generic (PLEG): container finished" podID="462c5bfd-f256-4620-87b3-20ed3c1fd9e4" containerID="12831d5535f775f63c8058daa29e761266df8b3de540ebc9ae6805d8c76d855f" exitCode=0 Nov 03 02:30:35.352017 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:35.351950 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-plhfr" Nov 03 02:30:35.352017 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:35.351965 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-plhfr" event={"ID":"462c5bfd-f256-4620-87b3-20ed3c1fd9e4","Type":"ContainerDied","Data":"12831d5535f775f63c8058daa29e761266df8b3de540ebc9ae6805d8c76d855f"} Nov 03 02:30:35.352017 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:35.351997 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-plhfr" event={"ID":"462c5bfd-f256-4620-87b3-20ed3c1fd9e4","Type":"ContainerDied","Data":"fe90232c414aeac8da3b46fbdeadde9397629b3d8a7d79ab8c3793cecdf92cc7"} Nov 03 02:30:35.352017 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:35.352014 5605 scope.go:122] "RemoveContainer" containerID="12831d5535f775f63c8058daa29e761266df8b3de540ebc9ae6805d8c76d855f" Nov 03 02:30:35.367738 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:35.367721 5605 scope.go:122] "RemoveContainer" containerID="12831d5535f775f63c8058daa29e761266df8b3de540ebc9ae6805d8c76d855f" Nov 03 02:30:35.367991 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:30:35.367972 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12831d5535f775f63c8058daa29e761266df8b3de540ebc9ae6805d8c76d855f\": container with ID starting with 12831d5535f775f63c8058daa29e761266df8b3de540ebc9ae6805d8c76d855f not found: ID does not exist" containerID="12831d5535f775f63c8058daa29e761266df8b3de540ebc9ae6805d8c76d855f" Nov 03 02:30:35.368049 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:35.368001 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12831d5535f775f63c8058daa29e761266df8b3de540ebc9ae6805d8c76d855f"} err="failed to get container status \"12831d5535f775f63c8058daa29e761266df8b3de540ebc9ae6805d8c76d855f\": rpc error: code = NotFound desc = could not find container \"12831d5535f775f63c8058daa29e761266df8b3de540ebc9ae6805d8c76d855f\": container with ID starting with 12831d5535f775f63c8058daa29e761266df8b3de540ebc9ae6805d8c76d855f not found: ID does not exist" Nov 03 02:30:35.406679 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:35.406658 5605 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/462c5bfd-f256-4620-87b3-20ed3c1fd9e4-cacerts\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:35.406679 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:35.406680 5605 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/462c5bfd-f256-4620-87b3-20ed3c1fd9e4-istio-kubeconfig\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:35.406810 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:35.406690 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb8tt\" (UniqueName: \"kubernetes.io/projected/462c5bfd-f256-4620-87b3-20ed3c1fd9e4-kube-api-access-zb8tt\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:35.406810 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:35.406699 5605 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/462c5bfd-f256-4620-87b3-20ed3c1fd9e4-istio-token\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:35.406810 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:35.406708 5605 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/462c5bfd-f256-4620-87b3-20ed3c1fd9e4-istio-csr-ca-configmap\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:35.406810 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:35.406717 5605 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/462c5bfd-f256-4620-87b3-20ed3c1fd9e4-istio-csr-dns-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:35.406810 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:35.406725 5605 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/462c5bfd-f256-4620-87b3-20ed3c1fd9e4-local-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:35.505527 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:35.505428 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-plhfr"] Nov 03 02:30:35.507739 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:35.507710 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-plhfr"] Nov 03 02:30:36.359988 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:36.359950 5605 generic.go:356] "Generic (PLEG): container finished" podID="3807ac44-ce6e-407f-a37f-77f9afb60819" containerID="e719f55515439e7fec00322737013fb550ecacb400b1df8192200222303a55ab" exitCode=0 Nov 03 02:30:36.360403 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:36.360028 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7ba433a-b5pwq" event={"ID":"3807ac44-ce6e-407f-a37f-77f9afb60819","Type":"ContainerDied","Data":"e719f55515439e7fec00322737013fb550ecacb400b1df8192200222303a55ab"} Nov 03 02:30:36.853504 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:36.853473 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="462c5bfd-f256-4620-87b3-20ed3c1fd9e4" path="/var/lib/kubelet/pods/462c5bfd-f256-4620-87b3-20ed3c1fd9e4/volumes" Nov 03 02:30:37.499648 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:37.499619 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7ba433a-b5pwq" Nov 03 02:30:37.630237 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:37.630159 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3807ac44-ce6e-407f-a37f-77f9afb60819-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3807ac44-ce6e-407f-a37f-77f9afb60819-maas-api-openshift-ca\") pod \"3807ac44-ce6e-407f-a37f-77f9afb60819\" (UID: \"3807ac44-ce6e-407f-a37f-77f9afb60819\") " Nov 03 02:30:37.630371 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:37.630318 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3807ac44-ce6e-407f-a37f-77f9afb60819-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3807ac44-ce6e-407f-a37f-77f9afb60819-maas-api-xks-ca\") pod \"3807ac44-ce6e-407f-a37f-77f9afb60819\" (UID: \"3807ac44-ce6e-407f-a37f-77f9afb60819\") " Nov 03 02:30:37.630430 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:37.630377 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3807ac44-ce6e-407f-a37f-77f9afb60819-kube-api-access-jnsdg\" (UniqueName: \"kubernetes.io/projected/3807ac44-ce6e-407f-a37f-77f9afb60819-kube-api-access-jnsdg\") pod \"3807ac44-ce6e-407f-a37f-77f9afb60819\" (UID: \"3807ac44-ce6e-407f-a37f-77f9afb60819\") " Nov 03 02:30:37.630600 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:37.630571 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3807ac44-ce6e-407f-a37f-77f9afb60819-maas-api-openshift-ca" pod "3807ac44-ce6e-407f-a37f-77f9afb60819" (UID: "3807ac44-ce6e-407f-a37f-77f9afb60819"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:30:37.630874 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:37.630850 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3807ac44-ce6e-407f-a37f-77f9afb60819-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:37.632795 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:37.632767 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3807ac44-ce6e-407f-a37f-77f9afb60819-kube-api-access-jnsdg" pod "3807ac44-ce6e-407f-a37f-77f9afb60819" (UID: "3807ac44-ce6e-407f-a37f-77f9afb60819"). InnerVolumeSpecName "kube-api-access-jnsdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:30:37.636531 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:37.636505 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3807ac44-ce6e-407f-a37f-77f9afb60819-maas-api-xks-ca" pod "3807ac44-ce6e-407f-a37f-77f9afb60819" (UID: "3807ac44-ce6e-407f-a37f-77f9afb60819"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:30:37.732364 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:37.732338 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3807ac44-ce6e-407f-a37f-77f9afb60819-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:37.732457 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:37.732379 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnsdg\" (UniqueName: \"kubernetes.io/projected/3807ac44-ce6e-407f-a37f-77f9afb60819-kube-api-access-jnsdg\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:38.372750 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:38.372715 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7ba433a-b5pwq" event={"ID":"3807ac44-ce6e-407f-a37f-77f9afb60819","Type":"ContainerDied","Data":"e5d8209aa0f6ca6b44f1a9322481d01c49e7538c3790ad1ebd043f0bbcf5825f"} Nov 03 02:30:38.372750 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:38.372738 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7ba433a-b5pwq" Nov 03 02:30:38.372930 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:38.372764 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5d8209aa0f6ca6b44f1a9322481d01c49e7538c3790ad1ebd043f0bbcf5825f" Nov 03 02:30:45.463042 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:45.463005 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bdc2839-dhzqw"] Nov 03 02:30:45.463473 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:45.463419 5605 cpu_manager.go:414] "containerMap: removing container" podUID="433647e9-f97a-470d-bd48-cf2c6f9d6064" containerName="cleanup" Nov 03 02:30:45.463473 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:45.463431 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="433647e9-f97a-470d-bd48-cf2c6f9d6064" containerName="cleanup" Nov 03 02:30:45.463473 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:45.463444 5605 cpu_manager.go:414] "containerMap: removing container" podUID="a9cd06a2-1beb-4d2b-9561-4e4a3b788e4d" containerName="cleanup" Nov 03 02:30:45.463473 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:45.463448 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9cd06a2-1beb-4d2b-9561-4e4a3b788e4d" containerName="cleanup" Nov 03 02:30:45.463473 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:45.463464 5605 cpu_manager.go:414] "containerMap: removing container" podUID="3bb55cd1-e726-45c4-b1fd-b03229ec0378" containerName="cleanup" Nov 03 02:30:45.463473 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:45.463470 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bb55cd1-e726-45c4-b1fd-b03229ec0378" containerName="cleanup" Nov 03 02:30:45.463676 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:45.463483 5605 cpu_manager.go:414] "containerMap: removing container" podUID="3807ac44-ce6e-407f-a37f-77f9afb60819" containerName="revoke-keys" Nov 03 02:30:45.463676 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:45.463489 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3807ac44-ce6e-407f-a37f-77f9afb60819" containerName="revoke-keys" Nov 03 02:30:45.463676 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:45.463499 5605 cpu_manager.go:414] "containerMap: removing container" podUID="462c5bfd-f256-4620-87b3-20ed3c1fd9e4" containerName="discovery" Nov 03 02:30:45.463676 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:45.463504 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="462c5bfd-f256-4620-87b3-20ed3c1fd9e4" containerName="discovery" Nov 03 02:30:45.463676 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:45.463513 5605 cpu_manager.go:414] "containerMap: removing container" podUID="963f49af-b49b-4664-a9a5-57937449e0df" containerName="cleanup" Nov 03 02:30:45.463676 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:45.463518 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="963f49af-b49b-4664-a9a5-57937449e0df" containerName="cleanup" Nov 03 02:30:45.463676 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:45.463575 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="963f49af-b49b-4664-a9a5-57937449e0df" containerName="cleanup" Nov 03 02:30:45.463676 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:45.463584 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="25790141-f19b-4e73-8b61-4c82f33a1825" containerName="cleanup" Nov 03 02:30:45.463676 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:45.463594 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="3807ac44-ce6e-407f-a37f-77f9afb60819" containerName="revoke-keys" Nov 03 02:30:45.463676 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:45.463601 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="433647e9-f97a-470d-bd48-cf2c6f9d6064" containerName="cleanup" Nov 03 02:30:45.463676 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:45.463606 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="462c5bfd-f256-4620-87b3-20ed3c1fd9e4" containerName="discovery" Nov 03 02:30:45.463676 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:45.463612 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="6eff793f-67d6-4044-87fb-ca5e46a5e6ac" containerName="cleanup" Nov 03 02:30:45.466421 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:45.466399 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdc2839-dhzqw" Nov 03 02:30:45.517013 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:45.516983 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phsfg\" (UniqueName: \"kubernetes.io/projected/ed4768a9-da35-463c-8001-4f8ed903a7cb-kube-api-access-phsfg\") pod \"maas-api-revoke-sub-4bdc2839-dhzqw\" (UID: \"ed4768a9-da35-463c-8001-4f8ed903a7cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdc2839-dhzqw" Nov 03 02:30:45.517121 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:45.517019 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed4768a9-da35-463c-8001-4f8ed903a7cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4bdc2839-dhzqw\" (UID: \"ed4768a9-da35-463c-8001-4f8ed903a7cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdc2839-dhzqw" Nov 03 02:30:45.517163 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:45.517145 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed4768a9-da35-463c-8001-4f8ed903a7cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4bdc2839-dhzqw\" (UID: \"ed4768a9-da35-463c-8001-4f8ed903a7cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdc2839-dhzqw" Nov 03 02:30:45.519067 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:45.519045 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:30:45.521582 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:45.521561 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bdc2839-dhzqw"] Nov 03 02:30:45.565507 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:45.565481 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:30:45.565620 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:45.565542 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:30:45.565620 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:45.565554 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:30:45.618752 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:45.618702 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed4768a9-da35-463c-8001-4f8ed903a7cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4bdc2839-dhzqw\" (UID: \"ed4768a9-da35-463c-8001-4f8ed903a7cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdc2839-dhzqw" Nov 03 02:30:45.618983 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:45.618854 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phsfg\" (UniqueName: \"kubernetes.io/projected/ed4768a9-da35-463c-8001-4f8ed903a7cb-kube-api-access-phsfg\") pod \"maas-api-revoke-sub-4bdc2839-dhzqw\" (UID: \"ed4768a9-da35-463c-8001-4f8ed903a7cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdc2839-dhzqw" Nov 03 02:30:45.618983 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:45.618888 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed4768a9-da35-463c-8001-4f8ed903a7cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4bdc2839-dhzqw\" (UID: \"ed4768a9-da35-463c-8001-4f8ed903a7cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdc2839-dhzqw" Nov 03 02:30:45.619447 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:45.619418 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed4768a9-da35-463c-8001-4f8ed903a7cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4bdc2839-dhzqw\" (UID: \"ed4768a9-da35-463c-8001-4f8ed903a7cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdc2839-dhzqw" Nov 03 02:30:45.621471 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:45.621452 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed4768a9-da35-463c-8001-4f8ed903a7cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4bdc2839-dhzqw\" (UID: \"ed4768a9-da35-463c-8001-4f8ed903a7cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdc2839-dhzqw" Nov 03 02:30:45.672255 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:45.672222 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phsfg\" (UniqueName: \"kubernetes.io/projected/ed4768a9-da35-463c-8001-4f8ed903a7cb-kube-api-access-phsfg\") pod \"maas-api-revoke-sub-4bdc2839-dhzqw\" (UID: \"ed4768a9-da35-463c-8001-4f8ed903a7cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdc2839-dhzqw" Nov 03 02:30:45.779658 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:45.779577 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdc2839-dhzqw" Nov 03 02:30:46.145642 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:46.145609 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bdc2839-dhzqw"] Nov 03 02:30:46.152012 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:30:46.151980 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded4768a9_da35_463c_8001_4f8ed903a7cb.slice/crio-ebf092a9eb4e0415aedb0a9563f24c69ea384b24a52aa98f8b8c48a7ebf20a0c WatchSource:0}: Error finding container ebf092a9eb4e0415aedb0a9563f24c69ea384b24a52aa98f8b8c48a7ebf20a0c: Status 404 returned error can't find the container with id ebf092a9eb4e0415aedb0a9563f24c69ea384b24a52aa98f8b8c48a7ebf20a0c Nov 03 02:30:46.415816 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:46.415727 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdc2839-dhzqw" event={"ID":"ed4768a9-da35-463c-8001-4f8ed903a7cb","Type":"ContainerStarted","Data":"ebf092a9eb4e0415aedb0a9563f24c69ea384b24a52aa98f8b8c48a7ebf20a0c"} Nov 03 02:30:46.860445 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:46.860413 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:30:47.424026 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:47.423991 5605 generic.go:356] "Generic (PLEG): container finished" podID="ed4768a9-da35-463c-8001-4f8ed903a7cb" containerID="bdf5054506d4bed72e674803dae3525dea06367732baf478c4ac56cb392c73a2" exitCode=0 Nov 03 02:30:47.424251 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:47.424068 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdc2839-dhzqw" event={"ID":"ed4768a9-da35-463c-8001-4f8ed903a7cb","Type":"ContainerDied","Data":"bdf5054506d4bed72e674803dae3525dea06367732baf478c4ac56cb392c73a2"} Nov 03 02:30:48.567969 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:48.567937 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdc2839-dhzqw" Nov 03 02:30:48.663053 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:48.663014 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed4768a9-da35-463c-8001-4f8ed903a7cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed4768a9-da35-463c-8001-4f8ed903a7cb-maas-api-xks-ca\") pod \"ed4768a9-da35-463c-8001-4f8ed903a7cb\" (UID: \"ed4768a9-da35-463c-8001-4f8ed903a7cb\") " Nov 03 02:30:48.663226 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:48.663106 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed4768a9-da35-463c-8001-4f8ed903a7cb-kube-api-access-phsfg\" (UniqueName: \"kubernetes.io/projected/ed4768a9-da35-463c-8001-4f8ed903a7cb-kube-api-access-phsfg\") pod \"ed4768a9-da35-463c-8001-4f8ed903a7cb\" (UID: \"ed4768a9-da35-463c-8001-4f8ed903a7cb\") " Nov 03 02:30:48.663226 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:48.663198 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed4768a9-da35-463c-8001-4f8ed903a7cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed4768a9-da35-463c-8001-4f8ed903a7cb-maas-api-openshift-ca\") pod \"ed4768a9-da35-463c-8001-4f8ed903a7cb\" (UID: \"ed4768a9-da35-463c-8001-4f8ed903a7cb\") " Nov 03 02:30:48.663670 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:48.663633 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed4768a9-da35-463c-8001-4f8ed903a7cb-maas-api-openshift-ca" pod "ed4768a9-da35-463c-8001-4f8ed903a7cb" (UID: "ed4768a9-da35-463c-8001-4f8ed903a7cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:30:48.665644 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:48.665615 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed4768a9-da35-463c-8001-4f8ed903a7cb-maas-api-xks-ca" pod "ed4768a9-da35-463c-8001-4f8ed903a7cb" (UID: "ed4768a9-da35-463c-8001-4f8ed903a7cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:30:48.665900 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:48.665873 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed4768a9-da35-463c-8001-4f8ed903a7cb-kube-api-access-phsfg" pod "ed4768a9-da35-463c-8001-4f8ed903a7cb" (UID: "ed4768a9-da35-463c-8001-4f8ed903a7cb"). InnerVolumeSpecName "kube-api-access-phsfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:30:48.764522 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:48.764430 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed4768a9-da35-463c-8001-4f8ed903a7cb-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:48.764522 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:48.764462 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed4768a9-da35-463c-8001-4f8ed903a7cb-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:48.764522 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:48.764482 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phsfg\" (UniqueName: \"kubernetes.io/projected/ed4768a9-da35-463c-8001-4f8ed903a7cb-kube-api-access-phsfg\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:49.436380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:49.436335 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdc2839-dhzqw" Nov 03 02:30:49.436556 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:49.436331 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdc2839-dhzqw" event={"ID":"ed4768a9-da35-463c-8001-4f8ed903a7cb","Type":"ContainerDied","Data":"ebf092a9eb4e0415aedb0a9563f24c69ea384b24a52aa98f8b8c48a7ebf20a0c"} Nov 03 02:30:49.436556 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:49.436504 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ebf092a9eb4e0415aedb0a9563f24c69ea384b24a52aa98f8b8c48a7ebf20a0c" Nov 03 02:30:53.914424 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:30:53.914379 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:30:55.345137 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:55.345100 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8"] Nov 03 02:30:55.353700 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:55.353668 5605 cpu_manager.go:414] "containerMap: removing container" podUID="ed4768a9-da35-463c-8001-4f8ed903a7cb" containerName="revoke-keys" Nov 03 02:30:55.353700 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:55.353694 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed4768a9-da35-463c-8001-4f8ed903a7cb" containerName="revoke-keys" Nov 03 02:30:55.353933 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:55.353809 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed4768a9-da35-463c-8001-4f8ed903a7cb" containerName="revoke-keys" Nov 03 02:30:55.357201 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:55.357175 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8" Nov 03 02:30:55.391485 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:55.391461 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn"] Nov 03 02:30:55.394753 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:55.394739 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn" Nov 03 02:30:55.405899 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:55.405878 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-f9bff9-openshift-default-dockercfg-nn2lt\"" Nov 03 02:30:55.434371 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:55.434327 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb319f5e-21f0-486e-b6c0-fe44875c545a-istiod-ca-cert\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8\" (UID: \"cb319f5e-21f0-486e-b6c0-fe44875c545a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8" Nov 03 02:30:55.434482 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:55.434461 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb319f5e-21f0-486e-b6c0-fe44875c545a-istio-token\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8\" (UID: \"cb319f5e-21f0-486e-b6c0-fe44875c545a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8" Nov 03 02:30:55.434532 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:55.434495 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb319f5e-21f0-486e-b6c0-fe44875c545a-workload-certs\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8\" (UID: \"cb319f5e-21f0-486e-b6c0-fe44875c545a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8" Nov 03 02:30:55.434581 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:55.434536 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb319f5e-21f0-486e-b6c0-fe44875c545a-istio-podinfo\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8\" (UID: \"cb319f5e-21f0-486e-b6c0-fe44875c545a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8" Nov 03 02:30:55.434663 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:55.434642 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb319f5e-21f0-486e-b6c0-fe44875c545a-workload-socket\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8\" (UID: \"cb319f5e-21f0-486e-b6c0-fe44875c545a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8" Nov 03 02:30:55.434700 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:55.434690 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb319f5e-21f0-486e-b6c0-fe44875c545a-credential-socket\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8\" (UID: \"cb319f5e-21f0-486e-b6c0-fe44875c545a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8" Nov 03 02:30:55.434751 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:55.434721 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb319f5e-21f0-486e-b6c0-fe44875c545a-istio-data\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8\" (UID: \"cb319f5e-21f0-486e-b6c0-fe44875c545a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8" Nov 03 02:30:55.434803 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:55.434775 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhc4f\" (UniqueName: \"kubernetes.io/projected/cb319f5e-21f0-486e-b6c0-fe44875c545a-kube-api-access-dhc4f\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8\" (UID: \"cb319f5e-21f0-486e-b6c0-fe44875c545a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8" Nov 03 02:30:55.434846 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:55.434834 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb319f5e-21f0-486e-b6c0-fe44875c545a-istio-envoy\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8\" (UID: \"cb319f5e-21f0-486e-b6c0-fe44875c545a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8" Nov 03 02:30:55.444219 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:55.444196 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8"] Nov 03 02:30:55.493042 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:55.493004 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn"] Nov 03 02:30:55.535602 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:55.535575 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce11384-8e67-4aef-80ab-3ddd74099bf7-credential-socket\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn\" (UID: \"dce11384-8e67-4aef-80ab-3ddd74099bf7\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn" Nov 03 02:30:55.535754 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:55.535610 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dce11384-8e67-4aef-80ab-3ddd74099bf7-istiod-ca-cert\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn\" (UID: \"dce11384-8e67-4aef-80ab-3ddd74099bf7\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn" Nov 03 02:30:55.535754 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:55.535647 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb319f5e-21f0-486e-b6c0-fe44875c545a-istio-podinfo\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8\" (UID: \"cb319f5e-21f0-486e-b6c0-fe44875c545a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8" Nov 03 02:30:55.535754 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:55.535716 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmbt8\" (UniqueName: \"kubernetes.io/projected/dce11384-8e67-4aef-80ab-3ddd74099bf7-kube-api-access-jmbt8\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn\" (UID: \"dce11384-8e67-4aef-80ab-3ddd74099bf7\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn" Nov 03 02:30:55.535928 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:55.535777 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb319f5e-21f0-486e-b6c0-fe44875c545a-workload-socket\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8\" (UID: \"cb319f5e-21f0-486e-b6c0-fe44875c545a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8" Nov 03 02:30:55.535928 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:55.535799 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb319f5e-21f0-486e-b6c0-fe44875c545a-credential-socket\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8\" (UID: \"cb319f5e-21f0-486e-b6c0-fe44875c545a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8" Nov 03 02:30:55.535928 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:55.535821 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb319f5e-21f0-486e-b6c0-fe44875c545a-istio-data\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8\" (UID: \"cb319f5e-21f0-486e-b6c0-fe44875c545a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8" Nov 03 02:30:55.535928 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:55.535837 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhc4f\" (UniqueName: \"kubernetes.io/projected/cb319f5e-21f0-486e-b6c0-fe44875c545a-kube-api-access-dhc4f\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8\" (UID: \"cb319f5e-21f0-486e-b6c0-fe44875c545a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8" Nov 03 02:30:55.535928 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:55.535856 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce11384-8e67-4aef-80ab-3ddd74099bf7-workload-socket\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn\" (UID: \"dce11384-8e67-4aef-80ab-3ddd74099bf7\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn" Nov 03 02:30:55.535928 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:55.535886 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dce11384-8e67-4aef-80ab-3ddd74099bf7-istio-data\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn\" (UID: \"dce11384-8e67-4aef-80ab-3ddd74099bf7\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn" Nov 03 02:30:55.535928 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:55.535911 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dce11384-8e67-4aef-80ab-3ddd74099bf7-istio-envoy\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn\" (UID: \"dce11384-8e67-4aef-80ab-3ddd74099bf7\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn" Nov 03 02:30:55.536240 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:55.535939 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dce11384-8e67-4aef-80ab-3ddd74099bf7-workload-certs\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn\" (UID: \"dce11384-8e67-4aef-80ab-3ddd74099bf7\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn" Nov 03 02:30:55.536240 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:55.536007 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dce11384-8e67-4aef-80ab-3ddd74099bf7-istio-token\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn\" (UID: \"dce11384-8e67-4aef-80ab-3ddd74099bf7\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn" Nov 03 02:30:55.536240 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:55.536129 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb319f5e-21f0-486e-b6c0-fe44875c545a-istio-envoy\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8\" (UID: \"cb319f5e-21f0-486e-b6c0-fe44875c545a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8" Nov 03 02:30:55.536240 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:55.536171 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb319f5e-21f0-486e-b6c0-fe44875c545a-istiod-ca-cert\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8\" (UID: \"cb319f5e-21f0-486e-b6c0-fe44875c545a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8" Nov 03 02:30:55.536240 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:55.536220 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb319f5e-21f0-486e-b6c0-fe44875c545a-workload-socket\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8\" (UID: \"cb319f5e-21f0-486e-b6c0-fe44875c545a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8" Nov 03 02:30:55.536481 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:55.536241 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb319f5e-21f0-486e-b6c0-fe44875c545a-istio-data\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8\" (UID: \"cb319f5e-21f0-486e-b6c0-fe44875c545a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8" Nov 03 02:30:55.536481 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:55.536309 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb319f5e-21f0-486e-b6c0-fe44875c545a-credential-socket\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8\" (UID: \"cb319f5e-21f0-486e-b6c0-fe44875c545a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8" Nov 03 02:30:55.536481 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:55.536333 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dce11384-8e67-4aef-80ab-3ddd74099bf7-istio-podinfo\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn\" (UID: \"dce11384-8e67-4aef-80ab-3ddd74099bf7\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn" Nov 03 02:30:55.536481 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:55.536421 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb319f5e-21f0-486e-b6c0-fe44875c545a-istio-token\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8\" (UID: \"cb319f5e-21f0-486e-b6c0-fe44875c545a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8" Nov 03 02:30:55.536481 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:55.536454 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb319f5e-21f0-486e-b6c0-fe44875c545a-workload-certs\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8\" (UID: \"cb319f5e-21f0-486e-b6c0-fe44875c545a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8" Nov 03 02:30:55.536787 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:55.536769 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb319f5e-21f0-486e-b6c0-fe44875c545a-workload-certs\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8\" (UID: \"cb319f5e-21f0-486e-b6c0-fe44875c545a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8" Nov 03 02:30:55.536848 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:55.536814 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb319f5e-21f0-486e-b6c0-fe44875c545a-istiod-ca-cert\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8\" (UID: \"cb319f5e-21f0-486e-b6c0-fe44875c545a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8" Nov 03 02:30:55.538309 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:55.538281 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb319f5e-21f0-486e-b6c0-fe44875c545a-istio-podinfo\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8\" (UID: \"cb319f5e-21f0-486e-b6c0-fe44875c545a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8" Nov 03 02:30:55.538434 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:55.538315 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb319f5e-21f0-486e-b6c0-fe44875c545a-istio-envoy\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8\" (UID: \"cb319f5e-21f0-486e-b6c0-fe44875c545a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8" Nov 03 02:30:55.589002 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:55.588975 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb319f5e-21f0-486e-b6c0-fe44875c545a-istio-token\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8\" (UID: \"cb319f5e-21f0-486e-b6c0-fe44875c545a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8" Nov 03 02:30:55.589211 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:55.589189 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhc4f\" (UniqueName: \"kubernetes.io/projected/cb319f5e-21f0-486e-b6c0-fe44875c545a-kube-api-access-dhc4f\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8\" (UID: \"cb319f5e-21f0-486e-b6c0-fe44875c545a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8" Nov 03 02:30:55.637426 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:55.637330 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dce11384-8e67-4aef-80ab-3ddd74099bf7-istio-podinfo\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn\" (UID: \"dce11384-8e67-4aef-80ab-3ddd74099bf7\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn" Nov 03 02:30:55.637426 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:55.637406 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce11384-8e67-4aef-80ab-3ddd74099bf7-credential-socket\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn\" (UID: \"dce11384-8e67-4aef-80ab-3ddd74099bf7\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn" Nov 03 02:30:55.637601 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:55.637429 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dce11384-8e67-4aef-80ab-3ddd74099bf7-istiod-ca-cert\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn\" (UID: \"dce11384-8e67-4aef-80ab-3ddd74099bf7\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn" Nov 03 02:30:55.637601 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:55.637468 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmbt8\" (UniqueName: \"kubernetes.io/projected/dce11384-8e67-4aef-80ab-3ddd74099bf7-kube-api-access-jmbt8\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn\" (UID: \"dce11384-8e67-4aef-80ab-3ddd74099bf7\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn" Nov 03 02:30:55.637601 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:55.637504 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce11384-8e67-4aef-80ab-3ddd74099bf7-workload-socket\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn\" (UID: \"dce11384-8e67-4aef-80ab-3ddd74099bf7\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn" Nov 03 02:30:55.637601 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:55.637532 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dce11384-8e67-4aef-80ab-3ddd74099bf7-istio-data\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn\" (UID: \"dce11384-8e67-4aef-80ab-3ddd74099bf7\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn" Nov 03 02:30:55.637601 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:55.637550 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dce11384-8e67-4aef-80ab-3ddd74099bf7-istio-envoy\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn\" (UID: \"dce11384-8e67-4aef-80ab-3ddd74099bf7\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn" Nov 03 02:30:55.637601 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:55.637566 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dce11384-8e67-4aef-80ab-3ddd74099bf7-workload-certs\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn\" (UID: \"dce11384-8e67-4aef-80ab-3ddd74099bf7\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn" Nov 03 02:30:55.637601 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:55.637593 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dce11384-8e67-4aef-80ab-3ddd74099bf7-istio-token\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn\" (UID: \"dce11384-8e67-4aef-80ab-3ddd74099bf7\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn" Nov 03 02:30:55.637976 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:55.637927 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce11384-8e67-4aef-80ab-3ddd74099bf7-credential-socket\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn\" (UID: \"dce11384-8e67-4aef-80ab-3ddd74099bf7\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn" Nov 03 02:30:55.638030 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:55.637965 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce11384-8e67-4aef-80ab-3ddd74099bf7-workload-socket\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn\" (UID: \"dce11384-8e67-4aef-80ab-3ddd74099bf7\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn" Nov 03 02:30:55.638030 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:55.638017 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dce11384-8e67-4aef-80ab-3ddd74099bf7-istio-data\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn\" (UID: \"dce11384-8e67-4aef-80ab-3ddd74099bf7\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn" Nov 03 02:30:55.638186 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:55.638158 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dce11384-8e67-4aef-80ab-3ddd74099bf7-workload-certs\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn\" (UID: \"dce11384-8e67-4aef-80ab-3ddd74099bf7\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn" Nov 03 02:30:55.638186 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:55.638180 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dce11384-8e67-4aef-80ab-3ddd74099bf7-istiod-ca-cert\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn\" (UID: \"dce11384-8e67-4aef-80ab-3ddd74099bf7\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn" Nov 03 02:30:55.639848 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:55.639822 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dce11384-8e67-4aef-80ab-3ddd74099bf7-istio-podinfo\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn\" (UID: \"dce11384-8e67-4aef-80ab-3ddd74099bf7\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn" Nov 03 02:30:55.639848 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:55.639834 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dce11384-8e67-4aef-80ab-3ddd74099bf7-istio-envoy\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn\" (UID: \"dce11384-8e67-4aef-80ab-3ddd74099bf7\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn" Nov 03 02:30:55.670979 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:55.670945 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8" Nov 03 02:30:55.690203 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:55.690177 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dce11384-8e67-4aef-80ab-3ddd74099bf7-istio-token\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn\" (UID: \"dce11384-8e67-4aef-80ab-3ddd74099bf7\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn" Nov 03 02:30:55.690604 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:55.690580 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmbt8\" (UniqueName: \"kubernetes.io/projected/dce11384-8e67-4aef-80ab-3ddd74099bf7-kube-api-access-jmbt8\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn\" (UID: \"dce11384-8e67-4aef-80ab-3ddd74099bf7\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn" Nov 03 02:30:55.707347 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:55.707324 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn" Nov 03 02:30:56.090126 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:56.090092 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8"] Nov 03 02:30:56.093241 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:56.093150 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn"] Nov 03 02:30:56.096074 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:30:56.096049 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb319f5e_21f0_486e_b6c0_fe44875c545a.slice/crio-324611e88fd2a33c2e3f6eb5c57fd417267feadb69f9ee34697dcc4f9ce72f7b WatchSource:0}: Error finding container 324611e88fd2a33c2e3f6eb5c57fd417267feadb69f9ee34697dcc4f9ce72f7b: Status 404 returned error can't find the container with id 324611e88fd2a33c2e3f6eb5c57fd417267feadb69f9ee34697dcc4f9ce72f7b Nov 03 02:30:56.098373 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:56.098323 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:30:56.098465 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:56.098406 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:30:56.098465 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:56.098439 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:30:56.145989 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:30:56.145963 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce11384_8e67_4aef_80ab_3ddd74099bf7.slice/crio-fa2026ec6b02569ae04f0e8d3bfff77c96e98756657ff6fa668c8b4f61f00ab6 WatchSource:0}: Error finding container fa2026ec6b02569ae04f0e8d3bfff77c96e98756657ff6fa668c8b4f61f00ab6: Status 404 returned error can't find the container with id fa2026ec6b02569ae04f0e8d3bfff77c96e98756657ff6fa668c8b4f61f00ab6 Nov 03 02:30:56.148414 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:56.148376 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:30:56.148516 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:56.148451 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:30:56.148516 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:56.148491 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:30:56.485024 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:56.484919 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8" event={"ID":"cb319f5e-21f0-486e-b6c0-fe44875c545a","Type":"ContainerStarted","Data":"f341e6ebb3322bbc49345149b6b05d23926f97cfdff43ec8a771e2dc5e74dd73"} Nov 03 02:30:56.485024 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:56.484992 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8" event={"ID":"cb319f5e-21f0-486e-b6c0-fe44875c545a","Type":"ContainerStarted","Data":"324611e88fd2a33c2e3f6eb5c57fd417267feadb69f9ee34697dcc4f9ce72f7b"} Nov 03 02:30:56.486782 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:56.486753 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn" event={"ID":"dce11384-8e67-4aef-80ab-3ddd74099bf7","Type":"ContainerStarted","Data":"dc4083f3aa1d1b5667dd36dbd8d19f11469cd71fa3dd59e23f70c65d26ab6634"} Nov 03 02:30:56.486916 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:56.486802 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn" event={"ID":"dce11384-8e67-4aef-80ab-3ddd74099bf7","Type":"ContainerStarted","Data":"fa2026ec6b02569ae04f0e8d3bfff77c96e98756657ff6fa668c8b4f61f00ab6"} Nov 03 02:30:56.595094 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:56.595024 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8" podStartSLOduration=1.595005776 podStartE2EDuration="1.595005776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:30:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:30:56.593649397 +0000 UTC m=+2333.852847552" watchObservedRunningTime="2026-11-03 02:30:56.595005776 +0000 UTC m=+2333.854203932" Nov 03 02:30:56.675381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:56.674444 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8" Nov 03 02:30:56.706990 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:56.706923 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn" podStartSLOduration=1.706902009 podStartE2EDuration="1.706902009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:30:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:30:56.705611609 +0000 UTC m=+2333.964809755" watchObservedRunningTime="2026-11-03 02:30:56.706902009 +0000 UTC m=+2333.966100166" Nov 03 02:30:56.708170 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:56.708142 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn" Nov 03 02:30:57.681384 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:57.679488 5605 patch_prober.go:29] interesting pod/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 03 02:30:57.681384 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:57.679554 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8" podUID="cb319f5e-21f0-486e-b6c0-fe44875c545a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 03 02:30:57.709719 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:57.709671 5605 patch_prober.go:29] interesting pod/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Nov 03 02:30:57.709871 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:57.709742 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn" podUID="dce11384-8e67-4aef-80ab-3ddd74099bf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Nov 03 02:30:58.679382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:58.677620 5605 patch_prober.go:29] interesting pod/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 03 02:30:58.679382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:58.677681 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8" podUID="cb319f5e-21f0-486e-b6c0-fe44875c545a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 03 02:30:58.707777 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:58.707741 5605 patch_prober.go:29] interesting pod/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Nov 03 02:30:58.708152 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:58.707793 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn" podUID="dce11384-8e67-4aef-80ab-3ddd74099bf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Nov 03 02:30:59.672385 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:59.671836 5605 patch_prober.go:29] interesting pod/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 03 02:30:59.672385 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:59.671912 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8" podUID="cb319f5e-21f0-486e-b6c0-fe44875c545a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 03 02:30:59.710742 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:59.710705 5605 patch_prober.go:29] interesting pod/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Nov 03 02:30:59.711327 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:30:59.711295 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn" podUID="dce11384-8e67-4aef-80ab-3ddd74099bf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Nov 03 02:31:00.671792 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:00.671752 5605 patch_prober.go:29] interesting pod/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 03 02:31:00.672061 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:00.672026 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8" podUID="cb319f5e-21f0-486e-b6c0-fe44875c545a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 03 02:31:00.708697 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:00.708662 5605 patch_prober.go:29] interesting pod/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Nov 03 02:31:00.708864 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:00.708716 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn" podUID="dce11384-8e67-4aef-80ab-3ddd74099bf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Nov 03 02:31:01.675383 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:01.673839 5605 patch_prober.go:29] interesting pod/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 03 02:31:01.675383 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:01.673914 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8" podUID="cb319f5e-21f0-486e-b6c0-fe44875c545a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 03 02:31:01.708379 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:01.708233 5605 patch_prober.go:29] interesting pod/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Nov 03 02:31:01.708379 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:01.708295 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn" podUID="dce11384-8e67-4aef-80ab-3ddd74099bf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Nov 03 02:31:01.814401 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:01.809664 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8566dce2-ttts6"] Nov 03 02:31:01.814594 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:01.814449 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8566dce2-ttts6" Nov 03 02:31:01.849287 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:01.849247 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8566dce2-ttts6"] Nov 03 02:31:01.853453 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:01.853431 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxg7f\" (UniqueName: \"kubernetes.io/projected/ab77c343-4072-41bc-9dac-384db1bd6a81-kube-api-access-bxg7f\") pod \"maas-api-revoke-sub-8566dce2-ttts6\" (UID: \"ab77c343-4072-41bc-9dac-384db1bd6a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8566dce2-ttts6" Nov 03 02:31:01.853603 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:01.853490 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab77c343-4072-41bc-9dac-384db1bd6a81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8566dce2-ttts6\" (UID: \"ab77c343-4072-41bc-9dac-384db1bd6a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8566dce2-ttts6" Nov 03 02:31:01.853657 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:01.853624 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab77c343-4072-41bc-9dac-384db1bd6a81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8566dce2-ttts6\" (UID: \"ab77c343-4072-41bc-9dac-384db1bd6a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8566dce2-ttts6" Nov 03 02:31:01.906195 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:01.906146 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:31:01.906818 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:01.906799 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:31:01.907231 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:01.907213 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:31:01.907572 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:01.907550 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:31:01.962429 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:01.954751 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab77c343-4072-41bc-9dac-384db1bd6a81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8566dce2-ttts6\" (UID: \"ab77c343-4072-41bc-9dac-384db1bd6a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8566dce2-ttts6" Nov 03 02:31:01.962429 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:01.954876 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxg7f\" (UniqueName: \"kubernetes.io/projected/ab77c343-4072-41bc-9dac-384db1bd6a81-kube-api-access-bxg7f\") pod \"maas-api-revoke-sub-8566dce2-ttts6\" (UID: \"ab77c343-4072-41bc-9dac-384db1bd6a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8566dce2-ttts6" Nov 03 02:31:01.962429 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:01.954919 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab77c343-4072-41bc-9dac-384db1bd6a81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8566dce2-ttts6\" (UID: \"ab77c343-4072-41bc-9dac-384db1bd6a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8566dce2-ttts6" Nov 03 02:31:01.962429 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:01.955764 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab77c343-4072-41bc-9dac-384db1bd6a81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8566dce2-ttts6\" (UID: \"ab77c343-4072-41bc-9dac-384db1bd6a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8566dce2-ttts6" Nov 03 02:31:02.005383 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:02.005273 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab77c343-4072-41bc-9dac-384db1bd6a81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8566dce2-ttts6\" (UID: \"ab77c343-4072-41bc-9dac-384db1bd6a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8566dce2-ttts6" Nov 03 02:31:02.082151 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:02.082116 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxg7f\" (UniqueName: \"kubernetes.io/projected/ab77c343-4072-41bc-9dac-384db1bd6a81-kube-api-access-bxg7f\") pod \"maas-api-revoke-sub-8566dce2-ttts6\" (UID: \"ab77c343-4072-41bc-9dac-384db1bd6a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8566dce2-ttts6" Nov 03 02:31:02.132339 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:02.132312 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8566dce2-ttts6" Nov 03 02:31:02.513465 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:02.513427 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8566dce2-ttts6"] Nov 03 02:31:02.519270 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:31:02.519242 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab77c343_4072_41bc_9dac_384db1bd6a81.slice/crio-5ad70e8b1e712bd050e9a0037c035f7a4d06417e9b8dea737a4ba15e68079eff WatchSource:0}: Error finding container 5ad70e8b1e712bd050e9a0037c035f7a4d06417e9b8dea737a4ba15e68079eff: Status 404 returned error can't find the container with id 5ad70e8b1e712bd050e9a0037c035f7a4d06417e9b8dea737a4ba15e68079eff Nov 03 02:31:02.540425 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:02.540399 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8566dce2-ttts6" event={"ID":"ab77c343-4072-41bc-9dac-384db1bd6a81","Type":"ContainerStarted","Data":"5ad70e8b1e712bd050e9a0037c035f7a4d06417e9b8dea737a4ba15e68079eff"} Nov 03 02:31:02.676574 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:02.676540 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8" Nov 03 02:31:02.676939 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:02.676904 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8" Nov 03 02:31:02.677608 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:02.677588 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8" Nov 03 02:31:02.712474 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:02.712432 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn" Nov 03 02:31:02.712718 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:02.712696 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn" Nov 03 02:31:02.713511 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:02.713495 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn" Nov 03 02:31:03.547666 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:03.547629 5605 generic.go:356] "Generic (PLEG): container finished" podID="ab77c343-4072-41bc-9dac-384db1bd6a81" containerID="72f2ba1030ea9a18bf019b07b1ffc0667976818ab4c60be7d805ca3cb5ad87cc" exitCode=0 Nov 03 02:31:03.547851 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:03.547712 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8566dce2-ttts6" event={"ID":"ab77c343-4072-41bc-9dac-384db1bd6a81","Type":"ContainerDied","Data":"72f2ba1030ea9a18bf019b07b1ffc0667976818ab4c60be7d805ca3cb5ad87cc"} Nov 03 02:31:04.694777 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:04.694750 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8566dce2-ttts6" Nov 03 02:31:04.807243 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:04.807151 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab77c343-4072-41bc-9dac-384db1bd6a81-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab77c343-4072-41bc-9dac-384db1bd6a81-maas-api-xks-ca\") pod \"ab77c343-4072-41bc-9dac-384db1bd6a81\" (UID: \"ab77c343-4072-41bc-9dac-384db1bd6a81\") " Nov 03 02:31:04.807243 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:04.807214 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab77c343-4072-41bc-9dac-384db1bd6a81-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab77c343-4072-41bc-9dac-384db1bd6a81-maas-api-openshift-ca\") pod \"ab77c343-4072-41bc-9dac-384db1bd6a81\" (UID: \"ab77c343-4072-41bc-9dac-384db1bd6a81\") " Nov 03 02:31:04.807520 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:04.807414 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab77c343-4072-41bc-9dac-384db1bd6a81-kube-api-access-bxg7f\" (UniqueName: \"kubernetes.io/projected/ab77c343-4072-41bc-9dac-384db1bd6a81-kube-api-access-bxg7f\") pod \"ab77c343-4072-41bc-9dac-384db1bd6a81\" (UID: \"ab77c343-4072-41bc-9dac-384db1bd6a81\") " Nov 03 02:31:04.807647 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:04.807617 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab77c343-4072-41bc-9dac-384db1bd6a81-maas-api-openshift-ca" pod "ab77c343-4072-41bc-9dac-384db1bd6a81" (UID: "ab77c343-4072-41bc-9dac-384db1bd6a81"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:31:04.807928 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:04.807905 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab77c343-4072-41bc-9dac-384db1bd6a81-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:04.809515 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:04.809491 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab77c343-4072-41bc-9dac-384db1bd6a81-maas-api-xks-ca" pod "ab77c343-4072-41bc-9dac-384db1bd6a81" (UID: "ab77c343-4072-41bc-9dac-384db1bd6a81"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:31:04.809613 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:04.809586 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab77c343-4072-41bc-9dac-384db1bd6a81-kube-api-access-bxg7f" pod "ab77c343-4072-41bc-9dac-384db1bd6a81" (UID: "ab77c343-4072-41bc-9dac-384db1bd6a81"). InnerVolumeSpecName "kube-api-access-bxg7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:31:04.909452 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:04.909419 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxg7f\" (UniqueName: \"kubernetes.io/projected/ab77c343-4072-41bc-9dac-384db1bd6a81-kube-api-access-bxg7f\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:04.909452 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:04.909452 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab77c343-4072-41bc-9dac-384db1bd6a81-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:05.560619 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:05.560586 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8566dce2-ttts6" Nov 03 02:31:05.560830 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:05.560582 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8566dce2-ttts6" event={"ID":"ab77c343-4072-41bc-9dac-384db1bd6a81","Type":"ContainerDied","Data":"5ad70e8b1e712bd050e9a0037c035f7a4d06417e9b8dea737a4ba15e68079eff"} Nov 03 02:31:05.560830 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:05.560730 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ad70e8b1e712bd050e9a0037c035f7a4d06417e9b8dea737a4ba15e68079eff" Nov 03 02:31:09.812077 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:09.812032 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-496177-5b44d87cbf-5p6qp"] Nov 03 02:31:14.812737 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:14.812629 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-496177-5b44d87cbf-5p6qp" podUID="0714b248-bec2-44ad-9ca3-e99d0bb89659" containerName="payload-processing" containerID="cri-o://bc5092b8966dc8c47569fd929c55c83434407e6cfbae1cc9dc5a1f813715c92b" gracePeriod=25 Nov 03 02:31:15.068688 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:15.068631 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-496177-5b44d87cbf-5p6qp" Nov 03 02:31:15.130542 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:15.130514 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0714b248-bec2-44ad-9ca3-e99d0bb89659-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0714b248-bec2-44ad-9ca3-e99d0bb89659-plugins-config-volume\") pod \"0714b248-bec2-44ad-9ca3-e99d0bb89659\" (UID: \"0714b248-bec2-44ad-9ca3-e99d0bb89659\") " Nov 03 02:31:15.130710 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:15.130683 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0714b248-bec2-44ad-9ca3-e99d0bb89659-kube-api-access-dp4jt\" (UniqueName: \"kubernetes.io/projected/0714b248-bec2-44ad-9ca3-e99d0bb89659-kube-api-access-dp4jt\") pod \"0714b248-bec2-44ad-9ca3-e99d0bb89659\" (UID: \"0714b248-bec2-44ad-9ca3-e99d0bb89659\") " Nov 03 02:31:15.130948 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:15.130925 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0714b248-bec2-44ad-9ca3-e99d0bb89659-plugins-config-volume" pod "0714b248-bec2-44ad-9ca3-e99d0bb89659" (UID: "0714b248-bec2-44ad-9ca3-e99d0bb89659"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:31:15.131090 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:15.131068 5605 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0714b248-bec2-44ad-9ca3-e99d0bb89659-plugins-config-volume\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:15.132750 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:15.132726 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0714b248-bec2-44ad-9ca3-e99d0bb89659-kube-api-access-dp4jt" pod "0714b248-bec2-44ad-9ca3-e99d0bb89659" (UID: "0714b248-bec2-44ad-9ca3-e99d0bb89659"). InnerVolumeSpecName "kube-api-access-dp4jt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:31:15.231889 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:15.231851 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dp4jt\" (UniqueName: \"kubernetes.io/projected/0714b248-bec2-44ad-9ca3-e99d0bb89659-kube-api-access-dp4jt\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:15.615350 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:15.615315 5605 generic.go:356] "Generic (PLEG): container finished" podID="0714b248-bec2-44ad-9ca3-e99d0bb89659" containerID="bc5092b8966dc8c47569fd929c55c83434407e6cfbae1cc9dc5a1f813715c92b" exitCode=0 Nov 03 02:31:15.615517 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:15.615376 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-496177-5b44d87cbf-5p6qp" Nov 03 02:31:15.615517 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:15.615403 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-496177-5b44d87cbf-5p6qp" event={"ID":"0714b248-bec2-44ad-9ca3-e99d0bb89659","Type":"ContainerDied","Data":"bc5092b8966dc8c47569fd929c55c83434407e6cfbae1cc9dc5a1f813715c92b"} Nov 03 02:31:15.615517 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:15.615457 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-496177-5b44d87cbf-5p6qp" event={"ID":"0714b248-bec2-44ad-9ca3-e99d0bb89659","Type":"ContainerDied","Data":"49442e37d3e7eb7c6c841596f46650c1b91a176fb758e7f1cbd7f9e1c0a0ec96"} Nov 03 02:31:15.615517 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:15.615485 5605 scope.go:122] "RemoveContainer" containerID="bc5092b8966dc8c47569fd929c55c83434407e6cfbae1cc9dc5a1f813715c92b" Nov 03 02:31:15.624703 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:15.624684 5605 scope.go:122] "RemoveContainer" containerID="bc5092b8966dc8c47569fd929c55c83434407e6cfbae1cc9dc5a1f813715c92b" Nov 03 02:31:15.624974 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:31:15.624953 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc5092b8966dc8c47569fd929c55c83434407e6cfbae1cc9dc5a1f813715c92b\": container with ID starting with bc5092b8966dc8c47569fd929c55c83434407e6cfbae1cc9dc5a1f813715c92b not found: ID does not exist" containerID="bc5092b8966dc8c47569fd929c55c83434407e6cfbae1cc9dc5a1f813715c92b" Nov 03 02:31:15.625020 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:15.624983 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc5092b8966dc8c47569fd929c55c83434407e6cfbae1cc9dc5a1f813715c92b"} err="failed to get container status \"bc5092b8966dc8c47569fd929c55c83434407e6cfbae1cc9dc5a1f813715c92b\": rpc error: code = NotFound desc = could not find container \"bc5092b8966dc8c47569fd929c55c83434407e6cfbae1cc9dc5a1f813715c92b\": container with ID starting with bc5092b8966dc8c47569fd929c55c83434407e6cfbae1cc9dc5a1f813715c92b not found: ID does not exist" Nov 03 02:31:15.768927 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:15.768897 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-496177-5b44d87cbf-5p6qp"] Nov 03 02:31:15.816064 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:15.816035 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-496177-5b44d87cbf-5p6qp"] Nov 03 02:31:16.849497 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:16.849467 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0714b248-bec2-44ad-9ca3-e99d0bb89659" path="/var/lib/kubelet/pods/0714b248-bec2-44ad-9ca3-e99d0bb89659/volumes" Nov 03 02:31:17.528553 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:17.528511 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e4b6450-ddp67"] Nov 03 02:31:17.529158 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:17.529137 5605 cpu_manager.go:414] "containerMap: removing container" podUID="0714b248-bec2-44ad-9ca3-e99d0bb89659" containerName="payload-processing" Nov 03 02:31:17.529209 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:17.529162 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0714b248-bec2-44ad-9ca3-e99d0bb89659" containerName="payload-processing" Nov 03 02:31:17.529209 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:17.529186 5605 cpu_manager.go:414] "containerMap: removing container" podUID="ab77c343-4072-41bc-9dac-384db1bd6a81" containerName="revoke-keys" Nov 03 02:31:17.529209 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:17.529194 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab77c343-4072-41bc-9dac-384db1bd6a81" containerName="revoke-keys" Nov 03 02:31:17.529442 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:17.529346 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="0714b248-bec2-44ad-9ca3-e99d0bb89659" containerName="payload-processing" Nov 03 02:31:17.529442 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:17.529446 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab77c343-4072-41bc-9dac-384db1bd6a81" containerName="revoke-keys" Nov 03 02:31:17.533874 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:17.533853 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e4b6450-ddp67" Nov 03 02:31:17.584953 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:17.584932 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e4b6450-ddp67"] Nov 03 02:31:17.628982 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:17.628952 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:31:17.629276 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:17.629251 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:31:17.629434 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:17.629256 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:31:17.629434 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:17.629298 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:31:17.659918 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:17.659892 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mz2z\" (UniqueName: \"kubernetes.io/projected/86089afc-62e2-4be9-b8a8-9d4926e68a11-kube-api-access-9mz2z\") pod \"maas-api-revoke-sub-2e4b6450-ddp67\" (UID: \"86089afc-62e2-4be9-b8a8-9d4926e68a11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e4b6450-ddp67" Nov 03 02:31:17.660057 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:17.659944 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86089afc-62e2-4be9-b8a8-9d4926e68a11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e4b6450-ddp67\" (UID: \"86089afc-62e2-4be9-b8a8-9d4926e68a11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e4b6450-ddp67" Nov 03 02:31:17.660057 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:17.660030 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86089afc-62e2-4be9-b8a8-9d4926e68a11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e4b6450-ddp67\" (UID: \"86089afc-62e2-4be9-b8a8-9d4926e68a11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e4b6450-ddp67" Nov 03 02:31:17.761190 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:17.761161 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86089afc-62e2-4be9-b8a8-9d4926e68a11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e4b6450-ddp67\" (UID: \"86089afc-62e2-4be9-b8a8-9d4926e68a11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e4b6450-ddp67" Nov 03 02:31:17.761399 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:17.761211 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86089afc-62e2-4be9-b8a8-9d4926e68a11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e4b6450-ddp67\" (UID: \"86089afc-62e2-4be9-b8a8-9d4926e68a11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e4b6450-ddp67" Nov 03 02:31:17.761483 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:17.761403 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mz2z\" (UniqueName: \"kubernetes.io/projected/86089afc-62e2-4be9-b8a8-9d4926e68a11-kube-api-access-9mz2z\") pod \"maas-api-revoke-sub-2e4b6450-ddp67\" (UID: \"86089afc-62e2-4be9-b8a8-9d4926e68a11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e4b6450-ddp67" Nov 03 02:31:17.762093 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:17.762058 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86089afc-62e2-4be9-b8a8-9d4926e68a11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e4b6450-ddp67\" (UID: \"86089afc-62e2-4be9-b8a8-9d4926e68a11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e4b6450-ddp67" Nov 03 02:31:17.763663 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:17.763638 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86089afc-62e2-4be9-b8a8-9d4926e68a11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e4b6450-ddp67\" (UID: \"86089afc-62e2-4be9-b8a8-9d4926e68a11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e4b6450-ddp67" Nov 03 02:31:17.814671 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:17.814643 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mz2z\" (UniqueName: \"kubernetes.io/projected/86089afc-62e2-4be9-b8a8-9d4926e68a11-kube-api-access-9mz2z\") pod \"maas-api-revoke-sub-2e4b6450-ddp67\" (UID: \"86089afc-62e2-4be9-b8a8-9d4926e68a11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e4b6450-ddp67" Nov 03 02:31:17.849203 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:17.849176 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e4b6450-ddp67" Nov 03 02:31:18.246034 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:18.245946 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e4b6450-ddp67"] Nov 03 02:31:18.252499 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:31:18.252466 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86089afc_62e2_4be9_b8a8_9d4926e68a11.slice/crio-4e3a2611ac617667bfe7ba583cc0e170b4e598909152a07a280364007a9f6814 WatchSource:0}: Error finding container 4e3a2611ac617667bfe7ba583cc0e170b4e598909152a07a280364007a9f6814: Status 404 returned error can't find the container with id 4e3a2611ac617667bfe7ba583cc0e170b4e598909152a07a280364007a9f6814 Nov 03 02:31:18.634891 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:18.634848 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e4b6450-ddp67" event={"ID":"86089afc-62e2-4be9-b8a8-9d4926e68a11","Type":"ContainerStarted","Data":"39f3f8edca2048dc9d94968d6b58451bccb2c6950435296ae2bd87cb2f51b88e"} Nov 03 02:31:18.635021 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:18.634910 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e4b6450-ddp67" event={"ID":"86089afc-62e2-4be9-b8a8-9d4926e68a11","Type":"ContainerStarted","Data":"4e3a2611ac617667bfe7ba583cc0e170b4e598909152a07a280364007a9f6814"} Nov 03 02:31:18.741966 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:18.741911 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e4b6450-ddp67" podStartSLOduration=1.7418957800000001 podStartE2EDuration="1.74189578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:31:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:31:18.740488222 +0000 UTC m=+2355.999686375" watchObservedRunningTime="2026-11-03 02:31:18.74189578 +0000 UTC m=+2356.001093933" Nov 03 02:31:19.647986 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:19.647949 5605 generic.go:356] "Generic (PLEG): container finished" podID="86089afc-62e2-4be9-b8a8-9d4926e68a11" containerID="39f3f8edca2048dc9d94968d6b58451bccb2c6950435296ae2bd87cb2f51b88e" exitCode=0 Nov 03 02:31:19.648392 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:19.648031 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e4b6450-ddp67" event={"ID":"86089afc-62e2-4be9-b8a8-9d4926e68a11","Type":"ContainerDied","Data":"39f3f8edca2048dc9d94968d6b58451bccb2c6950435296ae2bd87cb2f51b88e"} Nov 03 02:31:20.792946 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:20.792920 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e4b6450-ddp67" Nov 03 02:31:20.890719 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:20.890693 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw"] Nov 03 02:31:20.891084 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:20.891072 5605 cpu_manager.go:414] "containerMap: removing container" podUID="86089afc-62e2-4be9-b8a8-9d4926e68a11" containerName="revoke-keys" Nov 03 02:31:20.891132 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:20.891086 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86089afc-62e2-4be9-b8a8-9d4926e68a11" containerName="revoke-keys" Nov 03 02:31:20.891195 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:20.891187 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="86089afc-62e2-4be9-b8a8-9d4926e68a11" containerName="revoke-keys" Nov 03 02:31:20.892984 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:20.892961 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86089afc-62e2-4be9-b8a8-9d4926e68a11-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86089afc-62e2-4be9-b8a8-9d4926e68a11-maas-api-xks-ca\") pod \"86089afc-62e2-4be9-b8a8-9d4926e68a11\" (UID: \"86089afc-62e2-4be9-b8a8-9d4926e68a11\") " Nov 03 02:31:20.893056 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:20.893034 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86089afc-62e2-4be9-b8a8-9d4926e68a11-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86089afc-62e2-4be9-b8a8-9d4926e68a11-maas-api-openshift-ca\") pod \"86089afc-62e2-4be9-b8a8-9d4926e68a11\" (UID: \"86089afc-62e2-4be9-b8a8-9d4926e68a11\") " Nov 03 02:31:20.893120 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:20.893108 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86089afc-62e2-4be9-b8a8-9d4926e68a11-kube-api-access-9mz2z\" (UniqueName: \"kubernetes.io/projected/86089afc-62e2-4be9-b8a8-9d4926e68a11-kube-api-access-9mz2z\") pod \"86089afc-62e2-4be9-b8a8-9d4926e68a11\" (UID: \"86089afc-62e2-4be9-b8a8-9d4926e68a11\") " Nov 03 02:31:20.893416 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:20.893392 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86089afc-62e2-4be9-b8a8-9d4926e68a11-maas-api-openshift-ca" pod "86089afc-62e2-4be9-b8a8-9d4926e68a11" (UID: "86089afc-62e2-4be9-b8a8-9d4926e68a11"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:31:20.894135 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:20.894117 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29"] Nov 03 02:31:20.894243 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:20.894227 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw" Nov 03 02:31:20.895313 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:20.895289 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86089afc-62e2-4be9-b8a8-9d4926e68a11-maas-api-xks-ca" pod "86089afc-62e2-4be9-b8a8-9d4926e68a11" (UID: "86089afc-62e2-4be9-b8a8-9d4926e68a11"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:31:20.895455 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:20.895304 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86089afc-62e2-4be9-b8a8-9d4926e68a11-kube-api-access-9mz2z" pod "86089afc-62e2-4be9-b8a8-9d4926e68a11" (UID: "86089afc-62e2-4be9-b8a8-9d4926e68a11"). InnerVolumeSpecName "kube-api-access-9mz2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:31:20.898102 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:20.898084 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29" Nov 03 02:31:20.952037 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:20.952008 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw"] Nov 03 02:31:20.994077 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:20.994053 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75dg4\" (UniqueName: \"kubernetes.io/projected/b337ace7-57a1-4583-8591-9c2f3040dba6-kube-api-access-75dg4\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw\" (UID: \"b337ace7-57a1-4583-8591-9c2f3040dba6\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw" Nov 03 02:31:20.994217 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:20.994125 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b337ace7-57a1-4583-8591-9c2f3040dba6-istio-envoy\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw\" (UID: \"b337ace7-57a1-4583-8591-9c2f3040dba6\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw" Nov 03 02:31:20.994217 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:20.994173 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4846c4da-bf51-483f-8860-dc49d45f5d22-istio-data\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29\" (UID: \"4846c4da-bf51-483f-8860-dc49d45f5d22\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29" Nov 03 02:31:20.994294 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:20.994229 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4846c4da-bf51-483f-8860-dc49d45f5d22-workload-certs\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29\" (UID: \"4846c4da-bf51-483f-8860-dc49d45f5d22\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29" Nov 03 02:31:20.994329 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:20.994305 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b337ace7-57a1-4583-8591-9c2f3040dba6-istiod-ca-cert\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw\" (UID: \"b337ace7-57a1-4583-8591-9c2f3040dba6\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw" Nov 03 02:31:20.994381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:20.994343 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4846c4da-bf51-483f-8860-dc49d45f5d22-istio-token\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29\" (UID: \"4846c4da-bf51-483f-8860-dc49d45f5d22\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29" Nov 03 02:31:20.994442 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:20.994426 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b337ace7-57a1-4583-8591-9c2f3040dba6-istio-podinfo\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw\" (UID: \"b337ace7-57a1-4583-8591-9c2f3040dba6\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw" Nov 03 02:31:20.994546 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:20.994521 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b337ace7-57a1-4583-8591-9c2f3040dba6-workload-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw\" (UID: \"b337ace7-57a1-4583-8591-9c2f3040dba6\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw" Nov 03 02:31:20.994622 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:20.994605 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b337ace7-57a1-4583-8591-9c2f3040dba6-credential-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw\" (UID: \"b337ace7-57a1-4583-8591-9c2f3040dba6\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw" Nov 03 02:31:20.994683 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:20.994659 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnfv6\" (UniqueName: \"kubernetes.io/projected/4846c4da-bf51-483f-8860-dc49d45f5d22-kube-api-access-pnfv6\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29\" (UID: \"4846c4da-bf51-483f-8860-dc49d45f5d22\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29" Nov 03 02:31:20.994936 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:20.994896 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4846c4da-bf51-483f-8860-dc49d45f5d22-istiod-ca-cert\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29\" (UID: \"4846c4da-bf51-483f-8860-dc49d45f5d22\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29" Nov 03 02:31:20.995010 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:20.994957 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4846c4da-bf51-483f-8860-dc49d45f5d22-workload-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29\" (UID: \"4846c4da-bf51-483f-8860-dc49d45f5d22\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29" Nov 03 02:31:20.995010 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:20.994994 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b337ace7-57a1-4583-8591-9c2f3040dba6-istio-token\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw\" (UID: \"b337ace7-57a1-4583-8591-9c2f3040dba6\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw" Nov 03 02:31:20.995121 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:20.995049 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b337ace7-57a1-4583-8591-9c2f3040dba6-istio-data\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw\" (UID: \"b337ace7-57a1-4583-8591-9c2f3040dba6\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw" Nov 03 02:31:20.995121 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:20.995081 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4846c4da-bf51-483f-8860-dc49d45f5d22-credential-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29\" (UID: \"4846c4da-bf51-483f-8860-dc49d45f5d22\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29" Nov 03 02:31:20.995121 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:20.995099 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4846c4da-bf51-483f-8860-dc49d45f5d22-istio-envoy\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29\" (UID: \"4846c4da-bf51-483f-8860-dc49d45f5d22\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29" Nov 03 02:31:20.995274 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:20.995125 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b337ace7-57a1-4583-8591-9c2f3040dba6-workload-certs\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw\" (UID: \"b337ace7-57a1-4583-8591-9c2f3040dba6\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw" Nov 03 02:31:20.995274 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:20.995142 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4846c4da-bf51-483f-8860-dc49d45f5d22-istio-podinfo\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29\" (UID: \"4846c4da-bf51-483f-8860-dc49d45f5d22\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29" Nov 03 02:31:20.995410 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:20.995335 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86089afc-62e2-4be9-b8a8-9d4926e68a11-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:20.995410 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:20.995375 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86089afc-62e2-4be9-b8a8-9d4926e68a11-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:20.995410 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:20.995393 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mz2z\" (UniqueName: \"kubernetes.io/projected/86089afc-62e2-4be9-b8a8-9d4926e68a11-kube-api-access-9mz2z\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:20.998230 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:20.998205 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29"] Nov 03 02:31:21.096689 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:21.096656 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4846c4da-bf51-483f-8860-dc49d45f5d22-istio-data\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29\" (UID: \"4846c4da-bf51-483f-8860-dc49d45f5d22\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29" Nov 03 02:31:21.096689 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:21.096693 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4846c4da-bf51-483f-8860-dc49d45f5d22-workload-certs\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29\" (UID: \"4846c4da-bf51-483f-8860-dc49d45f5d22\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29" Nov 03 02:31:21.096916 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:21.096735 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b337ace7-57a1-4583-8591-9c2f3040dba6-istiod-ca-cert\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw\" (UID: \"b337ace7-57a1-4583-8591-9c2f3040dba6\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw" Nov 03 02:31:21.096916 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:21.096838 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4846c4da-bf51-483f-8860-dc49d45f5d22-istio-token\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29\" (UID: \"4846c4da-bf51-483f-8860-dc49d45f5d22\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29" Nov 03 02:31:21.096916 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:21.096874 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b337ace7-57a1-4583-8591-9c2f3040dba6-istio-podinfo\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw\" (UID: \"b337ace7-57a1-4583-8591-9c2f3040dba6\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw" Nov 03 02:31:21.097090 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:21.096919 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b337ace7-57a1-4583-8591-9c2f3040dba6-workload-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw\" (UID: \"b337ace7-57a1-4583-8591-9c2f3040dba6\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw" Nov 03 02:31:21.097090 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:21.096949 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b337ace7-57a1-4583-8591-9c2f3040dba6-credential-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw\" (UID: \"b337ace7-57a1-4583-8591-9c2f3040dba6\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw" Nov 03 02:31:21.097090 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:21.096998 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnfv6\" (UniqueName: \"kubernetes.io/projected/4846c4da-bf51-483f-8860-dc49d45f5d22-kube-api-access-pnfv6\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29\" (UID: \"4846c4da-bf51-483f-8860-dc49d45f5d22\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29" Nov 03 02:31:21.097090 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:21.097031 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4846c4da-bf51-483f-8860-dc49d45f5d22-istiod-ca-cert\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29\" (UID: \"4846c4da-bf51-483f-8860-dc49d45f5d22\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29" Nov 03 02:31:21.097090 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:21.097069 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4846c4da-bf51-483f-8860-dc49d45f5d22-workload-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29\" (UID: \"4846c4da-bf51-483f-8860-dc49d45f5d22\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29" Nov 03 02:31:21.097474 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:21.097111 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b337ace7-57a1-4583-8591-9c2f3040dba6-istio-token\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw\" (UID: \"b337ace7-57a1-4583-8591-9c2f3040dba6\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw" Nov 03 02:31:21.097474 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:21.097118 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4846c4da-bf51-483f-8860-dc49d45f5d22-istio-data\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29\" (UID: \"4846c4da-bf51-483f-8860-dc49d45f5d22\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29" Nov 03 02:31:21.097474 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:21.097143 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b337ace7-57a1-4583-8591-9c2f3040dba6-istio-data\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw\" (UID: \"b337ace7-57a1-4583-8591-9c2f3040dba6\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw" Nov 03 02:31:21.097474 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:21.097176 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4846c4da-bf51-483f-8860-dc49d45f5d22-credential-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29\" (UID: \"4846c4da-bf51-483f-8860-dc49d45f5d22\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29" Nov 03 02:31:21.097474 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:21.097200 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4846c4da-bf51-483f-8860-dc49d45f5d22-istio-envoy\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29\" (UID: \"4846c4da-bf51-483f-8860-dc49d45f5d22\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29" Nov 03 02:31:21.097474 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:21.097242 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b337ace7-57a1-4583-8591-9c2f3040dba6-workload-certs\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw\" (UID: \"b337ace7-57a1-4583-8591-9c2f3040dba6\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw" Nov 03 02:31:21.097474 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:21.097275 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4846c4da-bf51-483f-8860-dc49d45f5d22-istio-podinfo\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29\" (UID: \"4846c4da-bf51-483f-8860-dc49d45f5d22\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29" Nov 03 02:31:21.097474 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:21.097292 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b337ace7-57a1-4583-8591-9c2f3040dba6-workload-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw\" (UID: \"b337ace7-57a1-4583-8591-9c2f3040dba6\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw" Nov 03 02:31:21.097474 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:21.097371 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b337ace7-57a1-4583-8591-9c2f3040dba6-credential-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw\" (UID: \"b337ace7-57a1-4583-8591-9c2f3040dba6\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw" Nov 03 02:31:21.097474 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:21.097402 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75dg4\" (UniqueName: \"kubernetes.io/projected/b337ace7-57a1-4583-8591-9c2f3040dba6-kube-api-access-75dg4\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw\" (UID: \"b337ace7-57a1-4583-8591-9c2f3040dba6\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw" Nov 03 02:31:21.097474 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:21.097426 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4846c4da-bf51-483f-8860-dc49d45f5d22-workload-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29\" (UID: \"4846c4da-bf51-483f-8860-dc49d45f5d22\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29" Nov 03 02:31:21.097474 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:21.097450 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b337ace7-57a1-4583-8591-9c2f3040dba6-istio-envoy\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw\" (UID: \"b337ace7-57a1-4583-8591-9c2f3040dba6\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw" Nov 03 02:31:21.098098 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:21.097140 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4846c4da-bf51-483f-8860-dc49d45f5d22-workload-certs\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29\" (UID: \"4846c4da-bf51-483f-8860-dc49d45f5d22\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29" Nov 03 02:31:21.098098 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:21.097646 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4846c4da-bf51-483f-8860-dc49d45f5d22-credential-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29\" (UID: \"4846c4da-bf51-483f-8860-dc49d45f5d22\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29" Nov 03 02:31:21.098098 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:21.097664 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b337ace7-57a1-4583-8591-9c2f3040dba6-istio-data\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw\" (UID: \"b337ace7-57a1-4583-8591-9c2f3040dba6\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw" Nov 03 02:31:21.098098 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:21.097813 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4846c4da-bf51-483f-8860-dc49d45f5d22-istiod-ca-cert\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29\" (UID: \"4846c4da-bf51-483f-8860-dc49d45f5d22\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29" Nov 03 02:31:21.098098 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:21.097896 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b337ace7-57a1-4583-8591-9c2f3040dba6-workload-certs\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw\" (UID: \"b337ace7-57a1-4583-8591-9c2f3040dba6\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw" Nov 03 02:31:21.098261 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:21.098134 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b337ace7-57a1-4583-8591-9c2f3040dba6-istiod-ca-cert\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw\" (UID: \"b337ace7-57a1-4583-8591-9c2f3040dba6\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw" Nov 03 02:31:21.099731 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:21.099705 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4846c4da-bf51-483f-8860-dc49d45f5d22-istio-podinfo\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29\" (UID: \"4846c4da-bf51-483f-8860-dc49d45f5d22\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29" Nov 03 02:31:21.099861 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:21.099821 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b337ace7-57a1-4583-8591-9c2f3040dba6-istio-podinfo\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw\" (UID: \"b337ace7-57a1-4583-8591-9c2f3040dba6\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw" Nov 03 02:31:21.099918 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:21.099894 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b337ace7-57a1-4583-8591-9c2f3040dba6-istio-envoy\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw\" (UID: \"b337ace7-57a1-4583-8591-9c2f3040dba6\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw" Nov 03 02:31:21.099955 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:21.099924 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4846c4da-bf51-483f-8860-dc49d45f5d22-istio-envoy\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29\" (UID: \"4846c4da-bf51-483f-8860-dc49d45f5d22\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29" Nov 03 02:31:21.154020 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:21.153994 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75dg4\" (UniqueName: \"kubernetes.io/projected/b337ace7-57a1-4583-8591-9c2f3040dba6-kube-api-access-75dg4\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw\" (UID: \"b337ace7-57a1-4583-8591-9c2f3040dba6\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw" Nov 03 02:31:21.154154 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:21.154130 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b337ace7-57a1-4583-8591-9c2f3040dba6-istio-token\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw\" (UID: \"b337ace7-57a1-4583-8591-9c2f3040dba6\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw" Nov 03 02:31:21.154231 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:21.154196 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4846c4da-bf51-483f-8860-dc49d45f5d22-istio-token\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29\" (UID: \"4846c4da-bf51-483f-8860-dc49d45f5d22\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29" Nov 03 02:31:21.154703 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:21.154687 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnfv6\" (UniqueName: \"kubernetes.io/projected/4846c4da-bf51-483f-8860-dc49d45f5d22-kube-api-access-pnfv6\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29\" (UID: \"4846c4da-bf51-483f-8860-dc49d45f5d22\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29" Nov 03 02:31:21.218343 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:21.218294 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw" Nov 03 02:31:21.223261 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:21.223238 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29" Nov 03 02:31:21.643604 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:21.643567 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw"] Nov 03 02:31:21.646611 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:21.646522 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29"] Nov 03 02:31:21.649523 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:31:21.649490 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb337ace7_57a1_4583_8591_9c2f3040dba6.slice/crio-b6aed4dc3b9967030e3077b2c8af616c9e7116662e739be7a2ce899f28a7ab41 WatchSource:0}: Error finding container b6aed4dc3b9967030e3077b2c8af616c9e7116662e739be7a2ce899f28a7ab41: Status 404 returned error can't find the container with id b6aed4dc3b9967030e3077b2c8af616c9e7116662e739be7a2ce899f28a7ab41 Nov 03 02:31:21.650538 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:31:21.650510 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4846c4da_bf51_483f_8860_dc49d45f5d22.slice/crio-d66620b9ee044d122dbdfe46f2502fe7034f460d6e1475ac41bc77bb3938c035 WatchSource:0}: Error finding container d66620b9ee044d122dbdfe46f2502fe7034f460d6e1475ac41bc77bb3938c035: Status 404 returned error can't find the container with id d66620b9ee044d122dbdfe46f2502fe7034f460d6e1475ac41bc77bb3938c035 Nov 03 02:31:21.652344 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:21.652311 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:31:21.652454 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:21.652424 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:31:21.652529 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:21.652510 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:31:21.653226 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:21.653197 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:31:21.653283 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:21.653270 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:31:21.653319 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:21.653302 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:31:21.659655 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:21.659636 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e4b6450-ddp67" Nov 03 02:31:21.659768 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:21.659634 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e4b6450-ddp67" event={"ID":"86089afc-62e2-4be9-b8a8-9d4926e68a11","Type":"ContainerDied","Data":"4e3a2611ac617667bfe7ba583cc0e170b4e598909152a07a280364007a9f6814"} Nov 03 02:31:21.659834 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:21.659770 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e3a2611ac617667bfe7ba583cc0e170b4e598909152a07a280364007a9f6814" Nov 03 02:31:21.661084 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:21.661058 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw" event={"ID":"b337ace7-57a1-4583-8591-9c2f3040dba6","Type":"ContainerStarted","Data":"b6aed4dc3b9967030e3077b2c8af616c9e7116662e739be7a2ce899f28a7ab41"} Nov 03 02:31:21.662589 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:21.662561 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29" event={"ID":"4846c4da-bf51-483f-8860-dc49d45f5d22","Type":"ContainerStarted","Data":"d66620b9ee044d122dbdfe46f2502fe7034f460d6e1475ac41bc77bb3938c035"} Nov 03 02:31:22.685255 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:22.685216 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw" event={"ID":"b337ace7-57a1-4583-8591-9c2f3040dba6","Type":"ContainerStarted","Data":"0a547352e01d73c715ee3e182e060cd3f9a1a49618a56830f7f334d5d8e0ad18"} Nov 03 02:31:22.686914 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:22.686894 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29" event={"ID":"4846c4da-bf51-483f-8860-dc49d45f5d22","Type":"ContainerStarted","Data":"64773bd1dc784b2154f54a548b45a25f7be3f6c96fb58f616d6c989a52d966e1"} Nov 03 02:31:22.884023 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:22.883952 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw" podStartSLOduration=2.883931504 podStartE2EDuration="2.883931504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:31:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:31:22.84463554 +0000 UTC m=+2360.103833694" watchObservedRunningTime="2026-11-03 02:31:22.883931504 +0000 UTC m=+2360.143129658" Nov 03 02:31:22.989440 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:22.988792 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29" podStartSLOduration=2.988773258 podStartE2EDuration="2.988773258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:31:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:31:22.966830242 +0000 UTC m=+2360.226028397" watchObservedRunningTime="2026-11-03 02:31:22.988773258 +0000 UTC m=+2360.247971412" Nov 03 02:31:23.219422 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:23.219351 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw" Nov 03 02:31:23.220767 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:23.220745 5605 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 03 02:31:23.220833 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:23.220790 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw" podUID="b337ace7-57a1-4583-8591-9c2f3040dba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 03 02:31:23.224015 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:23.223997 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29" Nov 03 02:31:23.225017 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:23.224993 5605 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 03 02:31:23.225135 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:23.225035 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29" podUID="4846c4da-bf51-483f-8860-dc49d45f5d22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 03 02:31:23.726794 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:23.726742 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn"] Nov 03 02:31:23.727282 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:23.726995 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn" podUID="fb2bd860-6aa6-4f34-a90a-d599c74c4cba" containerName="istio-proxy" containerID="cri-o://dc5399d0c525d8ee88da7da8052ae368ccf311b045adc2048badb9d3596b978d" gracePeriod=30 Nov 03 02:31:24.013696 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:31:24.013613 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:31:24.206538 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:24.206508 5605 patch_prober.go:29] interesting pod/e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 02:31:24.206769 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:24.206749 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn" podUID="fb2bd860-6aa6-4f34-a90a-d599c74c4cba" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 02:31:24.219308 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:24.219287 5605 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 03 02:31:24.219401 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:24.219323 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw" podUID="b337ace7-57a1-4583-8591-9c2f3040dba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 03 02:31:24.223692 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:24.223674 5605 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 03 02:31:24.223750 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:24.223703 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29" podUID="4846c4da-bf51-483f-8860-dc49d45f5d22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 03 02:31:25.221051 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:25.221007 5605 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 03 02:31:25.221558 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:25.221078 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw" podUID="b337ace7-57a1-4583-8591-9c2f3040dba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 03 02:31:25.224104 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:25.224071 5605 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 03 02:31:25.224253 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:25.224119 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29" podUID="4846c4da-bf51-483f-8860-dc49d45f5d22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 03 02:31:25.606380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:25.605393 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw"] Nov 03 02:31:25.609738 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:25.609713 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw" Nov 03 02:31:25.784383 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:25.764208 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/229d9be7-d6b9-4760-80ed-84592e5693d4-workload-socket\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw\" (UID: \"229d9be7-d6b9-4760-80ed-84592e5693d4\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw" Nov 03 02:31:25.784850 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:25.784815 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/229d9be7-d6b9-4760-80ed-84592e5693d4-workload-certs\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw\" (UID: \"229d9be7-d6b9-4760-80ed-84592e5693d4\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw" Nov 03 02:31:25.785093 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:25.785075 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/229d9be7-d6b9-4760-80ed-84592e5693d4-credential-socket\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw\" (UID: \"229d9be7-d6b9-4760-80ed-84592e5693d4\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw" Nov 03 02:31:25.785240 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:25.785225 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/229d9be7-d6b9-4760-80ed-84592e5693d4-istio-envoy\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw\" (UID: \"229d9be7-d6b9-4760-80ed-84592e5693d4\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw" Nov 03 02:31:25.785346 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:25.785332 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpdmk\" (UniqueName: \"kubernetes.io/projected/229d9be7-d6b9-4760-80ed-84592e5693d4-kube-api-access-qpdmk\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw\" (UID: \"229d9be7-d6b9-4760-80ed-84592e5693d4\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw" Nov 03 02:31:25.785613 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:25.785591 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/229d9be7-d6b9-4760-80ed-84592e5693d4-istiod-ca-cert\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw\" (UID: \"229d9be7-d6b9-4760-80ed-84592e5693d4\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw" Nov 03 02:31:25.785840 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:25.785821 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/229d9be7-d6b9-4760-80ed-84592e5693d4-istio-data\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw\" (UID: \"229d9be7-d6b9-4760-80ed-84592e5693d4\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw" Nov 03 02:31:25.786036 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:25.786020 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/229d9be7-d6b9-4760-80ed-84592e5693d4-istio-podinfo\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw\" (UID: \"229d9be7-d6b9-4760-80ed-84592e5693d4\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw" Nov 03 02:31:25.786215 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:25.786199 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/229d9be7-d6b9-4760-80ed-84592e5693d4-istio-token\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw\" (UID: \"229d9be7-d6b9-4760-80ed-84592e5693d4\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw" Nov 03 02:31:25.812382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:25.809732 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw"] Nov 03 02:31:25.891234 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:25.891125 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/229d9be7-d6b9-4760-80ed-84592e5693d4-istio-envoy\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw\" (UID: \"229d9be7-d6b9-4760-80ed-84592e5693d4\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw" Nov 03 02:31:25.898465 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:25.898435 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpdmk\" (UniqueName: \"kubernetes.io/projected/229d9be7-d6b9-4760-80ed-84592e5693d4-kube-api-access-qpdmk\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw\" (UID: \"229d9be7-d6b9-4760-80ed-84592e5693d4\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw" Nov 03 02:31:25.907400 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:25.899344 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/229d9be7-d6b9-4760-80ed-84592e5693d4-istiod-ca-cert\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw\" (UID: \"229d9be7-d6b9-4760-80ed-84592e5693d4\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw" Nov 03 02:31:25.907400 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:25.905522 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/229d9be7-d6b9-4760-80ed-84592e5693d4-istio-data\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw\" (UID: \"229d9be7-d6b9-4760-80ed-84592e5693d4\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw" Nov 03 02:31:25.907400 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:25.905576 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/229d9be7-d6b9-4760-80ed-84592e5693d4-istio-podinfo\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw\" (UID: \"229d9be7-d6b9-4760-80ed-84592e5693d4\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw" Nov 03 02:31:25.907400 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:25.905631 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/229d9be7-d6b9-4760-80ed-84592e5693d4-istio-token\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw\" (UID: \"229d9be7-d6b9-4760-80ed-84592e5693d4\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw" Nov 03 02:31:25.907400 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:25.905688 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/229d9be7-d6b9-4760-80ed-84592e5693d4-istio-envoy\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw\" (UID: \"229d9be7-d6b9-4760-80ed-84592e5693d4\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw" Nov 03 02:31:25.907400 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:25.905954 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/229d9be7-d6b9-4760-80ed-84592e5693d4-workload-socket\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw\" (UID: \"229d9be7-d6b9-4760-80ed-84592e5693d4\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw" Nov 03 02:31:25.907400 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:25.905685 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/229d9be7-d6b9-4760-80ed-84592e5693d4-workload-socket\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw\" (UID: \"229d9be7-d6b9-4760-80ed-84592e5693d4\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw" Nov 03 02:31:25.907400 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:25.906186 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/229d9be7-d6b9-4760-80ed-84592e5693d4-istio-data\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw\" (UID: \"229d9be7-d6b9-4760-80ed-84592e5693d4\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw" Nov 03 02:31:25.907400 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:25.906198 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/229d9be7-d6b9-4760-80ed-84592e5693d4-workload-certs\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw\" (UID: \"229d9be7-d6b9-4760-80ed-84592e5693d4\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw" Nov 03 02:31:25.907400 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:25.906635 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/229d9be7-d6b9-4760-80ed-84592e5693d4-istiod-ca-cert\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw\" (UID: \"229d9be7-d6b9-4760-80ed-84592e5693d4\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw" Nov 03 02:31:25.908039 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:25.907434 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/229d9be7-d6b9-4760-80ed-84592e5693d4-credential-socket\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw\" (UID: \"229d9be7-d6b9-4760-80ed-84592e5693d4\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw" Nov 03 02:31:25.908039 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:25.907961 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/229d9be7-d6b9-4760-80ed-84592e5693d4-credential-socket\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw\" (UID: \"229d9be7-d6b9-4760-80ed-84592e5693d4\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw" Nov 03 02:31:25.908829 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:25.908292 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/229d9be7-d6b9-4760-80ed-84592e5693d4-workload-certs\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw\" (UID: \"229d9be7-d6b9-4760-80ed-84592e5693d4\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw" Nov 03 02:31:25.910917 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:25.910882 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/229d9be7-d6b9-4760-80ed-84592e5693d4-istio-podinfo\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw\" (UID: \"229d9be7-d6b9-4760-80ed-84592e5693d4\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw" Nov 03 02:31:25.951743 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:25.951679 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpdmk\" (UniqueName: \"kubernetes.io/projected/229d9be7-d6b9-4760-80ed-84592e5693d4-kube-api-access-qpdmk\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw\" (UID: \"229d9be7-d6b9-4760-80ed-84592e5693d4\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw" Nov 03 02:31:26.004939 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:26.004905 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/229d9be7-d6b9-4760-80ed-84592e5693d4-istio-token\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw\" (UID: \"229d9be7-d6b9-4760-80ed-84592e5693d4\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw" Nov 03 02:31:26.219346 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:26.219181 5605 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 03 02:31:26.219346 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:26.219255 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw" podUID="b337ace7-57a1-4583-8591-9c2f3040dba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 03 02:31:26.223320 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:26.223293 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw" Nov 03 02:31:26.223732 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:26.223608 5605 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 03 02:31:26.223732 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:26.223663 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29" podUID="4846c4da-bf51-483f-8860-dc49d45f5d22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 03 02:31:26.706705 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:26.705532 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw"] Nov 03 02:31:26.711402 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:31:26.711349 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod229d9be7_d6b9_4760_80ed_84592e5693d4.slice/crio-c8d278eb10ec880ee9f6bb68a35b283602dfc606c71d0b594fdf7d31d1316571 WatchSource:0}: Error finding container c8d278eb10ec880ee9f6bb68a35b283602dfc606c71d0b594fdf7d31d1316571: Status 404 returned error can't find the container with id c8d278eb10ec880ee9f6bb68a35b283602dfc606c71d0b594fdf7d31d1316571 Nov 03 02:31:26.714125 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:26.713745 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:31:26.714125 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:26.713809 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:31:26.714125 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:26.713838 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:31:26.820954 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:26.820920 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw" event={"ID":"229d9be7-d6b9-4760-80ed-84592e5693d4","Type":"ContainerStarted","Data":"c8d278eb10ec880ee9f6bb68a35b283602dfc606c71d0b594fdf7d31d1316571"} Nov 03 02:31:27.222997 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:27.222856 5605 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 03 02:31:27.222997 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:27.222939 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw" podUID="b337ace7-57a1-4583-8591-9c2f3040dba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 03 02:31:27.223702 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:27.223673 5605 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 03 02:31:27.224039 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:27.223727 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29" podUID="4846c4da-bf51-483f-8860-dc49d45f5d22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 03 02:31:27.835754 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:27.835713 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw" event={"ID":"229d9be7-d6b9-4760-80ed-84592e5693d4","Type":"ContainerStarted","Data":"8694c2b29032070891f4a82edcaa1aa30f11420a997230472ae76da1c7fef1de"} Nov 03 02:31:28.061384 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:28.056644 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw" podStartSLOduration=3.056619588 podStartE2EDuration="3.056619588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:31:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:31:28.006933918 +0000 UTC m=+2365.266132072" watchObservedRunningTime="2026-11-03 02:31:28.056619588 +0000 UTC m=+2365.315817743" Nov 03 02:31:28.220482 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:28.219772 5605 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 03 02:31:28.220482 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:28.219841 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw" podUID="b337ace7-57a1-4583-8591-9c2f3040dba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 03 02:31:28.223709 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:28.223679 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw" Nov 03 02:31:28.224156 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:28.223738 5605 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 03 02:31:28.224156 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:28.223807 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29" podUID="4846c4da-bf51-483f-8860-dc49d45f5d22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 03 02:31:28.225627 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:28.225595 5605 patch_prober.go:29] interesting pod/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 03 02:31:28.225777 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:28.225650 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw" podUID="229d9be7-d6b9-4760-80ed-84592e5693d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 03 02:31:29.106503 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:29.106471 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn" Nov 03 02:31:29.176382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:29.175772 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb2bd860-6aa6-4f34-a90a-d599c74c4cba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb2bd860-6aa6-4f34-a90a-d599c74c4cba-istiod-ca-cert\") pod \"fb2bd860-6aa6-4f34-a90a-d599c74c4cba\" (UID: \"fb2bd860-6aa6-4f34-a90a-d599c74c4cba\") " Nov 03 02:31:29.176382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:29.175868 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb2bd860-6aa6-4f34-a90a-d599c74c4cba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb2bd860-6aa6-4f34-a90a-d599c74c4cba-workload-certs\") pod \"fb2bd860-6aa6-4f34-a90a-d599c74c4cba\" (UID: \"fb2bd860-6aa6-4f34-a90a-d599c74c4cba\") " Nov 03 02:31:29.176382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:29.175916 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb2bd860-6aa6-4f34-a90a-d599c74c4cba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb2bd860-6aa6-4f34-a90a-d599c74c4cba-istio-envoy\") pod \"fb2bd860-6aa6-4f34-a90a-d599c74c4cba\" (UID: \"fb2bd860-6aa6-4f34-a90a-d599c74c4cba\") " Nov 03 02:31:29.176382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:29.176016 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb2bd860-6aa6-4f34-a90a-d599c74c4cba-istio-token\" (UniqueName: \"kubernetes.io/projected/fb2bd860-6aa6-4f34-a90a-d599c74c4cba-istio-token\") pod \"fb2bd860-6aa6-4f34-a90a-d599c74c4cba\" (UID: \"fb2bd860-6aa6-4f34-a90a-d599c74c4cba\") " Nov 03 02:31:29.176382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:29.176046 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb2bd860-6aa6-4f34-a90a-d599c74c4cba-kube-api-access-tbkdq\" (UniqueName: \"kubernetes.io/projected/fb2bd860-6aa6-4f34-a90a-d599c74c4cba-kube-api-access-tbkdq\") pod \"fb2bd860-6aa6-4f34-a90a-d599c74c4cba\" (UID: \"fb2bd860-6aa6-4f34-a90a-d599c74c4cba\") " Nov 03 02:31:29.176382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:29.176087 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb2bd860-6aa6-4f34-a90a-d599c74c4cba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb2bd860-6aa6-4f34-a90a-d599c74c4cba-istio-data\") pod \"fb2bd860-6aa6-4f34-a90a-d599c74c4cba\" (UID: \"fb2bd860-6aa6-4f34-a90a-d599c74c4cba\") " Nov 03 02:31:29.176382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:29.176143 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fb2bd860-6aa6-4f34-a90a-d599c74c4cba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb2bd860-6aa6-4f34-a90a-d599c74c4cba-istio-podinfo\") pod \"fb2bd860-6aa6-4f34-a90a-d599c74c4cba\" (UID: \"fb2bd860-6aa6-4f34-a90a-d599c74c4cba\") " Nov 03 02:31:29.176382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:29.176218 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb2bd860-6aa6-4f34-a90a-d599c74c4cba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb2bd860-6aa6-4f34-a90a-d599c74c4cba-workload-socket\") pod \"fb2bd860-6aa6-4f34-a90a-d599c74c4cba\" (UID: \"fb2bd860-6aa6-4f34-a90a-d599c74c4cba\") " Nov 03 02:31:29.176382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:29.176298 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb2bd860-6aa6-4f34-a90a-d599c74c4cba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb2bd860-6aa6-4f34-a90a-d599c74c4cba-credential-socket\") pod \"fb2bd860-6aa6-4f34-a90a-d599c74c4cba\" (UID: \"fb2bd860-6aa6-4f34-a90a-d599c74c4cba\") " Nov 03 02:31:29.181381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:29.177072 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb2bd860-6aa6-4f34-a90a-d599c74c4cba-credential-socket" pod "fb2bd860-6aa6-4f34-a90a-d599c74c4cba" (UID: "fb2bd860-6aa6-4f34-a90a-d599c74c4cba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:31:29.181381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:29.177468 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb2bd860-6aa6-4f34-a90a-d599c74c4cba-istiod-ca-cert" pod "fb2bd860-6aa6-4f34-a90a-d599c74c4cba" (UID: "fb2bd860-6aa6-4f34-a90a-d599c74c4cba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:31:29.181381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:29.177712 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb2bd860-6aa6-4f34-a90a-d599c74c4cba-workload-certs" pod "fb2bd860-6aa6-4f34-a90a-d599c74c4cba" (UID: "fb2bd860-6aa6-4f34-a90a-d599c74c4cba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:31:29.181381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:29.178527 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb2bd860-6aa6-4f34-a90a-d599c74c4cba-istio-data" pod "fb2bd860-6aa6-4f34-a90a-d599c74c4cba" (UID: "fb2bd860-6aa6-4f34-a90a-d599c74c4cba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:31:29.182000 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:29.181972 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb2bd860-6aa6-4f34-a90a-d599c74c4cba-workload-socket" pod "fb2bd860-6aa6-4f34-a90a-d599c74c4cba" (UID: "fb2bd860-6aa6-4f34-a90a-d599c74c4cba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:31:29.192595 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:29.192548 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb2bd860-6aa6-4f34-a90a-d599c74c4cba-kube-api-access-tbkdq" pod "fb2bd860-6aa6-4f34-a90a-d599c74c4cba" (UID: "fb2bd860-6aa6-4f34-a90a-d599c74c4cba"). InnerVolumeSpecName "kube-api-access-tbkdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:31:29.192735 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:29.192602 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb2bd860-6aa6-4f34-a90a-d599c74c4cba-istio-token" pod "fb2bd860-6aa6-4f34-a90a-d599c74c4cba" (UID: "fb2bd860-6aa6-4f34-a90a-d599c74c4cba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:31:29.192893 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:29.192847 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb2bd860-6aa6-4f34-a90a-d599c74c4cba-istio-envoy" pod "fb2bd860-6aa6-4f34-a90a-d599c74c4cba" (UID: "fb2bd860-6aa6-4f34-a90a-d599c74c4cba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:31:29.193724 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:29.193699 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb2bd860-6aa6-4f34-a90a-d599c74c4cba-istio-podinfo" pod "fb2bd860-6aa6-4f34-a90a-d599c74c4cba" (UID: "fb2bd860-6aa6-4f34-a90a-d599c74c4cba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:31:29.219158 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:29.219126 5605 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 03 02:31:29.219322 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:29.219177 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw" podUID="b337ace7-57a1-4583-8591-9c2f3040dba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 03 02:31:29.224567 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:29.224536 5605 patch_prober.go:29] interesting pod/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 03 02:31:29.224911 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:29.224589 5605 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 03 02:31:29.224911 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:29.224617 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29" podUID="4846c4da-bf51-483f-8860-dc49d45f5d22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 03 02:31:29.224911 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:29.224612 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw" podUID="229d9be7-d6b9-4760-80ed-84592e5693d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 03 02:31:29.287389 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:29.283842 5605 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb2bd860-6aa6-4f34-a90a-d599c74c4cba-credential-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:29.287389 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:29.283883 5605 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb2bd860-6aa6-4f34-a90a-d599c74c4cba-istiod-ca-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:29.287389 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:29.283904 5605 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb2bd860-6aa6-4f34-a90a-d599c74c4cba-workload-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:29.287389 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:29.283918 5605 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb2bd860-6aa6-4f34-a90a-d599c74c4cba-istio-envoy\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:29.287389 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:29.283934 5605 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb2bd860-6aa6-4f34-a90a-d599c74c4cba-istio-token\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:29.287389 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:29.283948 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbkdq\" (UniqueName: \"kubernetes.io/projected/fb2bd860-6aa6-4f34-a90a-d599c74c4cba-kube-api-access-tbkdq\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:29.287389 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:29.283963 5605 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb2bd860-6aa6-4f34-a90a-d599c74c4cba-istio-data\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:29.287389 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:29.283977 5605 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb2bd860-6aa6-4f34-a90a-d599c74c4cba-istio-podinfo\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:29.287389 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:29.283991 5605 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb2bd860-6aa6-4f34-a90a-d599c74c4cba-workload-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:29.889135 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:29.889093 5605 generic.go:356] "Generic (PLEG): container finished" podID="fb2bd860-6aa6-4f34-a90a-d599c74c4cba" containerID="dc5399d0c525d8ee88da7da8052ae368ccf311b045adc2048badb9d3596b978d" exitCode=0 Nov 03 02:31:29.890513 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:29.890485 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn" Nov 03 02:31:29.909391 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:29.907542 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn" event={"ID":"fb2bd860-6aa6-4f34-a90a-d599c74c4cba","Type":"ContainerDied","Data":"dc5399d0c525d8ee88da7da8052ae368ccf311b045adc2048badb9d3596b978d"} Nov 03 02:31:29.909391 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:29.907629 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn" event={"ID":"fb2bd860-6aa6-4f34-a90a-d599c74c4cba","Type":"ContainerDied","Data":"eda378f618aabd3f9c116d0e649f320bc085d69d858b9f42ab1fa40565408050"} Nov 03 02:31:29.909391 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:29.907662 5605 scope.go:122] "RemoveContainer" containerID="dc5399d0c525d8ee88da7da8052ae368ccf311b045adc2048badb9d3596b978d" Nov 03 02:31:29.962735 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:29.956325 5605 scope.go:122] "RemoveContainer" containerID="5f3cb2962cf3a2c311567b52bdf4f52f4217e88dceb011d141514e2b6096f540" Nov 03 02:31:30.067406 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:30.053564 5605 scope.go:122] "RemoveContainer" containerID="dc5399d0c525d8ee88da7da8052ae368ccf311b045adc2048badb9d3596b978d" Nov 03 02:31:30.067406 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:31:30.056960 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc5399d0c525d8ee88da7da8052ae368ccf311b045adc2048badb9d3596b978d\": container with ID starting with dc5399d0c525d8ee88da7da8052ae368ccf311b045adc2048badb9d3596b978d not found: ID does not exist" containerID="dc5399d0c525d8ee88da7da8052ae368ccf311b045adc2048badb9d3596b978d" Nov 03 02:31:30.067406 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:30.057007 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc5399d0c525d8ee88da7da8052ae368ccf311b045adc2048badb9d3596b978d"} err="failed to get container status \"dc5399d0c525d8ee88da7da8052ae368ccf311b045adc2048badb9d3596b978d\": rpc error: code = NotFound desc = could not find container \"dc5399d0c525d8ee88da7da8052ae368ccf311b045adc2048badb9d3596b978d\": container with ID starting with dc5399d0c525d8ee88da7da8052ae368ccf311b045adc2048badb9d3596b978d not found: ID does not exist" Nov 03 02:31:30.067406 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:30.057034 5605 scope.go:122] "RemoveContainer" containerID="5f3cb2962cf3a2c311567b52bdf4f52f4217e88dceb011d141514e2b6096f540" Nov 03 02:31:30.067406 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:31:30.065576 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f3cb2962cf3a2c311567b52bdf4f52f4217e88dceb011d141514e2b6096f540\": container with ID starting with 5f3cb2962cf3a2c311567b52bdf4f52f4217e88dceb011d141514e2b6096f540 not found: ID does not exist" containerID="5f3cb2962cf3a2c311567b52bdf4f52f4217e88dceb011d141514e2b6096f540" Nov 03 02:31:30.067406 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:30.065623 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f3cb2962cf3a2c311567b52bdf4f52f4217e88dceb011d141514e2b6096f540"} err="failed to get container status \"5f3cb2962cf3a2c311567b52bdf4f52f4217e88dceb011d141514e2b6096f540\": rpc error: code = NotFound desc = could not find container \"5f3cb2962cf3a2c311567b52bdf4f52f4217e88dceb011d141514e2b6096f540\": container with ID starting with 5f3cb2962cf3a2c311567b52bdf4f52f4217e88dceb011d141514e2b6096f540 not found: ID does not exist" Nov 03 02:31:30.123520 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:30.123479 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn"] Nov 03 02:31:30.125950 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:30.125924 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-496177-openshift-default-5bd7448c65-zvptn"] Nov 03 02:31:30.223778 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:30.223673 5605 patch_prober.go:29] interesting pod/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 03 02:31:30.223778 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:30.223745 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw" podUID="229d9be7-d6b9-4760-80ed-84592e5693d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 03 02:31:30.277248 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:30.277208 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw" Nov 03 02:31:30.280088 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:30.280062 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw" Nov 03 02:31:30.291780 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:30.291680 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw" Nov 03 02:31:30.323627 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:30.323591 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29" Nov 03 02:31:30.324426 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:30.324402 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29" Nov 03 02:31:30.325455 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:30.325431 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29" Nov 03 02:31:30.852061 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:30.852020 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb2bd860-6aa6-4f34-a90a-d599c74c4cba" path="/var/lib/kubelet/pods/fb2bd860-6aa6-4f34-a90a-d599c74c4cba/volumes" Nov 03 02:31:31.224144 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:31.224054 5605 patch_prober.go:29] interesting pod/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 03 02:31:31.224144 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:31.224114 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw" podUID="229d9be7-d6b9-4760-80ed-84592e5693d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 03 02:31:32.223737 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:32.223695 5605 patch_prober.go:29] interesting pod/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 03 02:31:32.224143 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:32.223752 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw" podUID="229d9be7-d6b9-4760-80ed-84592e5693d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 03 02:31:33.327126 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:33.327072 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw" Nov 03 02:31:33.327665 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:33.327477 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw" Nov 03 02:31:33.329257 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:33.329232 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw" Nov 03 02:31:36.869422 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:36.869382 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a035c29d-gphwj"] Nov 03 02:31:36.869842 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:36.869792 5605 cpu_manager.go:414] "containerMap: removing container" podUID="fb2bd860-6aa6-4f34-a90a-d599c74c4cba" containerName="istio-proxy" Nov 03 02:31:36.869842 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:36.869802 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb2bd860-6aa6-4f34-a90a-d599c74c4cba" containerName="istio-proxy" Nov 03 02:31:36.869935 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:36.869902 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb2bd860-6aa6-4f34-a90a-d599c74c4cba" containerName="istio-proxy" Nov 03 02:31:36.869935 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:36.869911 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb2bd860-6aa6-4f34-a90a-d599c74c4cba" containerName="istio-proxy" Nov 03 02:31:36.874548 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:36.874528 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a035c29d-gphwj" Nov 03 02:31:36.903719 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:36.903690 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7j7v\" (UniqueName: \"kubernetes.io/projected/d8f69569-01e2-4398-9ba7-de111b956ea0-kube-api-access-f7j7v\") pod \"maas-api-revoke-sub-a035c29d-gphwj\" (UID: \"d8f69569-01e2-4398-9ba7-de111b956ea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a035c29d-gphwj" Nov 03 02:31:36.903869 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:36.903743 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8f69569-01e2-4398-9ba7-de111b956ea0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a035c29d-gphwj\" (UID: \"d8f69569-01e2-4398-9ba7-de111b956ea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a035c29d-gphwj" Nov 03 02:31:36.904013 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:36.903988 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8f69569-01e2-4398-9ba7-de111b956ea0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a035c29d-gphwj\" (UID: \"d8f69569-01e2-4398-9ba7-de111b956ea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a035c29d-gphwj" Nov 03 02:31:36.926876 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:36.926847 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a035c29d-gphwj"] Nov 03 02:31:36.970185 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:36.970151 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:31:36.970390 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:36.970214 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:31:36.970390 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:36.970279 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:31:36.970390 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:36.970295 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:31:37.005128 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:37.005094 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8f69569-01e2-4398-9ba7-de111b956ea0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a035c29d-gphwj\" (UID: \"d8f69569-01e2-4398-9ba7-de111b956ea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a035c29d-gphwj" Nov 03 02:31:37.005259 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:37.005171 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7j7v\" (UniqueName: \"kubernetes.io/projected/d8f69569-01e2-4398-9ba7-de111b956ea0-kube-api-access-f7j7v\") pod \"maas-api-revoke-sub-a035c29d-gphwj\" (UID: \"d8f69569-01e2-4398-9ba7-de111b956ea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a035c29d-gphwj" Nov 03 02:31:37.005259 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:37.005203 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8f69569-01e2-4398-9ba7-de111b956ea0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a035c29d-gphwj\" (UID: \"d8f69569-01e2-4398-9ba7-de111b956ea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a035c29d-gphwj" Nov 03 02:31:37.005889 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:37.005858 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8f69569-01e2-4398-9ba7-de111b956ea0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a035c29d-gphwj\" (UID: \"d8f69569-01e2-4398-9ba7-de111b956ea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a035c29d-gphwj" Nov 03 02:31:37.007583 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:37.007560 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8f69569-01e2-4398-9ba7-de111b956ea0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a035c29d-gphwj\" (UID: \"d8f69569-01e2-4398-9ba7-de111b956ea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a035c29d-gphwj" Nov 03 02:31:37.058250 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:37.058223 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7j7v\" (UniqueName: \"kubernetes.io/projected/d8f69569-01e2-4398-9ba7-de111b956ea0-kube-api-access-f7j7v\") pod \"maas-api-revoke-sub-a035c29d-gphwj\" (UID: \"d8f69569-01e2-4398-9ba7-de111b956ea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a035c29d-gphwj" Nov 03 02:31:37.187186 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:37.187101 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a035c29d-gphwj" Nov 03 02:31:37.555463 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:37.555430 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a035c29d-gphwj"] Nov 03 02:31:37.560518 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:31:37.560490 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8f69569_01e2_4398_9ba7_de111b956ea0.slice/crio-9f6399fd2cc83b11b86f346d9da59da7a55c8c400b65169ae2e6a765563f89ff WatchSource:0}: Error finding container 9f6399fd2cc83b11b86f346d9da59da7a55c8c400b65169ae2e6a765563f89ff: Status 404 returned error can't find the container with id 9f6399fd2cc83b11b86f346d9da59da7a55c8c400b65169ae2e6a765563f89ff Nov 03 02:31:37.939008 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:37.938975 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a035c29d-gphwj" event={"ID":"d8f69569-01e2-4398-9ba7-de111b956ea0","Type":"ContainerStarted","Data":"5bfbd14e91f5154f28a4a9ba1ab9b1ad252553ae81038091f2d26e9735661cec"} Nov 03 02:31:37.939380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:37.939029 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a035c29d-gphwj" event={"ID":"d8f69569-01e2-4398-9ba7-de111b956ea0","Type":"ContainerStarted","Data":"9f6399fd2cc83b11b86f346d9da59da7a55c8c400b65169ae2e6a765563f89ff"} Nov 03 02:31:38.046996 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:38.046934 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a035c29d-gphwj" podStartSLOduration=2.046918564 podStartE2EDuration="2.046918564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:31:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:31:38.045437869 +0000 UTC m=+2375.304636019" watchObservedRunningTime="2026-11-03 02:31:38.046918564 +0000 UTC m=+2375.306116719" Nov 03 02:31:38.946863 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:38.946824 5605 generic.go:356] "Generic (PLEG): container finished" podID="d8f69569-01e2-4398-9ba7-de111b956ea0" containerID="5bfbd14e91f5154f28a4a9ba1ab9b1ad252553ae81038091f2d26e9735661cec" exitCode=0 Nov 03 02:31:38.947314 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:38.946876 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a035c29d-gphwj" event={"ID":"d8f69569-01e2-4398-9ba7-de111b956ea0","Type":"ContainerDied","Data":"5bfbd14e91f5154f28a4a9ba1ab9b1ad252553ae81038091f2d26e9735661cec"} Nov 03 02:31:39.786719 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:39.786683 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-976aed18-vqpgt"] Nov 03 02:31:39.787115 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:39.787103 5605 cpu_manager.go:414] "containerMap: removing container" podUID="fb2bd860-6aa6-4f34-a90a-d599c74c4cba" containerName="istio-proxy" Nov 03 02:31:39.787167 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:39.787116 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb2bd860-6aa6-4f34-a90a-d599c74c4cba" containerName="istio-proxy" Nov 03 02:31:39.790465 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:39.790444 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-976aed18-vqpgt" Nov 03 02:31:39.835174 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:39.835146 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a98cc88-77ca-4387-9d4c-62d907bdf21f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-976aed18-vqpgt\" (UID: \"8a98cc88-77ca-4387-9d4c-62d907bdf21f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-976aed18-vqpgt" Nov 03 02:31:39.835396 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:39.835374 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs6gd\" (UniqueName: \"kubernetes.io/projected/8a98cc88-77ca-4387-9d4c-62d907bdf21f-kube-api-access-qs6gd\") pod \"maas-api-revoke-sub-976aed18-vqpgt\" (UID: \"8a98cc88-77ca-4387-9d4c-62d907bdf21f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-976aed18-vqpgt" Nov 03 02:31:39.835454 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:39.835419 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a98cc88-77ca-4387-9d4c-62d907bdf21f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-976aed18-vqpgt\" (UID: \"8a98cc88-77ca-4387-9d4c-62d907bdf21f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-976aed18-vqpgt" Nov 03 02:31:39.884139 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:39.884115 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-976aed18-vqpgt"] Nov 03 02:31:39.936619 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:39.936584 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs6gd\" (UniqueName: \"kubernetes.io/projected/8a98cc88-77ca-4387-9d4c-62d907bdf21f-kube-api-access-qs6gd\") pod \"maas-api-revoke-sub-976aed18-vqpgt\" (UID: \"8a98cc88-77ca-4387-9d4c-62d907bdf21f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-976aed18-vqpgt" Nov 03 02:31:39.936788 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:39.936634 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a98cc88-77ca-4387-9d4c-62d907bdf21f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-976aed18-vqpgt\" (UID: \"8a98cc88-77ca-4387-9d4c-62d907bdf21f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-976aed18-vqpgt" Nov 03 02:31:39.936788 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:39.936688 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a98cc88-77ca-4387-9d4c-62d907bdf21f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-976aed18-vqpgt\" (UID: \"8a98cc88-77ca-4387-9d4c-62d907bdf21f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-976aed18-vqpgt" Nov 03 02:31:39.937406 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:39.937383 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a98cc88-77ca-4387-9d4c-62d907bdf21f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-976aed18-vqpgt\" (UID: \"8a98cc88-77ca-4387-9d4c-62d907bdf21f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-976aed18-vqpgt" Nov 03 02:31:39.938980 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:39.938959 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a98cc88-77ca-4387-9d4c-62d907bdf21f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-976aed18-vqpgt\" (UID: \"8a98cc88-77ca-4387-9d4c-62d907bdf21f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-976aed18-vqpgt" Nov 03 02:31:39.989749 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:39.989721 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs6gd\" (UniqueName: \"kubernetes.io/projected/8a98cc88-77ca-4387-9d4c-62d907bdf21f-kube-api-access-qs6gd\") pod \"maas-api-revoke-sub-976aed18-vqpgt\" (UID: \"8a98cc88-77ca-4387-9d4c-62d907bdf21f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-976aed18-vqpgt" Nov 03 02:31:40.092345 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:40.092313 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a035c29d-gphwj" Nov 03 02:31:40.103245 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:40.103226 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-976aed18-vqpgt" Nov 03 02:31:40.139791 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:40.139762 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8f69569-01e2-4398-9ba7-de111b956ea0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8f69569-01e2-4398-9ba7-de111b956ea0-maas-api-xks-ca\") pod \"d8f69569-01e2-4398-9ba7-de111b956ea0\" (UID: \"d8f69569-01e2-4398-9ba7-de111b956ea0\") " Nov 03 02:31:40.139942 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:40.139891 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8f69569-01e2-4398-9ba7-de111b956ea0-kube-api-access-f7j7v\" (UniqueName: \"kubernetes.io/projected/d8f69569-01e2-4398-9ba7-de111b956ea0-kube-api-access-f7j7v\") pod \"d8f69569-01e2-4398-9ba7-de111b956ea0\" (UID: \"d8f69569-01e2-4398-9ba7-de111b956ea0\") " Nov 03 02:31:40.139942 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:40.139936 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8f69569-01e2-4398-9ba7-de111b956ea0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8f69569-01e2-4398-9ba7-de111b956ea0-maas-api-openshift-ca\") pod \"d8f69569-01e2-4398-9ba7-de111b956ea0\" (UID: \"d8f69569-01e2-4398-9ba7-de111b956ea0\") " Nov 03 02:31:40.140277 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:40.140251 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8f69569-01e2-4398-9ba7-de111b956ea0-maas-api-openshift-ca" pod "d8f69569-01e2-4398-9ba7-de111b956ea0" (UID: "d8f69569-01e2-4398-9ba7-de111b956ea0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:31:40.140485 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:40.140468 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8f69569-01e2-4398-9ba7-de111b956ea0-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:40.141856 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:40.141829 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8f69569-01e2-4398-9ba7-de111b956ea0-maas-api-xks-ca" pod "d8f69569-01e2-4398-9ba7-de111b956ea0" (UID: "d8f69569-01e2-4398-9ba7-de111b956ea0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:31:40.142287 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:40.142250 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8f69569-01e2-4398-9ba7-de111b956ea0-kube-api-access-f7j7v" pod "d8f69569-01e2-4398-9ba7-de111b956ea0" (UID: "d8f69569-01e2-4398-9ba7-de111b956ea0"). InnerVolumeSpecName "kube-api-access-f7j7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:31:40.241100 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:40.241067 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7j7v\" (UniqueName: \"kubernetes.io/projected/d8f69569-01e2-4398-9ba7-de111b956ea0-kube-api-access-f7j7v\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:40.241100 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:40.241094 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8f69569-01e2-4398-9ba7-de111b956ea0-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:40.473021 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:40.472941 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-976aed18-vqpgt"] Nov 03 02:31:40.478916 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:31:40.478881 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a98cc88_77ca_4387_9d4c_62d907bdf21f.slice/crio-a7bd17e98c037b7d2aed485d6fdd10e5b8f781e30dbadf13bf45f46e5c781d53 WatchSource:0}: Error finding container a7bd17e98c037b7d2aed485d6fdd10e5b8f781e30dbadf13bf45f46e5c781d53: Status 404 returned error can't find the container with id a7bd17e98c037b7d2aed485d6fdd10e5b8f781e30dbadf13bf45f46e5c781d53 Nov 03 02:31:40.958252 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:40.958227 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a035c29d-gphwj" Nov 03 02:31:40.958407 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:40.958262 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a035c29d-gphwj" event={"ID":"d8f69569-01e2-4398-9ba7-de111b956ea0","Type":"ContainerDied","Data":"9f6399fd2cc83b11b86f346d9da59da7a55c8c400b65169ae2e6a765563f89ff"} Nov 03 02:31:40.958407 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:40.958318 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f6399fd2cc83b11b86f346d9da59da7a55c8c400b65169ae2e6a765563f89ff" Nov 03 02:31:40.960037 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:40.960008 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-976aed18-vqpgt" event={"ID":"8a98cc88-77ca-4387-9d4c-62d907bdf21f","Type":"ContainerStarted","Data":"10c1eb9e29a74a9065672924559b5dc1dad22509031dfc9d55f54feb9b236d98"} Nov 03 02:31:40.960166 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:40.960051 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-976aed18-vqpgt" event={"ID":"8a98cc88-77ca-4387-9d4c-62d907bdf21f","Type":"ContainerStarted","Data":"a7bd17e98c037b7d2aed485d6fdd10e5b8f781e30dbadf13bf45f46e5c781d53"} Nov 03 02:31:41.068037 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:41.067996 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-976aed18-vqpgt" podStartSLOduration=2.067982754 podStartE2EDuration="2.067982754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:31:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:31:41.065392854 +0000 UTC m=+2378.324591008" watchObservedRunningTime="2026-11-03 02:31:41.067982754 +0000 UTC m=+2378.327180949" Nov 03 02:31:41.967179 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:41.967144 5605 generic.go:356] "Generic (PLEG): container finished" podID="8a98cc88-77ca-4387-9d4c-62d907bdf21f" containerID="10c1eb9e29a74a9065672924559b5dc1dad22509031dfc9d55f54feb9b236d98" exitCode=0 Nov 03 02:31:41.967370 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:41.967191 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-976aed18-vqpgt" event={"ID":"8a98cc88-77ca-4387-9d4c-62d907bdf21f","Type":"ContainerDied","Data":"10c1eb9e29a74a9065672924559b5dc1dad22509031dfc9d55f54feb9b236d98"} Nov 03 02:31:43.104503 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:43.104471 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-976aed18-vqpgt" Nov 03 02:31:43.171295 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:43.171261 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a98cc88-77ca-4387-9d4c-62d907bdf21f-kube-api-access-qs6gd\" (UniqueName: \"kubernetes.io/projected/8a98cc88-77ca-4387-9d4c-62d907bdf21f-kube-api-access-qs6gd\") pod \"8a98cc88-77ca-4387-9d4c-62d907bdf21f\" (UID: \"8a98cc88-77ca-4387-9d4c-62d907bdf21f\") " Nov 03 02:31:43.171460 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:43.171341 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a98cc88-77ca-4387-9d4c-62d907bdf21f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a98cc88-77ca-4387-9d4c-62d907bdf21f-maas-api-openshift-ca\") pod \"8a98cc88-77ca-4387-9d4c-62d907bdf21f\" (UID: \"8a98cc88-77ca-4387-9d4c-62d907bdf21f\") " Nov 03 02:31:43.171514 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:43.171491 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a98cc88-77ca-4387-9d4c-62d907bdf21f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a98cc88-77ca-4387-9d4c-62d907bdf21f-maas-api-xks-ca\") pod \"8a98cc88-77ca-4387-9d4c-62d907bdf21f\" (UID: \"8a98cc88-77ca-4387-9d4c-62d907bdf21f\") " Nov 03 02:31:43.171692 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:43.171670 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a98cc88-77ca-4387-9d4c-62d907bdf21f-maas-api-openshift-ca" pod "8a98cc88-77ca-4387-9d4c-62d907bdf21f" (UID: "8a98cc88-77ca-4387-9d4c-62d907bdf21f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:31:43.171968 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:43.171940 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a98cc88-77ca-4387-9d4c-62d907bdf21f-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:43.173496 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:43.173464 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a98cc88-77ca-4387-9d4c-62d907bdf21f-maas-api-xks-ca" pod "8a98cc88-77ca-4387-9d4c-62d907bdf21f" (UID: "8a98cc88-77ca-4387-9d4c-62d907bdf21f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:31:43.173496 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:43.173473 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a98cc88-77ca-4387-9d4c-62d907bdf21f-kube-api-access-qs6gd" pod "8a98cc88-77ca-4387-9d4c-62d907bdf21f" (UID: "8a98cc88-77ca-4387-9d4c-62d907bdf21f"). InnerVolumeSpecName "kube-api-access-qs6gd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:31:43.273235 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:43.273159 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qs6gd\" (UniqueName: \"kubernetes.io/projected/8a98cc88-77ca-4387-9d4c-62d907bdf21f-kube-api-access-qs6gd\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:43.273235 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:43.273191 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a98cc88-77ca-4387-9d4c-62d907bdf21f-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:43.980263 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:43.980225 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-976aed18-vqpgt" event={"ID":"8a98cc88-77ca-4387-9d4c-62d907bdf21f","Type":"ContainerDied","Data":"a7bd17e98c037b7d2aed485d6fdd10e5b8f781e30dbadf13bf45f46e5c781d53"} Nov 03 02:31:43.980493 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:43.980283 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7bd17e98c037b7d2aed485d6fdd10e5b8f781e30dbadf13bf45f46e5c781d53" Nov 03 02:31:43.980493 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:43.980237 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-976aed18-vqpgt" Nov 03 02:31:46.239161 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:46.239130 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:31:47.817384 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:47.816080 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff928274-bshwh"] Nov 03 02:31:47.817384 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:47.816998 5605 cpu_manager.go:414] "containerMap: removing container" podUID="d8f69569-01e2-4398-9ba7-de111b956ea0" containerName="revoke-keys" Nov 03 02:31:47.817384 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:47.817020 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8f69569-01e2-4398-9ba7-de111b956ea0" containerName="revoke-keys" Nov 03 02:31:47.817384 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:47.817079 5605 cpu_manager.go:414] "containerMap: removing container" podUID="8a98cc88-77ca-4387-9d4c-62d907bdf21f" containerName="revoke-keys" Nov 03 02:31:47.817384 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:47.817089 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a98cc88-77ca-4387-9d4c-62d907bdf21f" containerName="revoke-keys" Nov 03 02:31:47.817384 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:47.817229 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a98cc88-77ca-4387-9d4c-62d907bdf21f" containerName="revoke-keys" Nov 03 02:31:47.817384 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:47.817246 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8f69569-01e2-4398-9ba7-de111b956ea0" containerName="revoke-keys" Nov 03 02:31:47.833713 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:47.833679 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff928274-bshwh" Nov 03 02:31:47.995381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:47.989745 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:31:47.995381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:47.990447 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:31:47.995381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:47.990529 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:31:47.995381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:47.990616 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:31:48.016380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:47.997985 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff928274-bshwh"] Nov 03 02:31:48.119443 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:48.111239 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e55dcb1-6255-441c-8f77-85b05e14727d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff928274-bshwh\" (UID: \"5e55dcb1-6255-441c-8f77-85b05e14727d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff928274-bshwh" Nov 03 02:31:48.119443 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:48.111391 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e55dcb1-6255-441c-8f77-85b05e14727d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff928274-bshwh\" (UID: \"5e55dcb1-6255-441c-8f77-85b05e14727d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff928274-bshwh" Nov 03 02:31:48.119443 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:48.111425 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbxzz\" (UniqueName: \"kubernetes.io/projected/5e55dcb1-6255-441c-8f77-85b05e14727d-kube-api-access-qbxzz\") pod \"maas-api-revoke-sub-ff928274-bshwh\" (UID: \"5e55dcb1-6255-441c-8f77-85b05e14727d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff928274-bshwh" Nov 03 02:31:48.238775 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:48.238737 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e55dcb1-6255-441c-8f77-85b05e14727d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff928274-bshwh\" (UID: \"5e55dcb1-6255-441c-8f77-85b05e14727d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff928274-bshwh" Nov 03 02:31:48.239909 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:48.239767 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e55dcb1-6255-441c-8f77-85b05e14727d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff928274-bshwh\" (UID: \"5e55dcb1-6255-441c-8f77-85b05e14727d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff928274-bshwh" Nov 03 02:31:48.240348 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:48.239840 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbxzz\" (UniqueName: \"kubernetes.io/projected/5e55dcb1-6255-441c-8f77-85b05e14727d-kube-api-access-qbxzz\") pod \"maas-api-revoke-sub-ff928274-bshwh\" (UID: \"5e55dcb1-6255-441c-8f77-85b05e14727d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff928274-bshwh" Nov 03 02:31:48.265377 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:48.240699 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e55dcb1-6255-441c-8f77-85b05e14727d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff928274-bshwh\" (UID: \"5e55dcb1-6255-441c-8f77-85b05e14727d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff928274-bshwh" Nov 03 02:31:48.353423 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:48.351324 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbxzz\" (UniqueName: \"kubernetes.io/projected/5e55dcb1-6255-441c-8f77-85b05e14727d-kube-api-access-qbxzz\") pod \"maas-api-revoke-sub-ff928274-bshwh\" (UID: \"5e55dcb1-6255-441c-8f77-85b05e14727d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff928274-bshwh" Nov 03 02:31:48.376423 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:48.356743 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e55dcb1-6255-441c-8f77-85b05e14727d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff928274-bshwh\" (UID: \"5e55dcb1-6255-441c-8f77-85b05e14727d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff928274-bshwh" Nov 03 02:31:48.467533 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:48.467495 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff928274-bshwh" Nov 03 02:31:49.190402 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:49.190307 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff928274-bshwh"] Nov 03 02:31:49.195714 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:31:49.195689 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e55dcb1_6255_441c_8f77_85b05e14727d.slice/crio-8b877c1cb2f7d987bd593372cf7081083a42ee7b9b47438d06ed69543bcead7c WatchSource:0}: Error finding container 8b877c1cb2f7d987bd593372cf7081083a42ee7b9b47438d06ed69543bcead7c: Status 404 returned error can't find the container with id 8b877c1cb2f7d987bd593372cf7081083a42ee7b9b47438d06ed69543bcead7c Nov 03 02:31:50.115915 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:50.115880 5605 generic.go:356] "Generic (PLEG): container finished" podID="5e55dcb1-6255-441c-8f77-85b05e14727d" containerID="1ef1c328f11169a92e3a3ad2a339b66b2f4173c733fdf688c75a1c5768797b00" exitCode=0 Nov 03 02:31:50.116103 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:50.115966 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff928274-bshwh" event={"ID":"5e55dcb1-6255-441c-8f77-85b05e14727d","Type":"ContainerDied","Data":"1ef1c328f11169a92e3a3ad2a339b66b2f4173c733fdf688c75a1c5768797b00"} Nov 03 02:31:50.116103 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:50.116035 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff928274-bshwh" event={"ID":"5e55dcb1-6255-441c-8f77-85b05e14727d","Type":"ContainerStarted","Data":"8b877c1cb2f7d987bd593372cf7081083a42ee7b9b47438d06ed69543bcead7c"} Nov 03 02:31:51.270129 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:51.270104 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff928274-bshwh" Nov 03 02:31:51.349802 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:51.349777 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e55dcb1-6255-441c-8f77-85b05e14727d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e55dcb1-6255-441c-8f77-85b05e14727d-maas-api-xks-ca\") pod \"5e55dcb1-6255-441c-8f77-85b05e14727d\" (UID: \"5e55dcb1-6255-441c-8f77-85b05e14727d\") " Nov 03 02:31:51.349955 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:51.349843 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e55dcb1-6255-441c-8f77-85b05e14727d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e55dcb1-6255-441c-8f77-85b05e14727d-maas-api-openshift-ca\") pod \"5e55dcb1-6255-441c-8f77-85b05e14727d\" (UID: \"5e55dcb1-6255-441c-8f77-85b05e14727d\") " Nov 03 02:31:51.350027 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:51.349950 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e55dcb1-6255-441c-8f77-85b05e14727d-kube-api-access-qbxzz\" (UniqueName: \"kubernetes.io/projected/5e55dcb1-6255-441c-8f77-85b05e14727d-kube-api-access-qbxzz\") pod \"5e55dcb1-6255-441c-8f77-85b05e14727d\" (UID: \"5e55dcb1-6255-441c-8f77-85b05e14727d\") " Nov 03 02:31:51.350198 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:51.350172 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e55dcb1-6255-441c-8f77-85b05e14727d-maas-api-openshift-ca" pod "5e55dcb1-6255-441c-8f77-85b05e14727d" (UID: "5e55dcb1-6255-441c-8f77-85b05e14727d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:31:51.350446 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:51.350430 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e55dcb1-6255-441c-8f77-85b05e14727d-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:51.352022 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:51.351998 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e55dcb1-6255-441c-8f77-85b05e14727d-kube-api-access-qbxzz" pod "5e55dcb1-6255-441c-8f77-85b05e14727d" (UID: "5e55dcb1-6255-441c-8f77-85b05e14727d"). InnerVolumeSpecName "kube-api-access-qbxzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:31:51.352099 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:51.352003 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e55dcb1-6255-441c-8f77-85b05e14727d-maas-api-xks-ca" pod "5e55dcb1-6255-441c-8f77-85b05e14727d" (UID: "5e55dcb1-6255-441c-8f77-85b05e14727d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:31:51.370110 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:51.370077 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae8e330a-lvnwg"] Nov 03 02:31:51.370750 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:51.370730 5605 cpu_manager.go:414] "containerMap: removing container" podUID="5e55dcb1-6255-441c-8f77-85b05e14727d" containerName="revoke-keys" Nov 03 02:31:51.370867 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:51.370752 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e55dcb1-6255-441c-8f77-85b05e14727d" containerName="revoke-keys" Nov 03 02:31:51.370930 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:51.370903 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e55dcb1-6255-441c-8f77-85b05e14727d" containerName="revoke-keys" Nov 03 02:31:51.374630 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:51.374575 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae8e330a-lvnwg" Nov 03 02:31:51.427428 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:51.427397 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae8e330a-lvnwg"] Nov 03 02:31:51.451875 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:51.451840 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfzjn\" (UniqueName: \"kubernetes.io/projected/85aa8ff4-f330-45e1-bb65-58ea54bbf0df-kube-api-access-mfzjn\") pod \"maas-api-revoke-sub-ae8e330a-lvnwg\" (UID: \"85aa8ff4-f330-45e1-bb65-58ea54bbf0df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae8e330a-lvnwg" Nov 03 02:31:51.452031 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:51.452009 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85aa8ff4-f330-45e1-bb65-58ea54bbf0df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae8e330a-lvnwg\" (UID: \"85aa8ff4-f330-45e1-bb65-58ea54bbf0df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae8e330a-lvnwg" Nov 03 02:31:51.452094 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:51.452056 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85aa8ff4-f330-45e1-bb65-58ea54bbf0df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae8e330a-lvnwg\" (UID: \"85aa8ff4-f330-45e1-bb65-58ea54bbf0df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae8e330a-lvnwg" Nov 03 02:31:51.452143 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:51.452131 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e55dcb1-6255-441c-8f77-85b05e14727d-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:51.452183 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:51.452146 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbxzz\" (UniqueName: \"kubernetes.io/projected/5e55dcb1-6255-441c-8f77-85b05e14727d-kube-api-access-qbxzz\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:51.553539 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:51.553502 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfzjn\" (UniqueName: \"kubernetes.io/projected/85aa8ff4-f330-45e1-bb65-58ea54bbf0df-kube-api-access-mfzjn\") pod \"maas-api-revoke-sub-ae8e330a-lvnwg\" (UID: \"85aa8ff4-f330-45e1-bb65-58ea54bbf0df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae8e330a-lvnwg" Nov 03 02:31:51.553727 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:51.553653 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85aa8ff4-f330-45e1-bb65-58ea54bbf0df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae8e330a-lvnwg\" (UID: \"85aa8ff4-f330-45e1-bb65-58ea54bbf0df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae8e330a-lvnwg" Nov 03 02:31:51.553727 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:51.553702 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85aa8ff4-f330-45e1-bb65-58ea54bbf0df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae8e330a-lvnwg\" (UID: \"85aa8ff4-f330-45e1-bb65-58ea54bbf0df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae8e330a-lvnwg" Nov 03 02:31:51.554510 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:51.554481 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85aa8ff4-f330-45e1-bb65-58ea54bbf0df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae8e330a-lvnwg\" (UID: \"85aa8ff4-f330-45e1-bb65-58ea54bbf0df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae8e330a-lvnwg" Nov 03 02:31:51.556271 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:51.556246 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85aa8ff4-f330-45e1-bb65-58ea54bbf0df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae8e330a-lvnwg\" (UID: \"85aa8ff4-f330-45e1-bb65-58ea54bbf0df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae8e330a-lvnwg" Nov 03 02:31:51.608560 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:51.608510 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfzjn\" (UniqueName: \"kubernetes.io/projected/85aa8ff4-f330-45e1-bb65-58ea54bbf0df-kube-api-access-mfzjn\") pod \"maas-api-revoke-sub-ae8e330a-lvnwg\" (UID: \"85aa8ff4-f330-45e1-bb65-58ea54bbf0df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae8e330a-lvnwg" Nov 03 02:31:51.695861 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:51.695775 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae8e330a-lvnwg" Nov 03 02:31:52.092268 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:52.092225 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae8e330a-lvnwg"] Nov 03 02:31:52.097714 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:31:52.097688 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85aa8ff4_f330_45e1_bb65_58ea54bbf0df.slice/crio-41608f860c5909bd2dd8b1ba13b39bb5201766e302ad526ae4a206d887925b84 WatchSource:0}: Error finding container 41608f860c5909bd2dd8b1ba13b39bb5201766e302ad526ae4a206d887925b84: Status 404 returned error can't find the container with id 41608f860c5909bd2dd8b1ba13b39bb5201766e302ad526ae4a206d887925b84 Nov 03 02:31:52.127940 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:52.127913 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae8e330a-lvnwg" event={"ID":"85aa8ff4-f330-45e1-bb65-58ea54bbf0df","Type":"ContainerStarted","Data":"41608f860c5909bd2dd8b1ba13b39bb5201766e302ad526ae4a206d887925b84"} Nov 03 02:31:52.129197 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:52.129178 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff928274-bshwh" Nov 03 02:31:52.129288 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:52.129217 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff928274-bshwh" event={"ID":"5e55dcb1-6255-441c-8f77-85b05e14727d","Type":"ContainerDied","Data":"8b877c1cb2f7d987bd593372cf7081083a42ee7b9b47438d06ed69543bcead7c"} Nov 03 02:31:52.129288 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:52.129257 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b877c1cb2f7d987bd593372cf7081083a42ee7b9b47438d06ed69543bcead7c" Nov 03 02:31:53.136768 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:53.136738 5605 generic.go:356] "Generic (PLEG): container finished" podID="85aa8ff4-f330-45e1-bb65-58ea54bbf0df" containerID="e1ed4d531758e6eb40403e9c999c8d7c35f1d8524f11f6d623f87f42788a9c5a" exitCode=0 Nov 03 02:31:53.137145 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:53.136816 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae8e330a-lvnwg" event={"ID":"85aa8ff4-f330-45e1-bb65-58ea54bbf0df","Type":"ContainerDied","Data":"e1ed4d531758e6eb40403e9c999c8d7c35f1d8524f11f6d623f87f42788a9c5a"} Nov 03 02:31:53.914792 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:31:53.914730 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:31:54.287534 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:54.287512 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae8e330a-lvnwg" Nov 03 02:31:54.387750 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:54.387726 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85aa8ff4-f330-45e1-bb65-58ea54bbf0df-kube-api-access-mfzjn\" (UniqueName: \"kubernetes.io/projected/85aa8ff4-f330-45e1-bb65-58ea54bbf0df-kube-api-access-mfzjn\") pod \"85aa8ff4-f330-45e1-bb65-58ea54bbf0df\" (UID: \"85aa8ff4-f330-45e1-bb65-58ea54bbf0df\") " Nov 03 02:31:54.387923 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:54.387905 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85aa8ff4-f330-45e1-bb65-58ea54bbf0df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85aa8ff4-f330-45e1-bb65-58ea54bbf0df-maas-api-openshift-ca\") pod \"85aa8ff4-f330-45e1-bb65-58ea54bbf0df\" (UID: \"85aa8ff4-f330-45e1-bb65-58ea54bbf0df\") " Nov 03 02:31:54.387980 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:54.387948 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85aa8ff4-f330-45e1-bb65-58ea54bbf0df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85aa8ff4-f330-45e1-bb65-58ea54bbf0df-maas-api-xks-ca\") pod \"85aa8ff4-f330-45e1-bb65-58ea54bbf0df\" (UID: \"85aa8ff4-f330-45e1-bb65-58ea54bbf0df\") " Nov 03 02:31:54.388269 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:54.388241 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85aa8ff4-f330-45e1-bb65-58ea54bbf0df-maas-api-openshift-ca" pod "85aa8ff4-f330-45e1-bb65-58ea54bbf0df" (UID: "85aa8ff4-f330-45e1-bb65-58ea54bbf0df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:31:54.388412 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:54.388393 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85aa8ff4-f330-45e1-bb65-58ea54bbf0df-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:54.389809 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:54.389776 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85aa8ff4-f330-45e1-bb65-58ea54bbf0df-kube-api-access-mfzjn" pod "85aa8ff4-f330-45e1-bb65-58ea54bbf0df" (UID: "85aa8ff4-f330-45e1-bb65-58ea54bbf0df"). InnerVolumeSpecName "kube-api-access-mfzjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:31:54.389896 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:54.389827 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85aa8ff4-f330-45e1-bb65-58ea54bbf0df-maas-api-xks-ca" pod "85aa8ff4-f330-45e1-bb65-58ea54bbf0df" (UID: "85aa8ff4-f330-45e1-bb65-58ea54bbf0df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:31:54.489817 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:54.489755 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85aa8ff4-f330-45e1-bb65-58ea54bbf0df-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:54.489817 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:54.489780 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfzjn\" (UniqueName: \"kubernetes.io/projected/85aa8ff4-f330-45e1-bb65-58ea54bbf0df-kube-api-access-mfzjn\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:55.149101 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:55.149022 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae8e330a-lvnwg" Nov 03 02:31:55.149101 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:55.149058 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae8e330a-lvnwg" event={"ID":"85aa8ff4-f330-45e1-bb65-58ea54bbf0df","Type":"ContainerDied","Data":"41608f860c5909bd2dd8b1ba13b39bb5201766e302ad526ae4a206d887925b84"} Nov 03 02:31:55.149289 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:31:55.149127 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41608f860c5909bd2dd8b1ba13b39bb5201766e302ad526ae4a206d887925b84" Nov 03 02:32:04.634902 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:04.634859 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-711318-5967965975-wf4df"] Nov 03 02:32:09.635706 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:09.635662 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-711318-5967965975-wf4df" podUID="15cdb9a2-9888-463a-85c0-a2b70d760b5d" containerName="payload-pre-processing" containerID="cri-o://7463dcf8859ed35c39434318b9dd77fd79245d9e61e29752c2fb6fdbe15deee1" gracePeriod=25 Nov 03 02:32:09.898083 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:09.898013 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-711318-5967965975-wf4df" Nov 03 02:32:10.061406 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:10.061351 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15cdb9a2-9888-463a-85c0-a2b70d760b5d-kube-api-access-89lv4\" (UniqueName: \"kubernetes.io/projected/15cdb9a2-9888-463a-85c0-a2b70d760b5d-kube-api-access-89lv4\") pod \"15cdb9a2-9888-463a-85c0-a2b70d760b5d\" (UID: \"15cdb9a2-9888-463a-85c0-a2b70d760b5d\") " Nov 03 02:32:10.061590 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:10.061453 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15cdb9a2-9888-463a-85c0-a2b70d760b5d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15cdb9a2-9888-463a-85c0-a2b70d760b5d-plugins-config-volume\") pod \"15cdb9a2-9888-463a-85c0-a2b70d760b5d\" (UID: \"15cdb9a2-9888-463a-85c0-a2b70d760b5d\") " Nov 03 02:32:10.061877 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:10.061845 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15cdb9a2-9888-463a-85c0-a2b70d760b5d-plugins-config-volume" pod "15cdb9a2-9888-463a-85c0-a2b70d760b5d" (UID: "15cdb9a2-9888-463a-85c0-a2b70d760b5d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:32:10.063558 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:10.063532 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15cdb9a2-9888-463a-85c0-a2b70d760b5d-kube-api-access-89lv4" pod "15cdb9a2-9888-463a-85c0-a2b70d760b5d" (UID: "15cdb9a2-9888-463a-85c0-a2b70d760b5d"). InnerVolumeSpecName "kube-api-access-89lv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:32:10.162613 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:10.162539 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89lv4\" (UniqueName: \"kubernetes.io/projected/15cdb9a2-9888-463a-85c0-a2b70d760b5d-kube-api-access-89lv4\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:32:10.162613 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:10.162575 5605 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15cdb9a2-9888-463a-85c0-a2b70d760b5d-plugins-config-volume\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:32:10.237545 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:10.237507 5605 generic.go:356] "Generic (PLEG): container finished" podID="15cdb9a2-9888-463a-85c0-a2b70d760b5d" containerID="7463dcf8859ed35c39434318b9dd77fd79245d9e61e29752c2fb6fdbe15deee1" exitCode=0 Nov 03 02:32:10.237679 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:10.237553 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-711318-5967965975-wf4df" Nov 03 02:32:10.237679 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:10.237584 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-711318-5967965975-wf4df" event={"ID":"15cdb9a2-9888-463a-85c0-a2b70d760b5d","Type":"ContainerDied","Data":"7463dcf8859ed35c39434318b9dd77fd79245d9e61e29752c2fb6fdbe15deee1"} Nov 03 02:32:10.237679 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:10.237643 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-711318-5967965975-wf4df" event={"ID":"15cdb9a2-9888-463a-85c0-a2b70d760b5d","Type":"ContainerDied","Data":"9e77f0b376003419b3e1ab87dcfe5615be86c179c50a4e01b75c0561974ddbbb"} Nov 03 02:32:10.237679 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:10.237666 5605 scope.go:122] "RemoveContainer" containerID="7463dcf8859ed35c39434318b9dd77fd79245d9e61e29752c2fb6fdbe15deee1" Nov 03 02:32:10.247108 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:10.247088 5605 scope.go:122] "RemoveContainer" containerID="7463dcf8859ed35c39434318b9dd77fd79245d9e61e29752c2fb6fdbe15deee1" Nov 03 02:32:10.247408 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:32:10.247351 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7463dcf8859ed35c39434318b9dd77fd79245d9e61e29752c2fb6fdbe15deee1\": container with ID starting with 7463dcf8859ed35c39434318b9dd77fd79245d9e61e29752c2fb6fdbe15deee1 not found: ID does not exist" containerID="7463dcf8859ed35c39434318b9dd77fd79245d9e61e29752c2fb6fdbe15deee1" Nov 03 02:32:10.247482 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:10.247418 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7463dcf8859ed35c39434318b9dd77fd79245d9e61e29752c2fb6fdbe15deee1"} err="failed to get container status \"7463dcf8859ed35c39434318b9dd77fd79245d9e61e29752c2fb6fdbe15deee1\": rpc error: code = NotFound desc = could not find container \"7463dcf8859ed35c39434318b9dd77fd79245d9e61e29752c2fb6fdbe15deee1\": container with ID starting with 7463dcf8859ed35c39434318b9dd77fd79245d9e61e29752c2fb6fdbe15deee1 not found: ID does not exist" Nov 03 02:32:10.393593 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:10.393556 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-711318-5967965975-wf4df"] Nov 03 02:32:10.440021 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:10.439947 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-711318-5967965975-wf4df"] Nov 03 02:32:10.575246 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:10.575212 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36dca76a-75qw9"] Nov 03 02:32:10.575686 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:10.575672 5605 cpu_manager.go:414] "containerMap: removing container" podUID="85aa8ff4-f330-45e1-bb65-58ea54bbf0df" containerName="revoke-keys" Nov 03 02:32:10.575733 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:10.575687 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85aa8ff4-f330-45e1-bb65-58ea54bbf0df" containerName="revoke-keys" Nov 03 02:32:10.575733 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:10.575696 5605 cpu_manager.go:414] "containerMap: removing container" podUID="15cdb9a2-9888-463a-85c0-a2b70d760b5d" containerName="payload-pre-processing" Nov 03 02:32:10.575733 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:10.575702 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15cdb9a2-9888-463a-85c0-a2b70d760b5d" containerName="payload-pre-processing" Nov 03 02:32:10.575824 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:10.575781 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="85aa8ff4-f330-45e1-bb65-58ea54bbf0df" containerName="revoke-keys" Nov 03 02:32:10.575824 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:10.575788 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="15cdb9a2-9888-463a-85c0-a2b70d760b5d" containerName="payload-pre-processing" Nov 03 02:32:10.580073 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:10.580053 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36dca76a-75qw9" Nov 03 02:32:10.632482 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:10.632391 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:32:10.632653 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:10.632563 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:32:10.632851 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:10.632827 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:32:10.633340 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:10.633320 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:32:10.636064 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:10.636043 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36dca76a-75qw9"] Nov 03 02:32:10.666881 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:10.666851 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3641616a-f449-4f07-876c-f2c1710e4edd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36dca76a-75qw9\" (UID: \"3641616a-f449-4f07-876c-f2c1710e4edd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36dca76a-75qw9" Nov 03 02:32:10.667068 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:10.667050 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89hdg\" (UniqueName: \"kubernetes.io/projected/3641616a-f449-4f07-876c-f2c1710e4edd-kube-api-access-89hdg\") pod \"maas-api-revoke-sub-36dca76a-75qw9\" (UID: \"3641616a-f449-4f07-876c-f2c1710e4edd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36dca76a-75qw9" Nov 03 02:32:10.667143 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:10.667080 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3641616a-f449-4f07-876c-f2c1710e4edd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36dca76a-75qw9\" (UID: \"3641616a-f449-4f07-876c-f2c1710e4edd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36dca76a-75qw9" Nov 03 02:32:10.768730 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:10.768666 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89hdg\" (UniqueName: \"kubernetes.io/projected/3641616a-f449-4f07-876c-f2c1710e4edd-kube-api-access-89hdg\") pod \"maas-api-revoke-sub-36dca76a-75qw9\" (UID: \"3641616a-f449-4f07-876c-f2c1710e4edd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36dca76a-75qw9" Nov 03 02:32:10.768730 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:10.768700 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3641616a-f449-4f07-876c-f2c1710e4edd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36dca76a-75qw9\" (UID: \"3641616a-f449-4f07-876c-f2c1710e4edd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36dca76a-75qw9" Nov 03 02:32:10.768883 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:10.768733 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3641616a-f449-4f07-876c-f2c1710e4edd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36dca76a-75qw9\" (UID: \"3641616a-f449-4f07-876c-f2c1710e4edd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36dca76a-75qw9" Nov 03 02:32:10.769462 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:10.769442 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3641616a-f449-4f07-876c-f2c1710e4edd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36dca76a-75qw9\" (UID: \"3641616a-f449-4f07-876c-f2c1710e4edd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36dca76a-75qw9" Nov 03 02:32:10.771091 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:10.771067 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3641616a-f449-4f07-876c-f2c1710e4edd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36dca76a-75qw9\" (UID: \"3641616a-f449-4f07-876c-f2c1710e4edd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36dca76a-75qw9" Nov 03 02:32:10.823193 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:10.823167 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89hdg\" (UniqueName: \"kubernetes.io/projected/3641616a-f449-4f07-876c-f2c1710e4edd-kube-api-access-89hdg\") pod \"maas-api-revoke-sub-36dca76a-75qw9\" (UID: \"3641616a-f449-4f07-876c-f2c1710e4edd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36dca76a-75qw9" Nov 03 02:32:10.849809 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:10.849785 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15cdb9a2-9888-463a-85c0-a2b70d760b5d" path="/var/lib/kubelet/pods/15cdb9a2-9888-463a-85c0-a2b70d760b5d/volumes" Nov 03 02:32:10.893388 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:10.893348 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36dca76a-75qw9" Nov 03 02:32:11.274626 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:11.274597 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36dca76a-75qw9"] Nov 03 02:32:11.280147 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:32:11.280119 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3641616a_f449_4f07_876c_f2c1710e4edd.slice/crio-de00cbb5815bb4271a66edaf221c5d5f37c449681364af9f9d673fe9a09a0040 WatchSource:0}: Error finding container de00cbb5815bb4271a66edaf221c5d5f37c449681364af9f9d673fe9a09a0040: Status 404 returned error can't find the container with id de00cbb5815bb4271a66edaf221c5d5f37c449681364af9f9d673fe9a09a0040 Nov 03 02:32:12.251455 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:12.251421 5605 generic.go:356] "Generic (PLEG): container finished" podID="3641616a-f449-4f07-876c-f2c1710e4edd" containerID="a9c18f4cb75508b88063b8933c6801cdebddfeff84e141fc2f51356a70b3623f" exitCode=0 Nov 03 02:32:12.251867 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:12.251518 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36dca76a-75qw9" event={"ID":"3641616a-f449-4f07-876c-f2c1710e4edd","Type":"ContainerDied","Data":"a9c18f4cb75508b88063b8933c6801cdebddfeff84e141fc2f51356a70b3623f"} Nov 03 02:32:12.251867 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:12.251579 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36dca76a-75qw9" event={"ID":"3641616a-f449-4f07-876c-f2c1710e4edd","Type":"ContainerStarted","Data":"de00cbb5815bb4271a66edaf221c5d5f37c449681364af9f9d673fe9a09a0040"} Nov 03 02:32:12.745092 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:12.745055 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:32:13.387277 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:13.387245 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36dca76a-75qw9" Nov 03 02:32:13.495853 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:13.495815 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3641616a-f449-4f07-876c-f2c1710e4edd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3641616a-f449-4f07-876c-f2c1710e4edd-maas-api-openshift-ca\") pod \"3641616a-f449-4f07-876c-f2c1710e4edd\" (UID: \"3641616a-f449-4f07-876c-f2c1710e4edd\") " Nov 03 02:32:13.496011 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:13.495906 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3641616a-f449-4f07-876c-f2c1710e4edd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3641616a-f449-4f07-876c-f2c1710e4edd-maas-api-xks-ca\") pod \"3641616a-f449-4f07-876c-f2c1710e4edd\" (UID: \"3641616a-f449-4f07-876c-f2c1710e4edd\") " Nov 03 02:32:13.496011 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:13.496005 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3641616a-f449-4f07-876c-f2c1710e4edd-kube-api-access-89hdg\" (UniqueName: \"kubernetes.io/projected/3641616a-f449-4f07-876c-f2c1710e4edd-kube-api-access-89hdg\") pod \"3641616a-f449-4f07-876c-f2c1710e4edd\" (UID: \"3641616a-f449-4f07-876c-f2c1710e4edd\") " Nov 03 02:32:13.496507 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:13.496197 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3641616a-f449-4f07-876c-f2c1710e4edd-maas-api-openshift-ca" pod "3641616a-f449-4f07-876c-f2c1710e4edd" (UID: "3641616a-f449-4f07-876c-f2c1710e4edd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:32:13.496507 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:13.496483 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3641616a-f449-4f07-876c-f2c1710e4edd-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:32:13.498503 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:13.498480 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3641616a-f449-4f07-876c-f2c1710e4edd-kube-api-access-89hdg" pod "3641616a-f449-4f07-876c-f2c1710e4edd" (UID: "3641616a-f449-4f07-876c-f2c1710e4edd"). InnerVolumeSpecName "kube-api-access-89hdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:32:13.498584 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:13.498485 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3641616a-f449-4f07-876c-f2c1710e4edd-maas-api-xks-ca" pod "3641616a-f449-4f07-876c-f2c1710e4edd" (UID: "3641616a-f449-4f07-876c-f2c1710e4edd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:32:13.597935 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:13.597899 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3641616a-f449-4f07-876c-f2c1710e4edd-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:32:13.597935 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:13.597932 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89hdg\" (UniqueName: \"kubernetes.io/projected/3641616a-f449-4f07-876c-f2c1710e4edd-kube-api-access-89hdg\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:32:14.267260 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:14.267217 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36dca76a-75qw9" event={"ID":"3641616a-f449-4f07-876c-f2c1710e4edd","Type":"ContainerDied","Data":"de00cbb5815bb4271a66edaf221c5d5f37c449681364af9f9d673fe9a09a0040"} Nov 03 02:32:14.267260 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:14.267246 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36dca76a-75qw9" Nov 03 02:32:14.267260 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:14.267269 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de00cbb5815bb4271a66edaf221c5d5f37c449681364af9f9d673fe9a09a0040" Nov 03 02:32:14.907168 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:14.907136 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7183b7ee-b8bn5"] Nov 03 02:32:14.908430 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:14.908399 5605 cpu_manager.go:414] "containerMap: removing container" podUID="3641616a-f449-4f07-876c-f2c1710e4edd" containerName="revoke-keys" Nov 03 02:32:14.908591 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:14.908576 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3641616a-f449-4f07-876c-f2c1710e4edd" containerName="revoke-keys" Nov 03 02:32:14.908870 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:14.908852 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="3641616a-f449-4f07-876c-f2c1710e4edd" containerName="revoke-keys" Nov 03 02:32:14.920626 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:14.920606 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7183b7ee-b8bn5" Nov 03 02:32:14.950375 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:14.950324 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7183b7ee-b8bn5"] Nov 03 02:32:15.027380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:15.027000 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98b93b8e-c7eb-40c5-896d-bb6d93d6df21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7183b7ee-b8bn5\" (UID: \"98b93b8e-c7eb-40c5-896d-bb6d93d6df21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7183b7ee-b8bn5" Nov 03 02:32:15.027380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:15.027072 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98b93b8e-c7eb-40c5-896d-bb6d93d6df21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7183b7ee-b8bn5\" (UID: \"98b93b8e-c7eb-40c5-896d-bb6d93d6df21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7183b7ee-b8bn5" Nov 03 02:32:15.027380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:15.027092 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlzvp\" (UniqueName: \"kubernetes.io/projected/98b93b8e-c7eb-40c5-896d-bb6d93d6df21-kube-api-access-rlzvp\") pod \"maas-api-revoke-sub-7183b7ee-b8bn5\" (UID: \"98b93b8e-c7eb-40c5-896d-bb6d93d6df21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7183b7ee-b8bn5" Nov 03 02:32:15.027380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:15.027323 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:32:15.027736 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:15.027425 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:32:15.027794 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:15.027749 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:32:15.078382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:15.063499 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:32:15.133068 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:15.131064 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98b93b8e-c7eb-40c5-896d-bb6d93d6df21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7183b7ee-b8bn5\" (UID: \"98b93b8e-c7eb-40c5-896d-bb6d93d6df21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7183b7ee-b8bn5" Nov 03 02:32:15.133068 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:15.131195 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98b93b8e-c7eb-40c5-896d-bb6d93d6df21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7183b7ee-b8bn5\" (UID: \"98b93b8e-c7eb-40c5-896d-bb6d93d6df21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7183b7ee-b8bn5" Nov 03 02:32:15.133068 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:15.131231 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlzvp\" (UniqueName: \"kubernetes.io/projected/98b93b8e-c7eb-40c5-896d-bb6d93d6df21-kube-api-access-rlzvp\") pod \"maas-api-revoke-sub-7183b7ee-b8bn5\" (UID: \"98b93b8e-c7eb-40c5-896d-bb6d93d6df21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7183b7ee-b8bn5" Nov 03 02:32:15.133068 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:15.132484 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98b93b8e-c7eb-40c5-896d-bb6d93d6df21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7183b7ee-b8bn5\" (UID: \"98b93b8e-c7eb-40c5-896d-bb6d93d6df21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7183b7ee-b8bn5" Nov 03 02:32:15.136300 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:15.136273 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98b93b8e-c7eb-40c5-896d-bb6d93d6df21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7183b7ee-b8bn5\" (UID: \"98b93b8e-c7eb-40c5-896d-bb6d93d6df21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7183b7ee-b8bn5" Nov 03 02:32:15.212427 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:15.203260 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlzvp\" (UniqueName: \"kubernetes.io/projected/98b93b8e-c7eb-40c5-896d-bb6d93d6df21-kube-api-access-rlzvp\") pod \"maas-api-revoke-sub-7183b7ee-b8bn5\" (UID: \"98b93b8e-c7eb-40c5-896d-bb6d93d6df21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7183b7ee-b8bn5" Nov 03 02:32:15.241761 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:15.241731 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7183b7ee-b8bn5" Nov 03 02:32:15.759403 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:15.759353 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7183b7ee-b8bn5"] Nov 03 02:32:15.764532 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:32:15.764496 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98b93b8e_c7eb_40c5_896d_bb6d93d6df21.slice/crio-fc356652e76d4452f3dda6ed7e1ca5854441d19b35bb1e661c8ff1a2d13da6b5 WatchSource:0}: Error finding container fc356652e76d4452f3dda6ed7e1ca5854441d19b35bb1e661c8ff1a2d13da6b5: Status 404 returned error can't find the container with id fc356652e76d4452f3dda6ed7e1ca5854441d19b35bb1e661c8ff1a2d13da6b5 Nov 03 02:32:16.291089 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:16.291056 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7183b7ee-b8bn5" event={"ID":"98b93b8e-c7eb-40c5-896d-bb6d93d6df21","Type":"ContainerStarted","Data":"875ef69df9b7aadaa0f8960739bcfefa753e3c6df0ba4aa19f7a4767f9df253e"} Nov 03 02:32:16.291466 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:16.291110 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7183b7ee-b8bn5" event={"ID":"98b93b8e-c7eb-40c5-896d-bb6d93d6df21","Type":"ContainerStarted","Data":"fc356652e76d4452f3dda6ed7e1ca5854441d19b35bb1e661c8ff1a2d13da6b5"} Nov 03 02:32:16.401633 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:16.401547 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7183b7ee-b8bn5" podStartSLOduration=2.40153288 podStartE2EDuration="2.40153288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:32:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:32:16.399585961 +0000 UTC m=+2413.658784114" watchObservedRunningTime="2026-11-03 02:32:16.40153288 +0000 UTC m=+2413.660731055" Nov 03 02:32:17.281674 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:17.281638 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-777defe4-4dcnd"] Nov 03 02:32:17.285959 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:17.285933 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-777defe4-4dcnd" Nov 03 02:32:17.300933 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:17.300903 5605 generic.go:356] "Generic (PLEG): container finished" podID="98b93b8e-c7eb-40c5-896d-bb6d93d6df21" containerID="875ef69df9b7aadaa0f8960739bcfefa753e3c6df0ba4aa19f7a4767f9df253e" exitCode=0 Nov 03 02:32:17.301220 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:17.300978 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7183b7ee-b8bn5" event={"ID":"98b93b8e-c7eb-40c5-896d-bb6d93d6df21","Type":"ContainerDied","Data":"875ef69df9b7aadaa0f8960739bcfefa753e3c6df0ba4aa19f7a4767f9df253e"} Nov 03 02:32:17.339894 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:17.339866 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-777defe4-4dcnd"] Nov 03 02:32:17.355938 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:17.355898 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a47a1d4-8b2c-4edd-9c89-6ca01b06bc76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-777defe4-4dcnd\" (UID: \"2a47a1d4-8b2c-4edd-9c89-6ca01b06bc76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-777defe4-4dcnd" Nov 03 02:32:17.356129 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:17.355971 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a47a1d4-8b2c-4edd-9c89-6ca01b06bc76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-777defe4-4dcnd\" (UID: \"2a47a1d4-8b2c-4edd-9c89-6ca01b06bc76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-777defe4-4dcnd" Nov 03 02:32:17.356216 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:17.356182 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldfjz\" (UniqueName: \"kubernetes.io/projected/2a47a1d4-8b2c-4edd-9c89-6ca01b06bc76-kube-api-access-ldfjz\") pod \"maas-api-revoke-sub-777defe4-4dcnd\" (UID: \"2a47a1d4-8b2c-4edd-9c89-6ca01b06bc76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-777defe4-4dcnd" Nov 03 02:32:17.457726 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:17.457694 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldfjz\" (UniqueName: \"kubernetes.io/projected/2a47a1d4-8b2c-4edd-9c89-6ca01b06bc76-kube-api-access-ldfjz\") pod \"maas-api-revoke-sub-777defe4-4dcnd\" (UID: \"2a47a1d4-8b2c-4edd-9c89-6ca01b06bc76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-777defe4-4dcnd" Nov 03 02:32:17.457898 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:17.457855 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a47a1d4-8b2c-4edd-9c89-6ca01b06bc76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-777defe4-4dcnd\" (UID: \"2a47a1d4-8b2c-4edd-9c89-6ca01b06bc76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-777defe4-4dcnd" Nov 03 02:32:17.457952 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:17.457910 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a47a1d4-8b2c-4edd-9c89-6ca01b06bc76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-777defe4-4dcnd\" (UID: \"2a47a1d4-8b2c-4edd-9c89-6ca01b06bc76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-777defe4-4dcnd" Nov 03 02:32:17.458589 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:17.458560 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a47a1d4-8b2c-4edd-9c89-6ca01b06bc76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-777defe4-4dcnd\" (UID: \"2a47a1d4-8b2c-4edd-9c89-6ca01b06bc76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-777defe4-4dcnd" Nov 03 02:32:17.460272 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:17.460241 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a47a1d4-8b2c-4edd-9c89-6ca01b06bc76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-777defe4-4dcnd\" (UID: \"2a47a1d4-8b2c-4edd-9c89-6ca01b06bc76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-777defe4-4dcnd" Nov 03 02:32:17.512196 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:17.512165 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldfjz\" (UniqueName: \"kubernetes.io/projected/2a47a1d4-8b2c-4edd-9c89-6ca01b06bc76-kube-api-access-ldfjz\") pod \"maas-api-revoke-sub-777defe4-4dcnd\" (UID: \"2a47a1d4-8b2c-4edd-9c89-6ca01b06bc76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-777defe4-4dcnd" Nov 03 02:32:17.599884 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:17.599858 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-777defe4-4dcnd" Nov 03 02:32:17.966928 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:17.966843 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-777defe4-4dcnd"] Nov 03 02:32:17.972183 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:32:17.972151 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a47a1d4_8b2c_4edd_9c89_6ca01b06bc76.slice/crio-5fab376e747f4ca2a39c60abd3486b7452adf2d97587f8fdfc02554568cc25e3 WatchSource:0}: Error finding container 5fab376e747f4ca2a39c60abd3486b7452adf2d97587f8fdfc02554568cc25e3: Status 404 returned error can't find the container with id 5fab376e747f4ca2a39c60abd3486b7452adf2d97587f8fdfc02554568cc25e3 Nov 03 02:32:18.308427 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:18.308384 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-777defe4-4dcnd" event={"ID":"2a47a1d4-8b2c-4edd-9c89-6ca01b06bc76","Type":"ContainerStarted","Data":"1d5ad459e171e22ec017f7af74297ebb0b0a751a5515ad6fd4886c41430bdc7b"} Nov 03 02:32:18.308777 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:18.308454 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-777defe4-4dcnd" event={"ID":"2a47a1d4-8b2c-4edd-9c89-6ca01b06bc76","Type":"ContainerStarted","Data":"5fab376e747f4ca2a39c60abd3486b7452adf2d97587f8fdfc02554568cc25e3"} Nov 03 02:32:18.417237 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:18.417183 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-777defe4-4dcnd" podStartSLOduration=1.417168123 podStartE2EDuration="1.417168123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:32:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:32:18.414962274 +0000 UTC m=+2415.674160429" watchObservedRunningTime="2026-11-03 02:32:18.417168123 +0000 UTC m=+2415.676366304" Nov 03 02:32:18.438349 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:18.438323 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7183b7ee-b8bn5" Nov 03 02:32:18.469610 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:18.469572 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98b93b8e-c7eb-40c5-896d-bb6d93d6df21-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98b93b8e-c7eb-40c5-896d-bb6d93d6df21-maas-api-openshift-ca\") pod \"98b93b8e-c7eb-40c5-896d-bb6d93d6df21\" (UID: \"98b93b8e-c7eb-40c5-896d-bb6d93d6df21\") " Nov 03 02:32:18.469719 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:18.469627 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98b93b8e-c7eb-40c5-896d-bb6d93d6df21-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98b93b8e-c7eb-40c5-896d-bb6d93d6df21-maas-api-xks-ca\") pod \"98b93b8e-c7eb-40c5-896d-bb6d93d6df21\" (UID: \"98b93b8e-c7eb-40c5-896d-bb6d93d6df21\") " Nov 03 02:32:18.469780 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:18.469759 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98b93b8e-c7eb-40c5-896d-bb6d93d6df21-kube-api-access-rlzvp\" (UniqueName: \"kubernetes.io/projected/98b93b8e-c7eb-40c5-896d-bb6d93d6df21-kube-api-access-rlzvp\") pod \"98b93b8e-c7eb-40c5-896d-bb6d93d6df21\" (UID: \"98b93b8e-c7eb-40c5-896d-bb6d93d6df21\") " Nov 03 02:32:18.469994 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:18.469973 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98b93b8e-c7eb-40c5-896d-bb6d93d6df21-maas-api-openshift-ca" pod "98b93b8e-c7eb-40c5-896d-bb6d93d6df21" (UID: "98b93b8e-c7eb-40c5-896d-bb6d93d6df21"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:32:18.470242 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:18.470177 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98b93b8e-c7eb-40c5-896d-bb6d93d6df21-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:32:18.471742 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:18.471717 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98b93b8e-c7eb-40c5-896d-bb6d93d6df21-maas-api-xks-ca" pod "98b93b8e-c7eb-40c5-896d-bb6d93d6df21" (UID: "98b93b8e-c7eb-40c5-896d-bb6d93d6df21"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:32:18.471874 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:18.471854 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98b93b8e-c7eb-40c5-896d-bb6d93d6df21-kube-api-access-rlzvp" pod "98b93b8e-c7eb-40c5-896d-bb6d93d6df21" (UID: "98b93b8e-c7eb-40c5-896d-bb6d93d6df21"). InnerVolumeSpecName "kube-api-access-rlzvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:32:18.571798 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:18.571770 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlzvp\" (UniqueName: \"kubernetes.io/projected/98b93b8e-c7eb-40c5-896d-bb6d93d6df21-kube-api-access-rlzvp\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:32:18.571798 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:18.571799 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98b93b8e-c7eb-40c5-896d-bb6d93d6df21-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:32:19.315947 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:19.315920 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7183b7ee-b8bn5" Nov 03 02:32:19.316430 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:19.315952 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7183b7ee-b8bn5" event={"ID":"98b93b8e-c7eb-40c5-896d-bb6d93d6df21","Type":"ContainerDied","Data":"fc356652e76d4452f3dda6ed7e1ca5854441d19b35bb1e661c8ff1a2d13da6b5"} Nov 03 02:32:19.316430 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:19.316001 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc356652e76d4452f3dda6ed7e1ca5854441d19b35bb1e661c8ff1a2d13da6b5" Nov 03 02:32:19.317654 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:19.317623 5605 generic.go:356] "Generic (PLEG): container finished" podID="2a47a1d4-8b2c-4edd-9c89-6ca01b06bc76" containerID="1d5ad459e171e22ec017f7af74297ebb0b0a751a5515ad6fd4886c41430bdc7b" exitCode=0 Nov 03 02:32:19.317757 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:19.317674 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-777defe4-4dcnd" event={"ID":"2a47a1d4-8b2c-4edd-9c89-6ca01b06bc76","Type":"ContainerDied","Data":"1d5ad459e171e22ec017f7af74297ebb0b0a751a5515ad6fd4886c41430bdc7b"} Nov 03 02:32:20.465789 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:20.465760 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-777defe4-4dcnd" Nov 03 02:32:20.495132 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:20.495106 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a47a1d4-8b2c-4edd-9c89-6ca01b06bc76-kube-api-access-ldfjz\" (UniqueName: \"kubernetes.io/projected/2a47a1d4-8b2c-4edd-9c89-6ca01b06bc76-kube-api-access-ldfjz\") pod \"2a47a1d4-8b2c-4edd-9c89-6ca01b06bc76\" (UID: \"2a47a1d4-8b2c-4edd-9c89-6ca01b06bc76\") " Nov 03 02:32:20.495258 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:20.495145 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a47a1d4-8b2c-4edd-9c89-6ca01b06bc76-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a47a1d4-8b2c-4edd-9c89-6ca01b06bc76-maas-api-openshift-ca\") pod \"2a47a1d4-8b2c-4edd-9c89-6ca01b06bc76\" (UID: \"2a47a1d4-8b2c-4edd-9c89-6ca01b06bc76\") " Nov 03 02:32:20.495258 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:20.495182 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a47a1d4-8b2c-4edd-9c89-6ca01b06bc76-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a47a1d4-8b2c-4edd-9c89-6ca01b06bc76-maas-api-xks-ca\") pod \"2a47a1d4-8b2c-4edd-9c89-6ca01b06bc76\" (UID: \"2a47a1d4-8b2c-4edd-9c89-6ca01b06bc76\") " Nov 03 02:32:20.495564 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:20.495542 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a47a1d4-8b2c-4edd-9c89-6ca01b06bc76-maas-api-openshift-ca" pod "2a47a1d4-8b2c-4edd-9c89-6ca01b06bc76" (UID: "2a47a1d4-8b2c-4edd-9c89-6ca01b06bc76"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:32:20.497177 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:20.497148 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a47a1d4-8b2c-4edd-9c89-6ca01b06bc76-maas-api-xks-ca" pod "2a47a1d4-8b2c-4edd-9c89-6ca01b06bc76" (UID: "2a47a1d4-8b2c-4edd-9c89-6ca01b06bc76"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:32:20.497279 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:20.497224 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a47a1d4-8b2c-4edd-9c89-6ca01b06bc76-kube-api-access-ldfjz" pod "2a47a1d4-8b2c-4edd-9c89-6ca01b06bc76" (UID: "2a47a1d4-8b2c-4edd-9c89-6ca01b06bc76"). InnerVolumeSpecName "kube-api-access-ldfjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:32:20.597154 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:20.597068 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldfjz\" (UniqueName: \"kubernetes.io/projected/2a47a1d4-8b2c-4edd-9c89-6ca01b06bc76-kube-api-access-ldfjz\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:32:20.597154 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:20.597097 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a47a1d4-8b2c-4edd-9c89-6ca01b06bc76-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:32:20.597154 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:20.597107 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a47a1d4-8b2c-4edd-9c89-6ca01b06bc76-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:32:21.331547 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:21.331506 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-777defe4-4dcnd" event={"ID":"2a47a1d4-8b2c-4edd-9c89-6ca01b06bc76","Type":"ContainerDied","Data":"5fab376e747f4ca2a39c60abd3486b7452adf2d97587f8fdfc02554568cc25e3"} Nov 03 02:32:21.331547 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:21.331541 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-777defe4-4dcnd" Nov 03 02:32:21.331759 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:21.331570 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fab376e747f4ca2a39c60abd3486b7452adf2d97587f8fdfc02554568cc25e3" Nov 03 02:32:23.914714 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:32:23.914678 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:32:32.840467 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:32.840428 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c5a3177-kx7m8"] Nov 03 02:32:32.841057 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:32.841035 5605 cpu_manager.go:414] "containerMap: removing container" podUID="2a47a1d4-8b2c-4edd-9c89-6ca01b06bc76" containerName="revoke-keys" Nov 03 02:32:32.841145 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:32.841059 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a47a1d4-8b2c-4edd-9c89-6ca01b06bc76" containerName="revoke-keys" Nov 03 02:32:32.841145 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:32.841076 5605 cpu_manager.go:414] "containerMap: removing container" podUID="98b93b8e-c7eb-40c5-896d-bb6d93d6df21" containerName="revoke-keys" Nov 03 02:32:32.841145 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:32.841083 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98b93b8e-c7eb-40c5-896d-bb6d93d6df21" containerName="revoke-keys" Nov 03 02:32:32.841406 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:32.841239 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a47a1d4-8b2c-4edd-9c89-6ca01b06bc76" containerName="revoke-keys" Nov 03 02:32:32.841406 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:32.841255 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="98b93b8e-c7eb-40c5-896d-bb6d93d6df21" containerName="revoke-keys" Nov 03 02:32:32.846844 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:32.846821 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5a3177-kx7m8" Nov 03 02:32:32.899156 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:32.899131 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c5a3177-kx7m8"] Nov 03 02:32:32.942181 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:32.942151 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:32:32.942181 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:32.942157 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:32:32.942409 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:32.942204 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:32:32.942409 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:32.942245 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:32:32.945816 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:32.945792 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6a49f49-9af3-4c84-8a68-5ff152f7154e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c5a3177-kx7m8\" (UID: \"b6a49f49-9af3-4c84-8a68-5ff152f7154e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5a3177-kx7m8" Nov 03 02:32:32.945932 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:32.945832 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6a49f49-9af3-4c84-8a68-5ff152f7154e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c5a3177-kx7m8\" (UID: \"b6a49f49-9af3-4c84-8a68-5ff152f7154e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5a3177-kx7m8" Nov 03 02:32:32.945974 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:32.945930 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4vrq\" (UniqueName: \"kubernetes.io/projected/b6a49f49-9af3-4c84-8a68-5ff152f7154e-kube-api-access-n4vrq\") pod \"maas-api-revoke-sub-8c5a3177-kx7m8\" (UID: \"b6a49f49-9af3-4c84-8a68-5ff152f7154e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5a3177-kx7m8" Nov 03 02:32:33.046953 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:33.046920 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4vrq\" (UniqueName: \"kubernetes.io/projected/b6a49f49-9af3-4c84-8a68-5ff152f7154e-kube-api-access-n4vrq\") pod \"maas-api-revoke-sub-8c5a3177-kx7m8\" (UID: \"b6a49f49-9af3-4c84-8a68-5ff152f7154e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5a3177-kx7m8" Nov 03 02:32:33.047115 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:33.047102 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6a49f49-9af3-4c84-8a68-5ff152f7154e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c5a3177-kx7m8\" (UID: \"b6a49f49-9af3-4c84-8a68-5ff152f7154e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5a3177-kx7m8" Nov 03 02:32:33.047163 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:33.047136 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6a49f49-9af3-4c84-8a68-5ff152f7154e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c5a3177-kx7m8\" (UID: \"b6a49f49-9af3-4c84-8a68-5ff152f7154e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5a3177-kx7m8" Nov 03 02:32:33.047842 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:33.047819 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6a49f49-9af3-4c84-8a68-5ff152f7154e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c5a3177-kx7m8\" (UID: \"b6a49f49-9af3-4c84-8a68-5ff152f7154e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5a3177-kx7m8" Nov 03 02:32:33.049432 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:33.049411 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6a49f49-9af3-4c84-8a68-5ff152f7154e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c5a3177-kx7m8\" (UID: \"b6a49f49-9af3-4c84-8a68-5ff152f7154e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5a3177-kx7m8" Nov 03 02:32:33.101314 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:33.101245 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4vrq\" (UniqueName: \"kubernetes.io/projected/b6a49f49-9af3-4c84-8a68-5ff152f7154e-kube-api-access-n4vrq\") pod \"maas-api-revoke-sub-8c5a3177-kx7m8\" (UID: \"b6a49f49-9af3-4c84-8a68-5ff152f7154e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5a3177-kx7m8" Nov 03 02:32:33.159947 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:33.159914 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5a3177-kx7m8" Nov 03 02:32:33.251092 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:33.250928 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cda482ac-rxtqh"] Nov 03 02:32:33.254856 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:33.254833 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cda482ac-rxtqh" Nov 03 02:32:33.308583 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:33.308545 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cda482ac-rxtqh"] Nov 03 02:32:33.350685 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:33.350651 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63fc50d9-d0a7-428a-8dcc-894bafea8987-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cda482ac-rxtqh\" (UID: \"63fc50d9-d0a7-428a-8dcc-894bafea8987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cda482ac-rxtqh" Nov 03 02:32:33.350869 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:33.350784 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63fc50d9-d0a7-428a-8dcc-894bafea8987-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cda482ac-rxtqh\" (UID: \"63fc50d9-d0a7-428a-8dcc-894bafea8987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cda482ac-rxtqh" Nov 03 02:32:33.350869 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:33.350825 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8kp5\" (UniqueName: \"kubernetes.io/projected/63fc50d9-d0a7-428a-8dcc-894bafea8987-kube-api-access-s8kp5\") pod \"maas-api-revoke-sub-cda482ac-rxtqh\" (UID: \"63fc50d9-d0a7-428a-8dcc-894bafea8987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cda482ac-rxtqh" Nov 03 02:32:33.452000 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:33.451923 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63fc50d9-d0a7-428a-8dcc-894bafea8987-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cda482ac-rxtqh\" (UID: \"63fc50d9-d0a7-428a-8dcc-894bafea8987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cda482ac-rxtqh" Nov 03 02:32:33.452163 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:33.452001 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63fc50d9-d0a7-428a-8dcc-894bafea8987-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cda482ac-rxtqh\" (UID: \"63fc50d9-d0a7-428a-8dcc-894bafea8987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cda482ac-rxtqh" Nov 03 02:32:33.452163 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:33.452054 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8kp5\" (UniqueName: \"kubernetes.io/projected/63fc50d9-d0a7-428a-8dcc-894bafea8987-kube-api-access-s8kp5\") pod \"maas-api-revoke-sub-cda482ac-rxtqh\" (UID: \"63fc50d9-d0a7-428a-8dcc-894bafea8987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cda482ac-rxtqh" Nov 03 02:32:33.452876 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:33.452846 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63fc50d9-d0a7-428a-8dcc-894bafea8987-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cda482ac-rxtqh\" (UID: \"63fc50d9-d0a7-428a-8dcc-894bafea8987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cda482ac-rxtqh" Nov 03 02:32:33.454477 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:33.454456 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63fc50d9-d0a7-428a-8dcc-894bafea8987-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cda482ac-rxtqh\" (UID: \"63fc50d9-d0a7-428a-8dcc-894bafea8987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cda482ac-rxtqh" Nov 03 02:32:33.505625 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:33.505595 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8kp5\" (UniqueName: \"kubernetes.io/projected/63fc50d9-d0a7-428a-8dcc-894bafea8987-kube-api-access-s8kp5\") pod \"maas-api-revoke-sub-cda482ac-rxtqh\" (UID: \"63fc50d9-d0a7-428a-8dcc-894bafea8987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cda482ac-rxtqh" Nov 03 02:32:33.529623 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:33.529584 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c5a3177-kx7m8"] Nov 03 02:32:33.534722 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:32:33.534695 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6a49f49_9af3_4c84_8a68_5ff152f7154e.slice/crio-8b3cf16d6d4770f23f13fa32bdbe3286c9cf4331b702ea60c77d4454151d7da3 WatchSource:0}: Error finding container 8b3cf16d6d4770f23f13fa32bdbe3286c9cf4331b702ea60c77d4454151d7da3: Status 404 returned error can't find the container with id 8b3cf16d6d4770f23f13fa32bdbe3286c9cf4331b702ea60c77d4454151d7da3 Nov 03 02:32:33.572157 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:33.572136 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cda482ac-rxtqh" Nov 03 02:32:33.939882 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:33.939850 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cda482ac-rxtqh"] Nov 03 02:32:33.945303 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:32:33.945275 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63fc50d9_d0a7_428a_8dcc_894bafea8987.slice/crio-3c31e5c314a73b5cdd14df3e754790a435bbaa11362a6371d540efbf3d58e08e WatchSource:0}: Error finding container 3c31e5c314a73b5cdd14df3e754790a435bbaa11362a6371d540efbf3d58e08e: Status 404 returned error can't find the container with id 3c31e5c314a73b5cdd14df3e754790a435bbaa11362a6371d540efbf3d58e08e Nov 03 02:32:34.415275 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:34.415226 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5a3177-kx7m8" event={"ID":"b6a49f49-9af3-4c84-8a68-5ff152f7154e","Type":"ContainerStarted","Data":"373a5861ef0c8c72652212fdcde83aef2272192d0ffa3e243dd9e1a69d3e9ee9"} Nov 03 02:32:34.415491 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:34.415332 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5a3177-kx7m8" event={"ID":"b6a49f49-9af3-4c84-8a68-5ff152f7154e","Type":"ContainerStarted","Data":"8b3cf16d6d4770f23f13fa32bdbe3286c9cf4331b702ea60c77d4454151d7da3"} Nov 03 02:32:34.417058 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:34.417026 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cda482ac-rxtqh" event={"ID":"63fc50d9-d0a7-428a-8dcc-894bafea8987","Type":"ContainerStarted","Data":"0f4a551f235a5c12b722108087fa287fee71658ee02836ba2aa609bb373a0f1c"} Nov 03 02:32:34.417180 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:34.417080 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cda482ac-rxtqh" event={"ID":"63fc50d9-d0a7-428a-8dcc-894bafea8987","Type":"ContainerStarted","Data":"3c31e5c314a73b5cdd14df3e754790a435bbaa11362a6371d540efbf3d58e08e"} Nov 03 02:32:34.524195 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:34.524141 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5a3177-kx7m8" podStartSLOduration=2.524126887 podStartE2EDuration="2.524126887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:32:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:32:34.522690377 +0000 UTC m=+2431.781888531" watchObservedRunningTime="2026-11-03 02:32:34.524126887 +0000 UTC m=+2431.783325041" Nov 03 02:32:34.630206 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:34.630143 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cda482ac-rxtqh" podStartSLOduration=1.6301308570000002 podStartE2EDuration="1.630130857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:32:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:32:34.629040726 +0000 UTC m=+2431.888238879" watchObservedRunningTime="2026-11-03 02:32:34.630130857 +0000 UTC m=+2431.889329284" Nov 03 02:32:35.425990 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:35.425956 5605 generic.go:356] "Generic (PLEG): container finished" podID="b6a49f49-9af3-4c84-8a68-5ff152f7154e" containerID="373a5861ef0c8c72652212fdcde83aef2272192d0ffa3e243dd9e1a69d3e9ee9" exitCode=0 Nov 03 02:32:35.426448 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:35.426048 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5a3177-kx7m8" event={"ID":"b6a49f49-9af3-4c84-8a68-5ff152f7154e","Type":"ContainerDied","Data":"373a5861ef0c8c72652212fdcde83aef2272192d0ffa3e243dd9e1a69d3e9ee9"} Nov 03 02:32:35.427661 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:35.427636 5605 generic.go:356] "Generic (PLEG): container finished" podID="63fc50d9-d0a7-428a-8dcc-894bafea8987" containerID="0f4a551f235a5c12b722108087fa287fee71658ee02836ba2aa609bb373a0f1c" exitCode=0 Nov 03 02:32:35.427862 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:35.427684 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cda482ac-rxtqh" event={"ID":"63fc50d9-d0a7-428a-8dcc-894bafea8987","Type":"ContainerDied","Data":"0f4a551f235a5c12b722108087fa287fee71658ee02836ba2aa609bb373a0f1c"} Nov 03 02:32:36.601098 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:36.601073 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cda482ac-rxtqh" Nov 03 02:32:36.622843 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:36.622821 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5a3177-kx7m8" Nov 03 02:32:36.695452 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:36.695346 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63fc50d9-d0a7-428a-8dcc-894bafea8987-kube-api-access-s8kp5\" (UniqueName: \"kubernetes.io/projected/63fc50d9-d0a7-428a-8dcc-894bafea8987-kube-api-access-s8kp5\") pod \"63fc50d9-d0a7-428a-8dcc-894bafea8987\" (UID: \"63fc50d9-d0a7-428a-8dcc-894bafea8987\") " Nov 03 02:32:36.695452 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:36.695426 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63fc50d9-d0a7-428a-8dcc-894bafea8987-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63fc50d9-d0a7-428a-8dcc-894bafea8987-maas-api-openshift-ca\") pod \"63fc50d9-d0a7-428a-8dcc-894bafea8987\" (UID: \"63fc50d9-d0a7-428a-8dcc-894bafea8987\") " Nov 03 02:32:36.695684 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:36.695514 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63fc50d9-d0a7-428a-8dcc-894bafea8987-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63fc50d9-d0a7-428a-8dcc-894bafea8987-maas-api-xks-ca\") pod \"63fc50d9-d0a7-428a-8dcc-894bafea8987\" (UID: \"63fc50d9-d0a7-428a-8dcc-894bafea8987\") " Nov 03 02:32:36.695794 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:36.695766 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63fc50d9-d0a7-428a-8dcc-894bafea8987-maas-api-openshift-ca" pod "63fc50d9-d0a7-428a-8dcc-894bafea8987" (UID: "63fc50d9-d0a7-428a-8dcc-894bafea8987"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:32:36.697657 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:36.697623 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63fc50d9-d0a7-428a-8dcc-894bafea8987-kube-api-access-s8kp5" pod "63fc50d9-d0a7-428a-8dcc-894bafea8987" (UID: "63fc50d9-d0a7-428a-8dcc-894bafea8987"). InnerVolumeSpecName "kube-api-access-s8kp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:32:36.697786 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:36.697742 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63fc50d9-d0a7-428a-8dcc-894bafea8987-maas-api-xks-ca" pod "63fc50d9-d0a7-428a-8dcc-894bafea8987" (UID: "63fc50d9-d0a7-428a-8dcc-894bafea8987"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:32:36.796650 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:36.796478 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6a49f49-9af3-4c84-8a68-5ff152f7154e-kube-api-access-n4vrq\" (UniqueName: \"kubernetes.io/projected/b6a49f49-9af3-4c84-8a68-5ff152f7154e-kube-api-access-n4vrq\") pod \"b6a49f49-9af3-4c84-8a68-5ff152f7154e\" (UID: \"b6a49f49-9af3-4c84-8a68-5ff152f7154e\") " Nov 03 02:32:36.796650 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:36.796595 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6a49f49-9af3-4c84-8a68-5ff152f7154e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6a49f49-9af3-4c84-8a68-5ff152f7154e-maas-api-openshift-ca\") pod \"b6a49f49-9af3-4c84-8a68-5ff152f7154e\" (UID: \"b6a49f49-9af3-4c84-8a68-5ff152f7154e\") " Nov 03 02:32:36.796960 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:36.796657 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6a49f49-9af3-4c84-8a68-5ff152f7154e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6a49f49-9af3-4c84-8a68-5ff152f7154e-maas-api-xks-ca\") pod \"b6a49f49-9af3-4c84-8a68-5ff152f7154e\" (UID: \"b6a49f49-9af3-4c84-8a68-5ff152f7154e\") " Nov 03 02:32:36.797085 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:36.797058 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6a49f49-9af3-4c84-8a68-5ff152f7154e-maas-api-openshift-ca" pod "b6a49f49-9af3-4c84-8a68-5ff152f7154e" (UID: "b6a49f49-9af3-4c84-8a68-5ff152f7154e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:32:36.797266 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:36.797235 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6a49f49-9af3-4c84-8a68-5ff152f7154e-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:32:36.797334 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:36.797265 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8kp5\" (UniqueName: \"kubernetes.io/projected/63fc50d9-d0a7-428a-8dcc-894bafea8987-kube-api-access-s8kp5\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:32:36.797334 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:36.797281 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63fc50d9-d0a7-428a-8dcc-894bafea8987-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:32:36.797334 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:36.797296 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63fc50d9-d0a7-428a-8dcc-894bafea8987-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:32:36.799188 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:36.799159 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6a49f49-9af3-4c84-8a68-5ff152f7154e-kube-api-access-n4vrq" pod "b6a49f49-9af3-4c84-8a68-5ff152f7154e" (UID: "b6a49f49-9af3-4c84-8a68-5ff152f7154e"). InnerVolumeSpecName "kube-api-access-n4vrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:32:36.799320 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:36.799220 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6a49f49-9af3-4c84-8a68-5ff152f7154e-maas-api-xks-ca" pod "b6a49f49-9af3-4c84-8a68-5ff152f7154e" (UID: "b6a49f49-9af3-4c84-8a68-5ff152f7154e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:32:36.897975 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:36.897942 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4vrq\" (UniqueName: \"kubernetes.io/projected/b6a49f49-9af3-4c84-8a68-5ff152f7154e-kube-api-access-n4vrq\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:32:36.897975 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:36.897974 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6a49f49-9af3-4c84-8a68-5ff152f7154e-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:32:37.137452 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:37.137416 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-484e0158-5f5v7"] Nov 03 02:32:37.138096 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:37.138075 5605 cpu_manager.go:414] "containerMap: removing container" podUID="63fc50d9-d0a7-428a-8dcc-894bafea8987" containerName="revoke-keys" Nov 03 02:32:37.138138 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:37.138101 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63fc50d9-d0a7-428a-8dcc-894bafea8987" containerName="revoke-keys" Nov 03 02:32:37.138138 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:37.138127 5605 cpu_manager.go:414] "containerMap: removing container" podUID="b6a49f49-9af3-4c84-8a68-5ff152f7154e" containerName="revoke-keys" Nov 03 02:32:37.138138 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:37.138135 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6a49f49-9af3-4c84-8a68-5ff152f7154e" containerName="revoke-keys" Nov 03 02:32:37.138319 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:37.138306 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="63fc50d9-d0a7-428a-8dcc-894bafea8987" containerName="revoke-keys" Nov 03 02:32:37.138382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:37.138326 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6a49f49-9af3-4c84-8a68-5ff152f7154e" containerName="revoke-keys" Nov 03 02:32:37.142309 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:37.142288 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-484e0158-5f5v7" Nov 03 02:32:37.194046 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:37.194013 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-484e0158-5f5v7"] Nov 03 02:32:37.302297 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:37.302260 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7d44d42-0cf3-4195-8e69-73bcd264619a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-484e0158-5f5v7\" (UID: \"e7d44d42-0cf3-4195-8e69-73bcd264619a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-484e0158-5f5v7" Nov 03 02:32:37.302509 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:37.302410 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7d44d42-0cf3-4195-8e69-73bcd264619a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-484e0158-5f5v7\" (UID: \"e7d44d42-0cf3-4195-8e69-73bcd264619a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-484e0158-5f5v7" Nov 03 02:32:37.302509 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:37.302472 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pfdz\" (UniqueName: \"kubernetes.io/projected/e7d44d42-0cf3-4195-8e69-73bcd264619a-kube-api-access-6pfdz\") pod \"maas-api-revoke-sub-484e0158-5f5v7\" (UID: \"e7d44d42-0cf3-4195-8e69-73bcd264619a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-484e0158-5f5v7" Nov 03 02:32:37.403534 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:37.403451 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7d44d42-0cf3-4195-8e69-73bcd264619a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-484e0158-5f5v7\" (UID: \"e7d44d42-0cf3-4195-8e69-73bcd264619a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-484e0158-5f5v7" Nov 03 02:32:37.403534 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:37.403487 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pfdz\" (UniqueName: \"kubernetes.io/projected/e7d44d42-0cf3-4195-8e69-73bcd264619a-kube-api-access-6pfdz\") pod \"maas-api-revoke-sub-484e0158-5f5v7\" (UID: \"e7d44d42-0cf3-4195-8e69-73bcd264619a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-484e0158-5f5v7" Nov 03 02:32:37.403761 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:37.403567 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7d44d42-0cf3-4195-8e69-73bcd264619a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-484e0158-5f5v7\" (UID: \"e7d44d42-0cf3-4195-8e69-73bcd264619a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-484e0158-5f5v7" Nov 03 02:32:37.404120 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:37.404098 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7d44d42-0cf3-4195-8e69-73bcd264619a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-484e0158-5f5v7\" (UID: \"e7d44d42-0cf3-4195-8e69-73bcd264619a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-484e0158-5f5v7" Nov 03 02:32:37.405845 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:37.405822 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7d44d42-0cf3-4195-8e69-73bcd264619a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-484e0158-5f5v7\" (UID: \"e7d44d42-0cf3-4195-8e69-73bcd264619a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-484e0158-5f5v7" Nov 03 02:32:37.440315 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:37.440286 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cda482ac-rxtqh" Nov 03 02:32:37.440449 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:37.440336 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cda482ac-rxtqh" event={"ID":"63fc50d9-d0a7-428a-8dcc-894bafea8987","Type":"ContainerDied","Data":"3c31e5c314a73b5cdd14df3e754790a435bbaa11362a6371d540efbf3d58e08e"} Nov 03 02:32:37.440449 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:37.440413 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c31e5c314a73b5cdd14df3e754790a435bbaa11362a6371d540efbf3d58e08e" Nov 03 02:32:37.441599 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:37.441576 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5a3177-kx7m8" event={"ID":"b6a49f49-9af3-4c84-8a68-5ff152f7154e","Type":"ContainerDied","Data":"8b3cf16d6d4770f23f13fa32bdbe3286c9cf4331b702ea60c77d4454151d7da3"} Nov 03 02:32:37.441692 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:37.441610 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b3cf16d6d4770f23f13fa32bdbe3286c9cf4331b702ea60c77d4454151d7da3" Nov 03 02:32:37.441692 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:37.441614 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5a3177-kx7m8" Nov 03 02:32:37.457841 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:37.457819 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pfdz\" (UniqueName: \"kubernetes.io/projected/e7d44d42-0cf3-4195-8e69-73bcd264619a-kube-api-access-6pfdz\") pod \"maas-api-revoke-sub-484e0158-5f5v7\" (UID: \"e7d44d42-0cf3-4195-8e69-73bcd264619a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-484e0158-5f5v7" Nov 03 02:32:37.755698 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:37.755607 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-484e0158-5f5v7" Nov 03 02:32:38.126955 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:38.126921 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-484e0158-5f5v7"] Nov 03 02:32:38.132706 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:32:38.132669 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7d44d42_0cf3_4195_8e69_73bcd264619a.slice/crio-da7a568bed77aa01d63f62b0ef382185449904c6c61267831b13faed22a3336e WatchSource:0}: Error finding container da7a568bed77aa01d63f62b0ef382185449904c6c61267831b13faed22a3336e: Status 404 returned error can't find the container with id da7a568bed77aa01d63f62b0ef382185449904c6c61267831b13faed22a3336e Nov 03 02:32:38.449654 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:38.449569 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-484e0158-5f5v7" event={"ID":"e7d44d42-0cf3-4195-8e69-73bcd264619a","Type":"ContainerStarted","Data":"da7a568bed77aa01d63f62b0ef382185449904c6c61267831b13faed22a3336e"} Nov 03 02:32:39.456946 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:39.456915 5605 generic.go:356] "Generic (PLEG): container finished" podID="e7d44d42-0cf3-4195-8e69-73bcd264619a" containerID="ca8c80f70fca0ef91f9caec8b34bf8d948a754bf4eb5cccf0a2e1e26d142205f" exitCode=0 Nov 03 02:32:39.457320 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:39.456967 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-484e0158-5f5v7" event={"ID":"e7d44d42-0cf3-4195-8e69-73bcd264619a","Type":"ContainerDied","Data":"ca8c80f70fca0ef91f9caec8b34bf8d948a754bf4eb5cccf0a2e1e26d142205f"} Nov 03 02:32:40.916040 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:40.915737 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-484e0158-5f5v7" Nov 03 02:32:41.017391 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:41.015742 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7d44d42-0cf3-4195-8e69-73bcd264619a-kube-api-access-6pfdz\" (UniqueName: \"kubernetes.io/projected/e7d44d42-0cf3-4195-8e69-73bcd264619a-kube-api-access-6pfdz\") pod \"e7d44d42-0cf3-4195-8e69-73bcd264619a\" (UID: \"e7d44d42-0cf3-4195-8e69-73bcd264619a\") " Nov 03 02:32:41.017391 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:41.015820 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7d44d42-0cf3-4195-8e69-73bcd264619a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7d44d42-0cf3-4195-8e69-73bcd264619a-maas-api-xks-ca\") pod \"e7d44d42-0cf3-4195-8e69-73bcd264619a\" (UID: \"e7d44d42-0cf3-4195-8e69-73bcd264619a\") " Nov 03 02:32:41.017391 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:41.015909 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7d44d42-0cf3-4195-8e69-73bcd264619a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7d44d42-0cf3-4195-8e69-73bcd264619a-maas-api-openshift-ca\") pod \"e7d44d42-0cf3-4195-8e69-73bcd264619a\" (UID: \"e7d44d42-0cf3-4195-8e69-73bcd264619a\") " Nov 03 02:32:41.017391 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:41.016660 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7d44d42-0cf3-4195-8e69-73bcd264619a-maas-api-openshift-ca" pod "e7d44d42-0cf3-4195-8e69-73bcd264619a" (UID: "e7d44d42-0cf3-4195-8e69-73bcd264619a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:32:41.025940 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:41.025892 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7d44d42-0cf3-4195-8e69-73bcd264619a-maas-api-xks-ca" pod "e7d44d42-0cf3-4195-8e69-73bcd264619a" (UID: "e7d44d42-0cf3-4195-8e69-73bcd264619a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:32:41.033162 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:41.032957 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7d44d42-0cf3-4195-8e69-73bcd264619a-kube-api-access-6pfdz" pod "e7d44d42-0cf3-4195-8e69-73bcd264619a" (UID: "e7d44d42-0cf3-4195-8e69-73bcd264619a"). InnerVolumeSpecName "kube-api-access-6pfdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:32:41.126260 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:41.126155 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pfdz\" (UniqueName: \"kubernetes.io/projected/e7d44d42-0cf3-4195-8e69-73bcd264619a-kube-api-access-6pfdz\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:32:41.126490 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:41.126463 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7d44d42-0cf3-4195-8e69-73bcd264619a-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:32:41.126578 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:41.126491 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7d44d42-0cf3-4195-8e69-73bcd264619a-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:32:41.522881 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:41.522805 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-484e0158-5f5v7" Nov 03 02:32:41.522881 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:41.522833 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-484e0158-5f5v7" event={"ID":"e7d44d42-0cf3-4195-8e69-73bcd264619a","Type":"ContainerDied","Data":"da7a568bed77aa01d63f62b0ef382185449904c6c61267831b13faed22a3336e"} Nov 03 02:32:41.523085 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:41.522886 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da7a568bed77aa01d63f62b0ef382185449904c6c61267831b13faed22a3336e" Nov 03 02:32:43.988886 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:43.988855 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1caa6e5-5t6j8"] Nov 03 02:32:43.989503 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:43.989483 5605 cpu_manager.go:414] "containerMap: removing container" podUID="e7d44d42-0cf3-4195-8e69-73bcd264619a" containerName="revoke-keys" Nov 03 02:32:43.989608 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:43.989504 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7d44d42-0cf3-4195-8e69-73bcd264619a" containerName="revoke-keys" Nov 03 02:32:43.989672 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:43.989610 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7d44d42-0cf3-4195-8e69-73bcd264619a" containerName="revoke-keys" Nov 03 02:32:43.993008 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:43.992988 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1caa6e5-5t6j8" Nov 03 02:32:44.043528 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:44.043504 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:32:44.058770 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:44.058740 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h767s\" (UniqueName: \"kubernetes.io/projected/4748e14a-d17c-43ab-8951-0a01f40b7512-kube-api-access-h767s\") pod \"maas-api-revoke-sub-b1caa6e5-5t6j8\" (UID: \"4748e14a-d17c-43ab-8951-0a01f40b7512\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1caa6e5-5t6j8" Nov 03 02:32:44.058959 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:44.058935 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4748e14a-d17c-43ab-8951-0a01f40b7512-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1caa6e5-5t6j8\" (UID: \"4748e14a-d17c-43ab-8951-0a01f40b7512\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1caa6e5-5t6j8" Nov 03 02:32:44.059045 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:44.058971 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4748e14a-d17c-43ab-8951-0a01f40b7512-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1caa6e5-5t6j8\" (UID: \"4748e14a-d17c-43ab-8951-0a01f40b7512\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1caa6e5-5t6j8" Nov 03 02:32:44.090398 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:44.090373 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:32:44.090532 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:44.090410 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:32:44.090532 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:44.090487 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:32:44.093187 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:44.093166 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1caa6e5-5t6j8"] Nov 03 02:32:44.159624 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:44.159592 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h767s\" (UniqueName: \"kubernetes.io/projected/4748e14a-d17c-43ab-8951-0a01f40b7512-kube-api-access-h767s\") pod \"maas-api-revoke-sub-b1caa6e5-5t6j8\" (UID: \"4748e14a-d17c-43ab-8951-0a01f40b7512\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1caa6e5-5t6j8" Nov 03 02:32:44.159803 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:44.159784 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4748e14a-d17c-43ab-8951-0a01f40b7512-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1caa6e5-5t6j8\" (UID: \"4748e14a-d17c-43ab-8951-0a01f40b7512\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1caa6e5-5t6j8" Nov 03 02:32:44.159873 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:44.159823 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4748e14a-d17c-43ab-8951-0a01f40b7512-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1caa6e5-5t6j8\" (UID: \"4748e14a-d17c-43ab-8951-0a01f40b7512\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1caa6e5-5t6j8" Nov 03 02:32:44.160649 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:44.160625 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4748e14a-d17c-43ab-8951-0a01f40b7512-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1caa6e5-5t6j8\" (UID: \"4748e14a-d17c-43ab-8951-0a01f40b7512\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1caa6e5-5t6j8" Nov 03 02:32:44.162406 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:44.162385 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4748e14a-d17c-43ab-8951-0a01f40b7512-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1caa6e5-5t6j8\" (UID: \"4748e14a-d17c-43ab-8951-0a01f40b7512\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1caa6e5-5t6j8" Nov 03 02:32:44.212795 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:44.212754 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h767s\" (UniqueName: \"kubernetes.io/projected/4748e14a-d17c-43ab-8951-0a01f40b7512-kube-api-access-h767s\") pod \"maas-api-revoke-sub-b1caa6e5-5t6j8\" (UID: \"4748e14a-d17c-43ab-8951-0a01f40b7512\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1caa6e5-5t6j8" Nov 03 02:32:44.305600 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:44.305556 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1caa6e5-5t6j8" Nov 03 02:32:44.677622 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:44.677545 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1caa6e5-5t6j8"] Nov 03 02:32:44.682526 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:32:44.682496 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4748e14a_d17c_43ab_8951_0a01f40b7512.slice/crio-335a95137126b074c6f69cda1925e6a4c25e3b771ffd2808f6a4fdbadbd5b652 WatchSource:0}: Error finding container 335a95137126b074c6f69cda1925e6a4c25e3b771ffd2808f6a4fdbadbd5b652: Status 404 returned error can't find the container with id 335a95137126b074c6f69cda1925e6a4c25e3b771ffd2808f6a4fdbadbd5b652 Nov 03 02:32:45.549080 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:45.549042 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1caa6e5-5t6j8" event={"ID":"4748e14a-d17c-43ab-8951-0a01f40b7512","Type":"ContainerStarted","Data":"41fa1bc5a0243e2b57427ce8a166d13f1fbab51076f5289b772000f65bd17c98"} Nov 03 02:32:45.549491 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:45.549095 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1caa6e5-5t6j8" event={"ID":"4748e14a-d17c-43ab-8951-0a01f40b7512","Type":"ContainerStarted","Data":"335a95137126b074c6f69cda1925e6a4c25e3b771ffd2808f6a4fdbadbd5b652"} Nov 03 02:32:45.657865 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:45.657804 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1caa6e5-5t6j8" podStartSLOduration=2.657789046 podStartE2EDuration="2.657789046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:32:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:32:45.654956101 +0000 UTC m=+2442.914154256" watchObservedRunningTime="2026-11-03 02:32:45.657789046 +0000 UTC m=+2442.916987199" Nov 03 02:32:46.556174 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:46.556135 5605 generic.go:356] "Generic (PLEG): container finished" podID="4748e14a-d17c-43ab-8951-0a01f40b7512" containerID="41fa1bc5a0243e2b57427ce8a166d13f1fbab51076f5289b772000f65bd17c98" exitCode=0 Nov 03 02:32:46.556627 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:46.556217 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1caa6e5-5t6j8" event={"ID":"4748e14a-d17c-43ab-8951-0a01f40b7512","Type":"ContainerDied","Data":"41fa1bc5a0243e2b57427ce8a166d13f1fbab51076f5289b772000f65bd17c98"} Nov 03 02:32:47.702418 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:47.702396 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1caa6e5-5t6j8" Nov 03 02:32:47.804027 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:47.803994 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4748e14a-d17c-43ab-8951-0a01f40b7512-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4748e14a-d17c-43ab-8951-0a01f40b7512-maas-api-openshift-ca\") pod \"4748e14a-d17c-43ab-8951-0a01f40b7512\" (UID: \"4748e14a-d17c-43ab-8951-0a01f40b7512\") " Nov 03 02:32:47.804027 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:47.804034 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4748e14a-d17c-43ab-8951-0a01f40b7512-kube-api-access-h767s\" (UniqueName: \"kubernetes.io/projected/4748e14a-d17c-43ab-8951-0a01f40b7512-kube-api-access-h767s\") pod \"4748e14a-d17c-43ab-8951-0a01f40b7512\" (UID: \"4748e14a-d17c-43ab-8951-0a01f40b7512\") " Nov 03 02:32:47.804242 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:47.804077 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4748e14a-d17c-43ab-8951-0a01f40b7512-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4748e14a-d17c-43ab-8951-0a01f40b7512-maas-api-xks-ca\") pod \"4748e14a-d17c-43ab-8951-0a01f40b7512\" (UID: \"4748e14a-d17c-43ab-8951-0a01f40b7512\") " Nov 03 02:32:47.804448 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:47.804421 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4748e14a-d17c-43ab-8951-0a01f40b7512-maas-api-openshift-ca" pod "4748e14a-d17c-43ab-8951-0a01f40b7512" (UID: "4748e14a-d17c-43ab-8951-0a01f40b7512"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:32:47.804589 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:47.804572 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4748e14a-d17c-43ab-8951-0a01f40b7512-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:32:47.806185 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:47.806162 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4748e14a-d17c-43ab-8951-0a01f40b7512-maas-api-xks-ca" pod "4748e14a-d17c-43ab-8951-0a01f40b7512" (UID: "4748e14a-d17c-43ab-8951-0a01f40b7512"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:32:47.806340 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:47.806319 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4748e14a-d17c-43ab-8951-0a01f40b7512-kube-api-access-h767s" pod "4748e14a-d17c-43ab-8951-0a01f40b7512" (UID: "4748e14a-d17c-43ab-8951-0a01f40b7512"). InnerVolumeSpecName "kube-api-access-h767s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:32:47.905962 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:47.905877 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h767s\" (UniqueName: \"kubernetes.io/projected/4748e14a-d17c-43ab-8951-0a01f40b7512-kube-api-access-h767s\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:32:47.905962 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:47.905909 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4748e14a-d17c-43ab-8951-0a01f40b7512-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:32:48.570609 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:48.570574 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1caa6e5-5t6j8" event={"ID":"4748e14a-d17c-43ab-8951-0a01f40b7512","Type":"ContainerDied","Data":"335a95137126b074c6f69cda1925e6a4c25e3b771ffd2808f6a4fdbadbd5b652"} Nov 03 02:32:48.570609 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:48.570606 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1caa6e5-5t6j8" Nov 03 02:32:48.570831 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:48.570627 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="335a95137126b074c6f69cda1925e6a4c25e3b771ffd2808f6a4fdbadbd5b652" Nov 03 02:32:52.583678 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:52.583641 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e16e9de-292gr"] Nov 03 02:32:52.584189 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:52.584070 5605 cpu_manager.go:414] "containerMap: removing container" podUID="4748e14a-d17c-43ab-8951-0a01f40b7512" containerName="revoke-keys" Nov 03 02:32:52.584189 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:52.584080 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4748e14a-d17c-43ab-8951-0a01f40b7512" containerName="revoke-keys" Nov 03 02:32:52.584274 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:52.584203 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="4748e14a-d17c-43ab-8951-0a01f40b7512" containerName="revoke-keys" Nov 03 02:32:52.588015 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:52.587992 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e16e9de-292gr" Nov 03 02:32:52.640763 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:52.640732 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:32:52.643463 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:52.643436 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e16e9de-292gr"] Nov 03 02:32:52.659643 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:52.659613 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72f7eed5-46c0-4c61-bb2e-7cfe945667c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e16e9de-292gr\" (UID: \"72f7eed5-46c0-4c61-bb2e-7cfe945667c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e16e9de-292gr" Nov 03 02:32:52.659797 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:52.659772 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qj2m\" (UniqueName: \"kubernetes.io/projected/72f7eed5-46c0-4c61-bb2e-7cfe945667c9-kube-api-access-6qj2m\") pod \"maas-api-revoke-sub-0e16e9de-292gr\" (UID: \"72f7eed5-46c0-4c61-bb2e-7cfe945667c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e16e9de-292gr" Nov 03 02:32:52.659870 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:52.659819 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72f7eed5-46c0-4c61-bb2e-7cfe945667c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e16e9de-292gr\" (UID: \"72f7eed5-46c0-4c61-bb2e-7cfe945667c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e16e9de-292gr" Nov 03 02:32:52.686541 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:52.686513 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:32:52.687425 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:52.687407 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:32:52.687480 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:52.687408 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:32:52.760971 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:52.760937 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72f7eed5-46c0-4c61-bb2e-7cfe945667c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e16e9de-292gr\" (UID: \"72f7eed5-46c0-4c61-bb2e-7cfe945667c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e16e9de-292gr" Nov 03 02:32:52.761149 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:52.761030 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qj2m\" (UniqueName: \"kubernetes.io/projected/72f7eed5-46c0-4c61-bb2e-7cfe945667c9-kube-api-access-6qj2m\") pod \"maas-api-revoke-sub-0e16e9de-292gr\" (UID: \"72f7eed5-46c0-4c61-bb2e-7cfe945667c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e16e9de-292gr" Nov 03 02:32:52.761149 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:52.761056 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72f7eed5-46c0-4c61-bb2e-7cfe945667c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e16e9de-292gr\" (UID: \"72f7eed5-46c0-4c61-bb2e-7cfe945667c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e16e9de-292gr" Nov 03 02:32:52.761693 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:52.761666 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72f7eed5-46c0-4c61-bb2e-7cfe945667c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e16e9de-292gr\" (UID: \"72f7eed5-46c0-4c61-bb2e-7cfe945667c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e16e9de-292gr" Nov 03 02:32:52.763507 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:52.763483 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72f7eed5-46c0-4c61-bb2e-7cfe945667c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e16e9de-292gr\" (UID: \"72f7eed5-46c0-4c61-bb2e-7cfe945667c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e16e9de-292gr" Nov 03 02:32:52.816819 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:52.816786 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qj2m\" (UniqueName: \"kubernetes.io/projected/72f7eed5-46c0-4c61-bb2e-7cfe945667c9-kube-api-access-6qj2m\") pod \"maas-api-revoke-sub-0e16e9de-292gr\" (UID: \"72f7eed5-46c0-4c61-bb2e-7cfe945667c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e16e9de-292gr" Nov 03 02:32:52.901820 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:52.901750 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e16e9de-292gr" Nov 03 02:32:53.280437 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:53.280334 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e16e9de-292gr"] Nov 03 02:32:53.286995 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:32:53.286963 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72f7eed5_46c0_4c61_bb2e_7cfe945667c9.slice/crio-1bdb05e6855fdbd6b8309971830323ae882e9e9e781bc5017ef3578a2cf02663 WatchSource:0}: Error finding container 1bdb05e6855fdbd6b8309971830323ae882e9e9e781bc5017ef3578a2cf02663: Status 404 returned error can't find the container with id 1bdb05e6855fdbd6b8309971830323ae882e9e9e781bc5017ef3578a2cf02663 Nov 03 02:32:53.608830 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:53.608794 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e16e9de-292gr" event={"ID":"72f7eed5-46c0-4c61-bb2e-7cfe945667c9","Type":"ContainerStarted","Data":"6d90d7638aa4ed0a78feed1f2d2d439a3198f94fe75e57dc98a3346627d71e2f"} Nov 03 02:32:53.609158 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:53.608879 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e16e9de-292gr" event={"ID":"72f7eed5-46c0-4c61-bb2e-7cfe945667c9","Type":"ContainerStarted","Data":"1bdb05e6855fdbd6b8309971830323ae882e9e9e781bc5017ef3578a2cf02663"} Nov 03 02:32:53.721225 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:53.721178 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e16e9de-292gr" podStartSLOduration=1.721165306 podStartE2EDuration="1.721165306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:32:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:32:53.719037169 +0000 UTC m=+2450.978235324" watchObservedRunningTime="2026-11-03 02:32:53.721165306 +0000 UTC m=+2450.980363501" Nov 03 02:32:53.917310 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:32:53.917224 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:32:54.616260 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:54.616228 5605 generic.go:356] "Generic (PLEG): container finished" podID="72f7eed5-46c0-4c61-bb2e-7cfe945667c9" containerID="6d90d7638aa4ed0a78feed1f2d2d439a3198f94fe75e57dc98a3346627d71e2f" exitCode=0 Nov 03 02:32:54.616682 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:54.616316 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e16e9de-292gr" event={"ID":"72f7eed5-46c0-4c61-bb2e-7cfe945667c9","Type":"ContainerDied","Data":"6d90d7638aa4ed0a78feed1f2d2d439a3198f94fe75e57dc98a3346627d71e2f"} Nov 03 02:32:55.164717 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:55.164687 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5"] Nov 03 02:32:55.164929 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:55.164907 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5" podUID="6e8485b2-ec6f-4344-bee7-37495006a34c" containerName="istio-proxy" containerID="cri-o://4fb8e5c48c374a4d444dac5503aa9b95e1f55b06ae604b8d66138b8c2f355d38" gracePeriod=30 Nov 03 02:32:55.762093 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:55.762069 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e16e9de-292gr" Nov 03 02:32:55.894257 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:55.894193 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72f7eed5-46c0-4c61-bb2e-7cfe945667c9-kube-api-access-6qj2m\" (UniqueName: \"kubernetes.io/projected/72f7eed5-46c0-4c61-bb2e-7cfe945667c9-kube-api-access-6qj2m\") pod \"72f7eed5-46c0-4c61-bb2e-7cfe945667c9\" (UID: \"72f7eed5-46c0-4c61-bb2e-7cfe945667c9\") " Nov 03 02:32:55.894257 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:55.894245 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72f7eed5-46c0-4c61-bb2e-7cfe945667c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72f7eed5-46c0-4c61-bb2e-7cfe945667c9-maas-api-openshift-ca\") pod \"72f7eed5-46c0-4c61-bb2e-7cfe945667c9\" (UID: \"72f7eed5-46c0-4c61-bb2e-7cfe945667c9\") " Nov 03 02:32:55.894486 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:55.894272 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72f7eed5-46c0-4c61-bb2e-7cfe945667c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72f7eed5-46c0-4c61-bb2e-7cfe945667c9-maas-api-xks-ca\") pod \"72f7eed5-46c0-4c61-bb2e-7cfe945667c9\" (UID: \"72f7eed5-46c0-4c61-bb2e-7cfe945667c9\") " Nov 03 02:32:55.894647 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:55.894616 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72f7eed5-46c0-4c61-bb2e-7cfe945667c9-maas-api-openshift-ca" pod "72f7eed5-46c0-4c61-bb2e-7cfe945667c9" (UID: "72f7eed5-46c0-4c61-bb2e-7cfe945667c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:32:55.894784 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:55.894766 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72f7eed5-46c0-4c61-bb2e-7cfe945667c9-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:32:55.896341 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:55.896312 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72f7eed5-46c0-4c61-bb2e-7cfe945667c9-maas-api-xks-ca" pod "72f7eed5-46c0-4c61-bb2e-7cfe945667c9" (UID: "72f7eed5-46c0-4c61-bb2e-7cfe945667c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:32:55.896583 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:55.896565 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72f7eed5-46c0-4c61-bb2e-7cfe945667c9-kube-api-access-6qj2m" pod "72f7eed5-46c0-4c61-bb2e-7cfe945667c9" (UID: "72f7eed5-46c0-4c61-bb2e-7cfe945667c9"). InnerVolumeSpecName "kube-api-access-6qj2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:32:55.995841 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:55.995800 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qj2m\" (UniqueName: \"kubernetes.io/projected/72f7eed5-46c0-4c61-bb2e-7cfe945667c9-kube-api-access-6qj2m\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:32:55.995841 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:55.995837 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72f7eed5-46c0-4c61-bb2e-7cfe945667c9-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:32:56.630552 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:56.630503 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e16e9de-292gr" event={"ID":"72f7eed5-46c0-4c61-bb2e-7cfe945667c9","Type":"ContainerDied","Data":"1bdb05e6855fdbd6b8309971830323ae882e9e9e781bc5017ef3578a2cf02663"} Nov 03 02:32:56.630763 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:56.630608 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1bdb05e6855fdbd6b8309971830323ae882e9e9e781bc5017ef3578a2cf02663" Nov 03 02:32:56.630763 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:56.630700 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e16e9de-292gr" Nov 03 02:32:56.740718 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:56.740681 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1df32a7a-nq7p2"] Nov 03 02:32:56.741167 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:56.741149 5605 cpu_manager.go:414] "containerMap: removing container" podUID="72f7eed5-46c0-4c61-bb2e-7cfe945667c9" containerName="revoke-keys" Nov 03 02:32:56.741260 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:56.741169 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72f7eed5-46c0-4c61-bb2e-7cfe945667c9" containerName="revoke-keys" Nov 03 02:32:56.741322 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:56.741303 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="72f7eed5-46c0-4c61-bb2e-7cfe945667c9" containerName="revoke-keys" Nov 03 02:32:56.744428 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:56.744408 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1df32a7a-nq7p2" Nov 03 02:32:56.780590 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:56.780563 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1df32a7a-nq7p2"] Nov 03 02:32:56.793782 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:56.793715 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:32:56.793922 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:56.793872 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:32:56.793999 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:56.793960 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:32:56.822836 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:56.822815 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:32:56.905488 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:56.905426 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbj88\" (UniqueName: \"kubernetes.io/projected/d9269e01-6892-4797-815a-b46f93340eae-kube-api-access-vbj88\") pod \"maas-api-revoke-sub-1df32a7a-nq7p2\" (UID: \"d9269e01-6892-4797-815a-b46f93340eae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1df32a7a-nq7p2" Nov 03 02:32:56.905488 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:56.905464 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9269e01-6892-4797-815a-b46f93340eae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1df32a7a-nq7p2\" (UID: \"d9269e01-6892-4797-815a-b46f93340eae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1df32a7a-nq7p2" Nov 03 02:32:56.905663 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:56.905501 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9269e01-6892-4797-815a-b46f93340eae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1df32a7a-nq7p2\" (UID: \"d9269e01-6892-4797-815a-b46f93340eae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1df32a7a-nq7p2" Nov 03 02:32:57.006635 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:57.006609 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9269e01-6892-4797-815a-b46f93340eae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1df32a7a-nq7p2\" (UID: \"d9269e01-6892-4797-815a-b46f93340eae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1df32a7a-nq7p2" Nov 03 02:32:57.006804 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:57.006683 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9269e01-6892-4797-815a-b46f93340eae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1df32a7a-nq7p2\" (UID: \"d9269e01-6892-4797-815a-b46f93340eae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1df32a7a-nq7p2" Nov 03 02:32:57.006904 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:57.006887 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbj88\" (UniqueName: \"kubernetes.io/projected/d9269e01-6892-4797-815a-b46f93340eae-kube-api-access-vbj88\") pod \"maas-api-revoke-sub-1df32a7a-nq7p2\" (UID: \"d9269e01-6892-4797-815a-b46f93340eae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1df32a7a-nq7p2" Nov 03 02:32:57.007515 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:57.007491 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9269e01-6892-4797-815a-b46f93340eae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1df32a7a-nq7p2\" (UID: \"d9269e01-6892-4797-815a-b46f93340eae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1df32a7a-nq7p2" Nov 03 02:32:57.008947 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:57.008930 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9269e01-6892-4797-815a-b46f93340eae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1df32a7a-nq7p2\" (UID: \"d9269e01-6892-4797-815a-b46f93340eae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1df32a7a-nq7p2" Nov 03 02:32:57.062106 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:57.062080 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbj88\" (UniqueName: \"kubernetes.io/projected/d9269e01-6892-4797-815a-b46f93340eae-kube-api-access-vbj88\") pod \"maas-api-revoke-sub-1df32a7a-nq7p2\" (UID: \"d9269e01-6892-4797-815a-b46f93340eae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1df32a7a-nq7p2" Nov 03 02:32:57.357185 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:57.357158 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1df32a7a-nq7p2" Nov 03 02:32:57.738805 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:57.738682 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1df32a7a-nq7p2"] Nov 03 02:32:57.741824 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:32:57.741795 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9269e01_6892_4797_815a_b46f93340eae.slice/crio-2cf723fa1a0a6ef568c03205051230fca57a2233359a1fafa9aa819dbcfc3ce9 WatchSource:0}: Error finding container 2cf723fa1a0a6ef568c03205051230fca57a2233359a1fafa9aa819dbcfc3ce9: Status 404 returned error can't find the container with id 2cf723fa1a0a6ef568c03205051230fca57a2233359a1fafa9aa819dbcfc3ce9 Nov 03 02:32:58.646844 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:58.646810 5605 generic.go:356] "Generic (PLEG): container finished" podID="d9269e01-6892-4797-815a-b46f93340eae" containerID="8faa9ea80d5c1cc9f447dc3b1d7c8606ce366f05f66ff6514d3502c5f12e952a" exitCode=0 Nov 03 02:32:58.647222 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:58.646890 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1df32a7a-nq7p2" event={"ID":"d9269e01-6892-4797-815a-b46f93340eae","Type":"ContainerDied","Data":"8faa9ea80d5c1cc9f447dc3b1d7c8606ce366f05f66ff6514d3502c5f12e952a"} Nov 03 02:32:58.647222 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:58.646947 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1df32a7a-nq7p2" event={"ID":"d9269e01-6892-4797-815a-b46f93340eae","Type":"ContainerStarted","Data":"2cf723fa1a0a6ef568c03205051230fca57a2233359a1fafa9aa819dbcfc3ce9"} Nov 03 02:32:58.868486 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:58.868412 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts"] Nov 03 02:32:58.872193 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:58.872171 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts" Nov 03 02:32:58.882127 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:58.882103 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9"] Nov 03 02:32:58.885624 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:58.885608 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9" Nov 03 02:32:58.921076 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:58.921051 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-a0e72e-openshift-default-dockercfg-cskrh\"" Nov 03 02:32:58.927435 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:58.927409 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts"] Nov 03 02:32:58.943605 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:58.943577 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9"] Nov 03 02:32:59.031240 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.031209 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8fps\" (UniqueName: \"kubernetes.io/projected/f8c546e7-4bc7-459f-8fa8-65e18c1585b3-kube-api-access-h8fps\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9\" (UID: \"f8c546e7-4bc7-459f-8fa8-65e18c1585b3\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9" Nov 03 02:32:59.031441 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.031289 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7-istiod-ca-cert\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts\" (UID: \"77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts" Nov 03 02:32:59.031441 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.031370 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7-workload-certs\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts\" (UID: \"77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts" Nov 03 02:32:59.031441 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.031431 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7-istio-data\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts\" (UID: \"77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts" Nov 03 02:32:59.031600 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.031465 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c546e7-4bc7-459f-8fa8-65e18c1585b3-credential-socket\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9\" (UID: \"f8c546e7-4bc7-459f-8fa8-65e18c1585b3\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9" Nov 03 02:32:59.031600 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.031485 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8c546e7-4bc7-459f-8fa8-65e18c1585b3-workload-certs\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9\" (UID: \"f8c546e7-4bc7-459f-8fa8-65e18c1585b3\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9" Nov 03 02:32:59.031600 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.031511 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7-credential-socket\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts\" (UID: \"77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts" Nov 03 02:32:59.031719 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.031617 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8c546e7-4bc7-459f-8fa8-65e18c1585b3-istiod-ca-cert\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9\" (UID: \"f8c546e7-4bc7-459f-8fa8-65e18c1585b3\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9" Nov 03 02:32:59.031719 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.031646 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7-istio-podinfo\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts\" (UID: \"77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts" Nov 03 02:32:59.031719 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.031682 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8c546e7-4bc7-459f-8fa8-65e18c1585b3-istio-token\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9\" (UID: \"f8c546e7-4bc7-459f-8fa8-65e18c1585b3\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9" Nov 03 02:32:59.031837 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.031744 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8c546e7-4bc7-459f-8fa8-65e18c1585b3-istio-podinfo\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9\" (UID: \"f8c546e7-4bc7-459f-8fa8-65e18c1585b3\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9" Nov 03 02:32:59.031837 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.031825 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7-istio-token\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts\" (UID: \"77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts" Nov 03 02:32:59.031904 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.031864 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7-istio-envoy\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts\" (UID: \"77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts" Nov 03 02:32:59.031904 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.031886 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8lck\" (UniqueName: \"kubernetes.io/projected/77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7-kube-api-access-x8lck\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts\" (UID: \"77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts" Nov 03 02:32:59.032003 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.031909 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c546e7-4bc7-459f-8fa8-65e18c1585b3-workload-socket\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9\" (UID: \"f8c546e7-4bc7-459f-8fa8-65e18c1585b3\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9" Nov 03 02:32:59.032003 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.031945 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8c546e7-4bc7-459f-8fa8-65e18c1585b3-istio-data\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9\" (UID: \"f8c546e7-4bc7-459f-8fa8-65e18c1585b3\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9" Nov 03 02:32:59.032110 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.032059 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8c546e7-4bc7-459f-8fa8-65e18c1585b3-istio-envoy\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9\" (UID: \"f8c546e7-4bc7-459f-8fa8-65e18c1585b3\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9" Nov 03 02:32:59.032110 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.032097 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7-workload-socket\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts\" (UID: \"77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts" Nov 03 02:32:59.133553 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.133453 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7-istiod-ca-cert\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts\" (UID: \"77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts" Nov 03 02:32:59.133553 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.133508 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7-workload-certs\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts\" (UID: \"77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts" Nov 03 02:32:59.133855 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.133559 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7-istio-data\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts\" (UID: \"77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts" Nov 03 02:32:59.133855 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.133595 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c546e7-4bc7-459f-8fa8-65e18c1585b3-credential-socket\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9\" (UID: \"f8c546e7-4bc7-459f-8fa8-65e18c1585b3\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9" Nov 03 02:32:59.133855 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.133620 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8c546e7-4bc7-459f-8fa8-65e18c1585b3-workload-certs\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9\" (UID: \"f8c546e7-4bc7-459f-8fa8-65e18c1585b3\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9" Nov 03 02:32:59.133855 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.133655 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7-credential-socket\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts\" (UID: \"77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts" Nov 03 02:32:59.133855 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.133690 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8c546e7-4bc7-459f-8fa8-65e18c1585b3-istiod-ca-cert\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9\" (UID: \"f8c546e7-4bc7-459f-8fa8-65e18c1585b3\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9" Nov 03 02:32:59.133855 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.133715 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7-istio-podinfo\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts\" (UID: \"77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts" Nov 03 02:32:59.133855 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.133747 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8c546e7-4bc7-459f-8fa8-65e18c1585b3-istio-token\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9\" (UID: \"f8c546e7-4bc7-459f-8fa8-65e18c1585b3\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9" Nov 03 02:32:59.133855 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.133779 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8c546e7-4bc7-459f-8fa8-65e18c1585b3-istio-podinfo\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9\" (UID: \"f8c546e7-4bc7-459f-8fa8-65e18c1585b3\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9" Nov 03 02:32:59.133855 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.133836 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7-istio-token\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts\" (UID: \"77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts" Nov 03 02:32:59.134311 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.133884 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7-istio-envoy\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts\" (UID: \"77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts" Nov 03 02:32:59.134311 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.133912 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8lck\" (UniqueName: \"kubernetes.io/projected/77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7-kube-api-access-x8lck\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts\" (UID: \"77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts" Nov 03 02:32:59.134311 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.133949 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c546e7-4bc7-459f-8fa8-65e18c1585b3-workload-socket\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9\" (UID: \"f8c546e7-4bc7-459f-8fa8-65e18c1585b3\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9" Nov 03 02:32:59.134311 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.133980 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8c546e7-4bc7-459f-8fa8-65e18c1585b3-istio-data\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9\" (UID: \"f8c546e7-4bc7-459f-8fa8-65e18c1585b3\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9" Nov 03 02:32:59.134311 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.134007 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c546e7-4bc7-459f-8fa8-65e18c1585b3-credential-socket\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9\" (UID: \"f8c546e7-4bc7-459f-8fa8-65e18c1585b3\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9" Nov 03 02:32:59.134311 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.134027 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8c546e7-4bc7-459f-8fa8-65e18c1585b3-istio-envoy\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9\" (UID: \"f8c546e7-4bc7-459f-8fa8-65e18c1585b3\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9" Nov 03 02:32:59.134311 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.134085 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7-workload-socket\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts\" (UID: \"77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts" Nov 03 02:32:59.134311 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.134007 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7-workload-certs\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts\" (UID: \"77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts" Nov 03 02:32:59.134311 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.134105 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7-credential-socket\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts\" (UID: \"77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts" Nov 03 02:32:59.134311 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.134083 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8c546e7-4bc7-459f-8fa8-65e18c1585b3-workload-certs\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9\" (UID: \"f8c546e7-4bc7-459f-8fa8-65e18c1585b3\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9" Nov 03 02:32:59.134311 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.134151 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8fps\" (UniqueName: \"kubernetes.io/projected/f8c546e7-4bc7-459f-8fa8-65e18c1585b3-kube-api-access-h8fps\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9\" (UID: \"f8c546e7-4bc7-459f-8fa8-65e18c1585b3\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9" Nov 03 02:32:59.134930 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.134327 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7-istiod-ca-cert\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts\" (UID: \"77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts" Nov 03 02:32:59.134930 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.134419 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7-workload-socket\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts\" (UID: \"77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts" Nov 03 02:32:59.134930 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.134678 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c546e7-4bc7-459f-8fa8-65e18c1585b3-workload-socket\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9\" (UID: \"f8c546e7-4bc7-459f-8fa8-65e18c1585b3\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9" Nov 03 02:32:59.134930 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.134912 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8c546e7-4bc7-459f-8fa8-65e18c1585b3-istio-data\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9\" (UID: \"f8c546e7-4bc7-459f-8fa8-65e18c1585b3\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9" Nov 03 02:32:59.135150 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.134940 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8c546e7-4bc7-459f-8fa8-65e18c1585b3-istiod-ca-cert\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9\" (UID: \"f8c546e7-4bc7-459f-8fa8-65e18c1585b3\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9" Nov 03 02:32:59.135150 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.134974 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7-istio-data\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts\" (UID: \"77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts" Nov 03 02:32:59.136816 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.136737 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7-istio-envoy\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts\" (UID: \"77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts" Nov 03 02:32:59.136816 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.136738 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8c546e7-4bc7-459f-8fa8-65e18c1585b3-istio-envoy\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9\" (UID: \"f8c546e7-4bc7-459f-8fa8-65e18c1585b3\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9" Nov 03 02:32:59.137040 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.137017 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8c546e7-4bc7-459f-8fa8-65e18c1585b3-istio-podinfo\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9\" (UID: \"f8c546e7-4bc7-459f-8fa8-65e18c1585b3\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9" Nov 03 02:32:59.137232 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.137211 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7-istio-podinfo\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts\" (UID: \"77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts" Nov 03 02:32:59.188799 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.188760 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8c546e7-4bc7-459f-8fa8-65e18c1585b3-istio-token\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9\" (UID: \"f8c546e7-4bc7-459f-8fa8-65e18c1585b3\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9" Nov 03 02:32:59.189044 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.189027 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8fps\" (UniqueName: \"kubernetes.io/projected/f8c546e7-4bc7-459f-8fa8-65e18c1585b3-kube-api-access-h8fps\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9\" (UID: \"f8c546e7-4bc7-459f-8fa8-65e18c1585b3\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9" Nov 03 02:32:59.189206 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.189173 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7-istio-token\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts\" (UID: \"77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts" Nov 03 02:32:59.189312 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.189284 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8lck\" (UniqueName: \"kubernetes.io/projected/77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7-kube-api-access-x8lck\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts\" (UID: \"77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts" Nov 03 02:32:59.198133 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.198113 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9" Nov 03 02:32:59.485805 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.485727 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts" Nov 03 02:32:59.625407 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.623076 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9"] Nov 03 02:32:59.628658 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:32:59.628626 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8c546e7_4bc7_459f_8fa8_65e18c1585b3.slice/crio-a3728bfcf6732b2dd20ba14f711856c329fd32bf6a13b1be61452766c957afec WatchSource:0}: Error finding container a3728bfcf6732b2dd20ba14f711856c329fd32bf6a13b1be61452766c957afec: Status 404 returned error can't find the container with id a3728bfcf6732b2dd20ba14f711856c329fd32bf6a13b1be61452766c957afec Nov 03 02:32:59.631260 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.631223 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:32:59.631371 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.631293 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:32:59.631371 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.631322 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:32:59.655500 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.655472 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9" event={"ID":"f8c546e7-4bc7-459f-8fa8-65e18c1585b3","Type":"ContainerStarted","Data":"a3728bfcf6732b2dd20ba14f711856c329fd32bf6a13b1be61452766c957afec"} Nov 03 02:32:59.795779 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.795755 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1df32a7a-nq7p2" Nov 03 02:32:59.865294 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.865258 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts"] Nov 03 02:32:59.869598 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:32:59.869568 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77ce2f96_bb53_40d1_bfd4_a0c5b7ca75f7.slice/crio-5aa35fee679478965d913aba4c113b27629234208d27d5466e74df37f243cf0c WatchSource:0}: Error finding container 5aa35fee679478965d913aba4c113b27629234208d27d5466e74df37f243cf0c: Status 404 returned error can't find the container with id 5aa35fee679478965d913aba4c113b27629234208d27d5466e74df37f243cf0c Nov 03 02:32:59.872692 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.872655 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:32:59.872807 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.872718 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:32:59.872807 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.872749 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:32:59.949915 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.949878 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9269e01-6892-4797-815a-b46f93340eae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9269e01-6892-4797-815a-b46f93340eae-maas-api-xks-ca\") pod \"d9269e01-6892-4797-815a-b46f93340eae\" (UID: \"d9269e01-6892-4797-815a-b46f93340eae\") " Nov 03 02:32:59.950103 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.950085 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9269e01-6892-4797-815a-b46f93340eae-kube-api-access-vbj88\" (UniqueName: \"kubernetes.io/projected/d9269e01-6892-4797-815a-b46f93340eae-kube-api-access-vbj88\") pod \"d9269e01-6892-4797-815a-b46f93340eae\" (UID: \"d9269e01-6892-4797-815a-b46f93340eae\") " Nov 03 02:32:59.950158 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.950131 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9269e01-6892-4797-815a-b46f93340eae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9269e01-6892-4797-815a-b46f93340eae-maas-api-openshift-ca\") pod \"d9269e01-6892-4797-815a-b46f93340eae\" (UID: \"d9269e01-6892-4797-815a-b46f93340eae\") " Nov 03 02:32:59.950824 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.950798 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9269e01-6892-4797-815a-b46f93340eae-maas-api-openshift-ca" pod "d9269e01-6892-4797-815a-b46f93340eae" (UID: "d9269e01-6892-4797-815a-b46f93340eae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:32:59.954928 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.954900 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9269e01-6892-4797-815a-b46f93340eae-kube-api-access-vbj88" pod "d9269e01-6892-4797-815a-b46f93340eae" (UID: "d9269e01-6892-4797-815a-b46f93340eae"). InnerVolumeSpecName "kube-api-access-vbj88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:32:59.957535 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:32:59.957499 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9269e01-6892-4797-815a-b46f93340eae-maas-api-xks-ca" pod "d9269e01-6892-4797-815a-b46f93340eae" (UID: "d9269e01-6892-4797-815a-b46f93340eae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:33:00.052209 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:00.052172 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9269e01-6892-4797-815a-b46f93340eae-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:00.052209 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:00.052209 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbj88\" (UniqueName: \"kubernetes.io/projected/d9269e01-6892-4797-815a-b46f93340eae-kube-api-access-vbj88\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:00.052500 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:00.052225 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9269e01-6892-4797-815a-b46f93340eae-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:00.664660 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:00.664622 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1df32a7a-nq7p2" Nov 03 02:33:00.664660 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:00.664635 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1df32a7a-nq7p2" event={"ID":"d9269e01-6892-4797-815a-b46f93340eae","Type":"ContainerDied","Data":"2cf723fa1a0a6ef568c03205051230fca57a2233359a1fafa9aa819dbcfc3ce9"} Nov 03 02:33:00.665261 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:00.664689 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2cf723fa1a0a6ef568c03205051230fca57a2233359a1fafa9aa819dbcfc3ce9" Nov 03 02:33:00.666425 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:00.666396 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9" event={"ID":"f8c546e7-4bc7-459f-8fa8-65e18c1585b3","Type":"ContainerStarted","Data":"3295455d507f7dd60fb02ea4f8e29dbf7f192fed0c1f023c1da904a6fd24c8b7"} Nov 03 02:33:00.668188 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:00.668160 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts" event={"ID":"77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7","Type":"ContainerStarted","Data":"bb5d9985b1995763ff8eae2b182a6162980bbb304b9f543a73f56c1b6244cac2"} Nov 03 02:33:00.668293 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:00.668211 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts" event={"ID":"77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7","Type":"ContainerStarted","Data":"5aa35fee679478965d913aba4c113b27629234208d27d5466e74df37f243cf0c"} Nov 03 02:33:00.670075 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:00.670056 5605 generic.go:356] "Generic (PLEG): container finished" podID="6e8485b2-ec6f-4344-bee7-37495006a34c" containerID="4fb8e5c48c374a4d444dac5503aa9b95e1f55b06ae604b8d66138b8c2f355d38" exitCode=0 Nov 03 02:33:00.670175 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:00.670116 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5" event={"ID":"6e8485b2-ec6f-4344-bee7-37495006a34c","Type":"ContainerDied","Data":"4fb8e5c48c374a4d444dac5503aa9b95e1f55b06ae604b8d66138b8c2f355d38"} Nov 03 02:33:00.670214 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:00.670156 5605 scope.go:122] "RemoveContainer" containerID="d0a29f070628364b4e12e88048780351f44c00f620f53883a89bebec92ab2128" Nov 03 02:33:00.890670 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:00.890627 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5" Nov 03 02:33:00.895025 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:00.894963 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts" podStartSLOduration=2.8949467909999997 podStartE2EDuration="2.894946791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:32:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:33:00.893764711 +0000 UTC m=+2458.152962865" watchObservedRunningTime="2026-11-03 02:33:00.894946791 +0000 UTC m=+2458.154144945" Nov 03 02:33:00.895268 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:00.895156 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9" podStartSLOduration=2.8951496839999997 podStartE2EDuration="2.895149684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:32:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:33:00.784740118 +0000 UTC m=+2458.043938285" watchObservedRunningTime="2026-11-03 02:33:00.895149684 +0000 UTC m=+2458.154347842" Nov 03 02:33:01.081381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:01.080756 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6e8485b2-ec6f-4344-bee7-37495006a34c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e8485b2-ec6f-4344-bee7-37495006a34c-istio-podinfo\") pod \"6e8485b2-ec6f-4344-bee7-37495006a34c\" (UID: \"6e8485b2-ec6f-4344-bee7-37495006a34c\") " Nov 03 02:33:01.081381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:01.080816 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e8485b2-ec6f-4344-bee7-37495006a34c-istio-token\" (UniqueName: \"kubernetes.io/projected/6e8485b2-ec6f-4344-bee7-37495006a34c-istio-token\") pod \"6e8485b2-ec6f-4344-bee7-37495006a34c\" (UID: \"6e8485b2-ec6f-4344-bee7-37495006a34c\") " Nov 03 02:33:01.081381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:01.080922 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e8485b2-ec6f-4344-bee7-37495006a34c-kube-api-access-wkzx8\" (UniqueName: \"kubernetes.io/projected/6e8485b2-ec6f-4344-bee7-37495006a34c-kube-api-access-wkzx8\") pod \"6e8485b2-ec6f-4344-bee7-37495006a34c\" (UID: \"6e8485b2-ec6f-4344-bee7-37495006a34c\") " Nov 03 02:33:01.081381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:01.080987 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e8485b2-ec6f-4344-bee7-37495006a34c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e8485b2-ec6f-4344-bee7-37495006a34c-workload-socket\") pod \"6e8485b2-ec6f-4344-bee7-37495006a34c\" (UID: \"6e8485b2-ec6f-4344-bee7-37495006a34c\") " Nov 03 02:33:01.081381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:01.081077 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e8485b2-ec6f-4344-bee7-37495006a34c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e8485b2-ec6f-4344-bee7-37495006a34c-istio-data\") pod \"6e8485b2-ec6f-4344-bee7-37495006a34c\" (UID: \"6e8485b2-ec6f-4344-bee7-37495006a34c\") " Nov 03 02:33:01.081381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:01.081177 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e8485b2-ec6f-4344-bee7-37495006a34c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e8485b2-ec6f-4344-bee7-37495006a34c-istio-envoy\") pod \"6e8485b2-ec6f-4344-bee7-37495006a34c\" (UID: \"6e8485b2-ec6f-4344-bee7-37495006a34c\") " Nov 03 02:33:01.081381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:01.081218 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e8485b2-ec6f-4344-bee7-37495006a34c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e8485b2-ec6f-4344-bee7-37495006a34c-credential-socket\") pod \"6e8485b2-ec6f-4344-bee7-37495006a34c\" (UID: \"6e8485b2-ec6f-4344-bee7-37495006a34c\") " Nov 03 02:33:01.081381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:01.081253 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e8485b2-ec6f-4344-bee7-37495006a34c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e8485b2-ec6f-4344-bee7-37495006a34c-istiod-ca-cert\") pod \"6e8485b2-ec6f-4344-bee7-37495006a34c\" (UID: \"6e8485b2-ec6f-4344-bee7-37495006a34c\") " Nov 03 02:33:01.081381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:01.081298 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e8485b2-ec6f-4344-bee7-37495006a34c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e8485b2-ec6f-4344-bee7-37495006a34c-workload-certs\") pod \"6e8485b2-ec6f-4344-bee7-37495006a34c\" (UID: \"6e8485b2-ec6f-4344-bee7-37495006a34c\") " Nov 03 02:33:01.093381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:01.082982 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e8485b2-ec6f-4344-bee7-37495006a34c-workload-certs" pod "6e8485b2-ec6f-4344-bee7-37495006a34c" (UID: "6e8485b2-ec6f-4344-bee7-37495006a34c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:33:01.093381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:01.083320 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e8485b2-ec6f-4344-bee7-37495006a34c-workload-socket" pod "6e8485b2-ec6f-4344-bee7-37495006a34c" (UID: "6e8485b2-ec6f-4344-bee7-37495006a34c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:33:01.093381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:01.083613 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e8485b2-ec6f-4344-bee7-37495006a34c-istio-data" pod "6e8485b2-ec6f-4344-bee7-37495006a34c" (UID: "6e8485b2-ec6f-4344-bee7-37495006a34c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:33:01.097309 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:01.095700 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e8485b2-ec6f-4344-bee7-37495006a34c-credential-socket" pod "6e8485b2-ec6f-4344-bee7-37495006a34c" (UID: "6e8485b2-ec6f-4344-bee7-37495006a34c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:33:01.097309 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:01.096022 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e8485b2-ec6f-4344-bee7-37495006a34c-istiod-ca-cert" pod "6e8485b2-ec6f-4344-bee7-37495006a34c" (UID: "6e8485b2-ec6f-4344-bee7-37495006a34c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:33:01.103817 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:01.103781 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e8485b2-ec6f-4344-bee7-37495006a34c-istio-envoy" pod "6e8485b2-ec6f-4344-bee7-37495006a34c" (UID: "6e8485b2-ec6f-4344-bee7-37495006a34c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:33:01.103817 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:01.103800 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e8485b2-ec6f-4344-bee7-37495006a34c-istio-token" pod "6e8485b2-ec6f-4344-bee7-37495006a34c" (UID: "6e8485b2-ec6f-4344-bee7-37495006a34c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:33:01.103817 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:01.103788 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e8485b2-ec6f-4344-bee7-37495006a34c-kube-api-access-wkzx8" pod "6e8485b2-ec6f-4344-bee7-37495006a34c" (UID: "6e8485b2-ec6f-4344-bee7-37495006a34c"). InnerVolumeSpecName "kube-api-access-wkzx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:33:01.105403 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:01.105351 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6e8485b2-ec6f-4344-bee7-37495006a34c-istio-podinfo" pod "6e8485b2-ec6f-4344-bee7-37495006a34c" (UID: "6e8485b2-ec6f-4344-bee7-37495006a34c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:33:01.183466 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:01.183414 5605 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e8485b2-ec6f-4344-bee7-37495006a34c-workload-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:01.183466 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:01.183460 5605 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e8485b2-ec6f-4344-bee7-37495006a34c-istio-data\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:01.183726 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:01.183475 5605 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e8485b2-ec6f-4344-bee7-37495006a34c-istio-envoy\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:01.183726 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:01.183490 5605 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e8485b2-ec6f-4344-bee7-37495006a34c-credential-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:01.183726 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:01.183508 5605 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e8485b2-ec6f-4344-bee7-37495006a34c-istiod-ca-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:01.183726 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:01.183523 5605 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e8485b2-ec6f-4344-bee7-37495006a34c-workload-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:01.183726 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:01.183537 5605 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e8485b2-ec6f-4344-bee7-37495006a34c-istio-podinfo\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:01.183726 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:01.183550 5605 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e8485b2-ec6f-4344-bee7-37495006a34c-istio-token\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:01.183726 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:01.183565 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkzx8\" (UniqueName: \"kubernetes.io/projected/6e8485b2-ec6f-4344-bee7-37495006a34c-kube-api-access-wkzx8\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:01.198808 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:01.198768 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9" Nov 03 02:33:01.199943 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:01.199915 5605 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 03 02:33:01.200057 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:01.199961 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9" podUID="f8c546e7-4bc7-459f-8fa8-65e18c1585b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 03 02:33:01.489457 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:01.488480 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts" Nov 03 02:33:01.505383 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:01.500527 5605 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 03 02:33:01.505383 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:01.500610 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts" podUID="77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 03 02:33:01.713086 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:01.713054 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5" Nov 03 02:33:01.713527 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:01.713097 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5" event={"ID":"6e8485b2-ec6f-4344-bee7-37495006a34c","Type":"ContainerDied","Data":"eef375138aaff8b9c291d4b1d8f1e9880d2477511f7a8992417e7b9e1788eb38"} Nov 03 02:33:01.713527 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:01.713152 5605 scope.go:122] "RemoveContainer" containerID="4fb8e5c48c374a4d444dac5503aa9b95e1f55b06ae604b8d66138b8c2f355d38" Nov 03 02:33:01.916393 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:01.895428 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5"] Nov 03 02:33:01.916739 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:01.916596 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-711318-openshift-default-68459c5bf7-csck5"] Nov 03 02:33:02.203440 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:02.202803 5605 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 03 02:33:02.203440 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:02.202868 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9" podUID="f8c546e7-4bc7-459f-8fa8-65e18c1585b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 03 02:33:02.492446 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:02.491934 5605 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 03 02:33:02.492446 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:02.492006 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts" podUID="77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 03 02:33:02.850098 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:02.850056 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e8485b2-ec6f-4344-bee7-37495006a34c" path="/var/lib/kubelet/pods/6e8485b2-ec6f-4344-bee7-37495006a34c/volumes" Nov 03 02:33:03.199439 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:03.198763 5605 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 03 02:33:03.199439 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:03.198828 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9" podUID="f8c546e7-4bc7-459f-8fa8-65e18c1585b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 03 02:33:03.494442 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:03.487769 5605 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 03 02:33:03.494442 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:03.487832 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts" podUID="77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 03 02:33:04.202490 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:04.202451 5605 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 03 02:33:04.203094 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:04.203062 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9" podUID="f8c546e7-4bc7-459f-8fa8-65e18c1585b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 03 02:33:04.489836 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:04.489743 5605 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 03 02:33:04.490125 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:04.490094 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts" podUID="77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 03 02:33:04.923417 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:04.921041 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl"] Nov 03 02:33:04.923417 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:04.921733 5605 cpu_manager.go:414] "containerMap: removing container" podUID="6e8485b2-ec6f-4344-bee7-37495006a34c" containerName="istio-proxy" Nov 03 02:33:04.923417 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:04.921752 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e8485b2-ec6f-4344-bee7-37495006a34c" containerName="istio-proxy" Nov 03 02:33:04.923417 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:04.921773 5605 cpu_manager.go:414] "containerMap: removing container" podUID="6e8485b2-ec6f-4344-bee7-37495006a34c" containerName="istio-proxy" Nov 03 02:33:04.923417 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:04.921782 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e8485b2-ec6f-4344-bee7-37495006a34c" containerName="istio-proxy" Nov 03 02:33:04.923417 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:04.921828 5605 cpu_manager.go:414] "containerMap: removing container" podUID="d9269e01-6892-4797-815a-b46f93340eae" containerName="revoke-keys" Nov 03 02:33:04.923417 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:04.921837 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9269e01-6892-4797-815a-b46f93340eae" containerName="revoke-keys" Nov 03 02:33:04.923417 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:04.921992 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e8485b2-ec6f-4344-bee7-37495006a34c" containerName="istio-proxy" Nov 03 02:33:04.923417 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:04.922005 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9269e01-6892-4797-815a-b46f93340eae" containerName="revoke-keys" Nov 03 02:33:04.934382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:04.928993 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl" Nov 03 02:33:04.954570 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:04.954527 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/804673d5-10dd-463e-9c52-fc67bd1c307a-workload-certs\") pod \"e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl\" (UID: \"804673d5-10dd-463e-9c52-fc67bd1c307a\") " pod="openshift-ingress/e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl" Nov 03 02:33:04.954927 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:04.954904 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/804673d5-10dd-463e-9c52-fc67bd1c307a-credential-socket\") pod \"e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl\" (UID: \"804673d5-10dd-463e-9c52-fc67bd1c307a\") " pod="openshift-ingress/e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl" Nov 03 02:33:04.955144 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:04.955126 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tfnn\" (UniqueName: \"kubernetes.io/projected/804673d5-10dd-463e-9c52-fc67bd1c307a-kube-api-access-8tfnn\") pod \"e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl\" (UID: \"804673d5-10dd-463e-9c52-fc67bd1c307a\") " pod="openshift-ingress/e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl" Nov 03 02:33:04.955467 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:04.955447 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/804673d5-10dd-463e-9c52-fc67bd1c307a-workload-socket\") pod \"e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl\" (UID: \"804673d5-10dd-463e-9c52-fc67bd1c307a\") " pod="openshift-ingress/e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl" Nov 03 02:33:04.956860 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:04.956837 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/804673d5-10dd-463e-9c52-fc67bd1c307a-istio-token\") pod \"e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl\" (UID: \"804673d5-10dd-463e-9c52-fc67bd1c307a\") " pod="openshift-ingress/e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl" Nov 03 02:33:05.006608 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:05.006563 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/804673d5-10dd-463e-9c52-fc67bd1c307a-istio-podinfo\") pod \"e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl\" (UID: \"804673d5-10dd-463e-9c52-fc67bd1c307a\") " pod="openshift-ingress/e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl" Nov 03 02:33:05.007006 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:05.006981 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/804673d5-10dd-463e-9c52-fc67bd1c307a-istio-envoy\") pod \"e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl\" (UID: \"804673d5-10dd-463e-9c52-fc67bd1c307a\") " pod="openshift-ingress/e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl" Nov 03 02:33:05.007174 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:05.007156 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/804673d5-10dd-463e-9c52-fc67bd1c307a-istio-data\") pod \"e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl\" (UID: \"804673d5-10dd-463e-9c52-fc67bd1c307a\") " pod="openshift-ingress/e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl" Nov 03 02:33:05.007414 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:05.007394 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/804673d5-10dd-463e-9c52-fc67bd1c307a-istiod-ca-cert\") pod \"e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl\" (UID: \"804673d5-10dd-463e-9c52-fc67bd1c307a\") " pod="openshift-ingress/e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl" Nov 03 02:33:05.097412 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:05.072744 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-3a4683-openshift-default-dockercfg-hv4sq\"" Nov 03 02:33:05.113514 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:05.113476 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/804673d5-10dd-463e-9c52-fc67bd1c307a-istiod-ca-cert\") pod \"e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl\" (UID: \"804673d5-10dd-463e-9c52-fc67bd1c307a\") " pod="openshift-ingress/e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl" Nov 03 02:33:05.113813 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:05.113785 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/804673d5-10dd-463e-9c52-fc67bd1c307a-workload-certs\") pod \"e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl\" (UID: \"804673d5-10dd-463e-9c52-fc67bd1c307a\") " pod="openshift-ingress/e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl" Nov 03 02:33:05.113982 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:05.113966 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/804673d5-10dd-463e-9c52-fc67bd1c307a-credential-socket\") pod \"e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl\" (UID: \"804673d5-10dd-463e-9c52-fc67bd1c307a\") " pod="openshift-ingress/e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl" Nov 03 02:33:05.114167 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:05.114149 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tfnn\" (UniqueName: \"kubernetes.io/projected/804673d5-10dd-463e-9c52-fc67bd1c307a-kube-api-access-8tfnn\") pod \"e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl\" (UID: \"804673d5-10dd-463e-9c52-fc67bd1c307a\") " pod="openshift-ingress/e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl" Nov 03 02:33:05.114305 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:05.114291 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/804673d5-10dd-463e-9c52-fc67bd1c307a-workload-socket\") pod \"e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl\" (UID: \"804673d5-10dd-463e-9c52-fc67bd1c307a\") " pod="openshift-ingress/e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl" Nov 03 02:33:05.114444 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:05.114424 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/804673d5-10dd-463e-9c52-fc67bd1c307a-istio-token\") pod \"e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl\" (UID: \"804673d5-10dd-463e-9c52-fc67bd1c307a\") " pod="openshift-ingress/e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl" Nov 03 02:33:05.114584 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:05.114568 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/804673d5-10dd-463e-9c52-fc67bd1c307a-istio-podinfo\") pod \"e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl\" (UID: \"804673d5-10dd-463e-9c52-fc67bd1c307a\") " pod="openshift-ingress/e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl" Nov 03 02:33:05.114745 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:05.114730 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/804673d5-10dd-463e-9c52-fc67bd1c307a-istio-envoy\") pod \"e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl\" (UID: \"804673d5-10dd-463e-9c52-fc67bd1c307a\") " pod="openshift-ingress/e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl" Nov 03 02:33:05.114849 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:05.114834 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/804673d5-10dd-463e-9c52-fc67bd1c307a-istio-data\") pod \"e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl\" (UID: \"804673d5-10dd-463e-9c52-fc67bd1c307a\") " pod="openshift-ingress/e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl" Nov 03 02:33:05.126381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:05.115391 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/804673d5-10dd-463e-9c52-fc67bd1c307a-istio-data\") pod \"e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl\" (UID: \"804673d5-10dd-463e-9c52-fc67bd1c307a\") " pod="openshift-ingress/e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl" Nov 03 02:33:05.127398 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:05.127349 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/804673d5-10dd-463e-9c52-fc67bd1c307a-istiod-ca-cert\") pod \"e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl\" (UID: \"804673d5-10dd-463e-9c52-fc67bd1c307a\") " pod="openshift-ingress/e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl" Nov 03 02:33:05.128380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:05.127797 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/804673d5-10dd-463e-9c52-fc67bd1c307a-workload-certs\") pod \"e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl\" (UID: \"804673d5-10dd-463e-9c52-fc67bd1c307a\") " pod="openshift-ingress/e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl" Nov 03 02:33:05.128886 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:05.128861 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/804673d5-10dd-463e-9c52-fc67bd1c307a-workload-socket\") pod \"e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl\" (UID: \"804673d5-10dd-463e-9c52-fc67bd1c307a\") " pod="openshift-ingress/e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl" Nov 03 02:33:05.129113 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:05.129082 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/804673d5-10dd-463e-9c52-fc67bd1c307a-credential-socket\") pod \"e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl\" (UID: \"804673d5-10dd-463e-9c52-fc67bd1c307a\") " pod="openshift-ingress/e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl" Nov 03 02:33:05.172708 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:05.172672 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/804673d5-10dd-463e-9c52-fc67bd1c307a-istio-podinfo\") pod \"e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl\" (UID: \"804673d5-10dd-463e-9c52-fc67bd1c307a\") " pod="openshift-ingress/e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl" Nov 03 02:33:05.175783 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:05.175678 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/804673d5-10dd-463e-9c52-fc67bd1c307a-istio-envoy\") pod \"e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl\" (UID: \"804673d5-10dd-463e-9c52-fc67bd1c307a\") " pod="openshift-ingress/e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl" Nov 03 02:33:05.198877 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:05.198822 5605 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 03 02:33:05.199565 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:05.199119 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9" podUID="f8c546e7-4bc7-459f-8fa8-65e18c1585b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 03 02:33:05.231580 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:05.229568 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl"] Nov 03 02:33:05.303381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:05.298536 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/804673d5-10dd-463e-9c52-fc67bd1c307a-istio-token\") pod \"e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl\" (UID: \"804673d5-10dd-463e-9c52-fc67bd1c307a\") " pod="openshift-ingress/e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl" Nov 03 02:33:05.309386 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:05.307117 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tfnn\" (UniqueName: \"kubernetes.io/projected/804673d5-10dd-463e-9c52-fc67bd1c307a-kube-api-access-8tfnn\") pod \"e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl\" (UID: \"804673d5-10dd-463e-9c52-fc67bd1c307a\") " pod="openshift-ingress/e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl" Nov 03 02:33:05.325400 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:05.322204 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl" Nov 03 02:33:05.342822 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:05.342793 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:33:05.538423 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:05.518870 5605 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 03 02:33:05.538423 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:05.518931 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts" podUID="77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 03 02:33:05.837642 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:05.837583 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl"] Nov 03 02:33:05.867783 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:05.847776 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:33:05.867783 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:05.847871 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:33:05.867783 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:05.847913 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:33:06.206442 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:06.201770 5605 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 03 02:33:06.206442 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:06.201833 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9" podUID="f8c546e7-4bc7-459f-8fa8-65e18c1585b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 03 02:33:06.487445 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:06.487104 5605 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 03 02:33:06.487445 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:06.487163 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts" podUID="77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 03 02:33:06.788093 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:06.787996 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl" event={"ID":"804673d5-10dd-463e-9c52-fc67bd1c307a","Type":"ContainerStarted","Data":"ad496f17e0211a23c7841cc6aeee7430bf3e2835b24fdb916a66b3ee65b60328"} Nov 03 02:33:06.788346 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:06.788325 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl" event={"ID":"804673d5-10dd-463e-9c52-fc67bd1c307a","Type":"ContainerStarted","Data":"61fe1c49ace8bbbe052f70a96b82b5e7d0f8eb041065941b72741b7d0c39388a"} Nov 03 02:33:07.210383 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:07.199851 5605 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 03 02:33:07.210383 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:07.199918 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9" podUID="f8c546e7-4bc7-459f-8fa8-65e18c1585b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 03 02:33:07.337381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:07.336665 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl" Nov 03 02:33:07.366382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:07.360982 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl" Nov 03 02:33:07.486441 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:07.486328 5605 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 03 02:33:07.486441 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:07.486406 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts" podUID="77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 03 02:33:07.490380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:07.487962 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl" podStartSLOduration=3.487943901 podStartE2EDuration="3.487943901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:33:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:33:06.981621065 +0000 UTC m=+2464.240819212" watchObservedRunningTime="2026-11-03 02:33:07.487943901 +0000 UTC m=+2464.747142056" Nov 03 02:33:07.807766 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:07.807736 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl" Nov 03 02:33:07.816638 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:07.816607 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl" Nov 03 02:33:08.488808 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:08.488770 5605 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 03 02:33:08.489178 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:08.489146 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts" podUID="77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 03 02:33:08.694382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:08.693669 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-965a07-64c88dd8c-p8xmd"] Nov 03 02:33:08.699381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:08.695456 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e8485b2-ec6f-4344-bee7-37495006a34c" containerName="istio-proxy" Nov 03 02:33:08.711476 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:08.709705 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-965a07-64c88dd8c-p8xmd" Nov 03 02:33:08.771435 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:08.760343 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-965a07-dockercfg-kjxms\"" Nov 03 02:33:08.771435 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:08.768433 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-965a07-64c88dd8c-p8xmd"] Nov 03 02:33:08.771435 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:08.768578 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-965a07\"" Nov 03 02:33:08.822047 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:08.821994 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18215d27-a924-4c3b-8faa-47fbc0f7cb26-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-965a07-64c88dd8c-p8xmd\" (UID: \"18215d27-a924-4c3b-8faa-47fbc0f7cb26\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-965a07-64c88dd8c-p8xmd" Nov 03 02:33:08.822327 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:08.822305 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdjw4\" (UniqueName: \"kubernetes.io/projected/18215d27-a924-4c3b-8faa-47fbc0f7cb26-kube-api-access-qdjw4\") pod \"payload-processing-e2e-shared-a-w3-965a07-64c88dd8c-p8xmd\" (UID: \"18215d27-a924-4c3b-8faa-47fbc0f7cb26\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-965a07-64c88dd8c-p8xmd" Nov 03 02:33:08.849383 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:08.827693 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-521f1f/simulator-w2-521f1f-kserve-5f47b86f6-mgksf" podUID="6e1b7fda-0ade-4042-ad63-5115bc66b454" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Nov 03 02:33:08.931048 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:08.931003 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18215d27-a924-4c3b-8faa-47fbc0f7cb26-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-965a07-64c88dd8c-p8xmd\" (UID: \"18215d27-a924-4c3b-8faa-47fbc0f7cb26\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-965a07-64c88dd8c-p8xmd" Nov 03 02:33:08.942820 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:08.942788 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdjw4\" (UniqueName: \"kubernetes.io/projected/18215d27-a924-4c3b-8faa-47fbc0f7cb26-kube-api-access-qdjw4\") pod \"payload-processing-e2e-shared-a-w3-965a07-64c88dd8c-p8xmd\" (UID: \"18215d27-a924-4c3b-8faa-47fbc0f7cb26\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-965a07-64c88dd8c-p8xmd" Nov 03 02:33:08.980624 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:08.980582 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18215d27-a924-4c3b-8faa-47fbc0f7cb26-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-965a07-64c88dd8c-p8xmd\" (UID: \"18215d27-a924-4c3b-8faa-47fbc0f7cb26\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-965a07-64c88dd8c-p8xmd" Nov 03 02:33:09.035255 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:09.035173 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdjw4\" (UniqueName: \"kubernetes.io/projected/18215d27-a924-4c3b-8faa-47fbc0f7cb26-kube-api-access-qdjw4\") pod \"payload-processing-e2e-shared-a-w3-965a07-64c88dd8c-p8xmd\" (UID: \"18215d27-a924-4c3b-8faa-47fbc0f7cb26\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-965a07-64c88dd8c-p8xmd" Nov 03 02:33:09.297927 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:09.297818 5605 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:33:09.297927 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:09.297903 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9" podUID="f8c546e7-4bc7-459f-8fa8-65e18c1585b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:33:09.323774 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:09.323738 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9" Nov 03 02:33:09.324217 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:09.324160 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9" Nov 03 02:33:09.324974 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:09.324953 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9" Nov 03 02:33:09.332853 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:09.332824 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-965a07-64c88dd8c-p8xmd" Nov 03 02:33:09.737710 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:09.737673 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-965a07-64c88dd8c-p8xmd"] Nov 03 02:33:09.897521 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:09.897484 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-965a07-64c88dd8c-p8xmd" event={"ID":"18215d27-a924-4c3b-8faa-47fbc0f7cb26","Type":"ContainerStarted","Data":"7a97e6001fb1b40a6f87314f4ea01ffa0326ffabe457d9d3a2d9216fd5be096f"} Nov 03 02:33:09.897700 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:09.897556 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-965a07-64c88dd8c-p8xmd" event={"ID":"18215d27-a924-4c3b-8faa-47fbc0f7cb26","Type":"ContainerStarted","Data":"141b652b50032cbef45451f5311534a14bc9f35ef0e8299962351c45f7ee8c8c"} Nov 03 02:33:09.898228 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:09.898206 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-965a07-64c88dd8c-p8xmd" Nov 03 02:33:10.017632 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:10.017512 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-965a07-64c88dd8c-p8xmd" podStartSLOduration=2.017490652 podStartE2EDuration="2.017490652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:33:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:33:10.00671738 +0000 UTC m=+2467.265915534" watchObservedRunningTime="2026-11-03 02:33:10.017490652 +0000 UTC m=+2467.276688808" Nov 03 02:33:10.527563 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:10.527511 5605 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:33:10.527803 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:10.527582 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts" podUID="77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:33:10.975576 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:10.975544 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts" Nov 03 02:33:10.976010 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:10.975991 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts" Nov 03 02:33:10.989646 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:10.989620 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts" Nov 03 02:33:12.456854 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:12.456821 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a192b197-72s2r"] Nov 03 02:33:12.460917 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:12.460897 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a192b197-72s2r" Nov 03 02:33:12.510320 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:12.510291 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:33:12.529789 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:12.529767 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bb3aeb2-84e3-4880-9baa-36abe371f261-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a192b197-72s2r\" (UID: \"0bb3aeb2-84e3-4880-9baa-36abe371f261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a192b197-72s2r" Nov 03 02:33:12.529880 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:12.529804 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m84f9\" (UniqueName: \"kubernetes.io/projected/0bb3aeb2-84e3-4880-9baa-36abe371f261-kube-api-access-m84f9\") pod \"maas-api-revoke-sub-a192b197-72s2r\" (UID: \"0bb3aeb2-84e3-4880-9baa-36abe371f261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a192b197-72s2r" Nov 03 02:33:12.529880 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:12.529834 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bb3aeb2-84e3-4880-9baa-36abe371f261-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a192b197-72s2r\" (UID: \"0bb3aeb2-84e3-4880-9baa-36abe371f261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a192b197-72s2r" Nov 03 02:33:12.557146 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:12.557125 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:33:12.561160 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:12.561135 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a192b197-72s2r"] Nov 03 02:33:12.603635 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:12.603612 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:33:12.603779 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:12.603764 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:33:12.631555 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:12.631530 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bb3aeb2-84e3-4880-9baa-36abe371f261-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a192b197-72s2r\" (UID: \"0bb3aeb2-84e3-4880-9baa-36abe371f261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a192b197-72s2r" Nov 03 02:33:12.631639 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:12.631569 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m84f9\" (UniqueName: \"kubernetes.io/projected/0bb3aeb2-84e3-4880-9baa-36abe371f261-kube-api-access-m84f9\") pod \"maas-api-revoke-sub-a192b197-72s2r\" (UID: \"0bb3aeb2-84e3-4880-9baa-36abe371f261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a192b197-72s2r" Nov 03 02:33:12.631721 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:12.631700 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bb3aeb2-84e3-4880-9baa-36abe371f261-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a192b197-72s2r\" (UID: \"0bb3aeb2-84e3-4880-9baa-36abe371f261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a192b197-72s2r" Nov 03 02:33:12.632287 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:12.632270 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bb3aeb2-84e3-4880-9baa-36abe371f261-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a192b197-72s2r\" (UID: \"0bb3aeb2-84e3-4880-9baa-36abe371f261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a192b197-72s2r" Nov 03 02:33:12.633774 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:12.633756 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bb3aeb2-84e3-4880-9baa-36abe371f261-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a192b197-72s2r\" (UID: \"0bb3aeb2-84e3-4880-9baa-36abe371f261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a192b197-72s2r" Nov 03 02:33:12.684446 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:12.684419 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m84f9\" (UniqueName: \"kubernetes.io/projected/0bb3aeb2-84e3-4880-9baa-36abe371f261-kube-api-access-m84f9\") pod \"maas-api-revoke-sub-a192b197-72s2r\" (UID: \"0bb3aeb2-84e3-4880-9baa-36abe371f261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a192b197-72s2r" Nov 03 02:33:12.774176 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:12.774119 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a192b197-72s2r" Nov 03 02:33:13.145739 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:13.145709 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a192b197-72s2r"] Nov 03 02:33:13.150630 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:33:13.150602 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bb3aeb2_84e3_4880_9baa_36abe371f261.slice/crio-e64842e406c3cfb33985d1c06794f64bcc03093565cfffc7cc0b2fa5d4c353cf WatchSource:0}: Error finding container e64842e406c3cfb33985d1c06794f64bcc03093565cfffc7cc0b2fa5d4c353cf: Status 404 returned error can't find the container with id e64842e406c3cfb33985d1c06794f64bcc03093565cfffc7cc0b2fa5d4c353cf Nov 03 02:33:13.935374 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:13.935318 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a192b197-72s2r" event={"ID":"0bb3aeb2-84e3-4880-9baa-36abe371f261","Type":"ContainerStarted","Data":"e4d32e09eea7895e4cc3bd8165d50e307123480badc04c606055be17268c5825"} Nov 03 02:33:13.935807 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:13.935423 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a192b197-72s2r" event={"ID":"0bb3aeb2-84e3-4880-9baa-36abe371f261","Type":"ContainerStarted","Data":"e64842e406c3cfb33985d1c06794f64bcc03093565cfffc7cc0b2fa5d4c353cf"} Nov 03 02:33:14.050034 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:14.049987 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a192b197-72s2r" podStartSLOduration=2.049974583 podStartE2EDuration="2.049974583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:33:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:33:14.04692168 +0000 UTC m=+2471.306119836" watchObservedRunningTime="2026-11-03 02:33:14.049974583 +0000 UTC m=+2471.309172730" Nov 03 02:33:14.943740 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:14.943704 5605 generic.go:356] "Generic (PLEG): container finished" podID="0bb3aeb2-84e3-4880-9baa-36abe371f261" containerID="e4d32e09eea7895e4cc3bd8165d50e307123480badc04c606055be17268c5825" exitCode=0 Nov 03 02:33:14.944167 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:14.943787 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a192b197-72s2r" event={"ID":"0bb3aeb2-84e3-4880-9baa-36abe371f261","Type":"ContainerDied","Data":"e4d32e09eea7895e4cc3bd8165d50e307123480badc04c606055be17268c5825"} Nov 03 02:33:16.095500 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:16.095475 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a192b197-72s2r" Nov 03 02:33:16.175865 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:16.175837 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bb3aeb2-84e3-4880-9baa-36abe371f261-kube-api-access-m84f9\" (UniqueName: \"kubernetes.io/projected/0bb3aeb2-84e3-4880-9baa-36abe371f261-kube-api-access-m84f9\") pod \"0bb3aeb2-84e3-4880-9baa-36abe371f261\" (UID: \"0bb3aeb2-84e3-4880-9baa-36abe371f261\") " Nov 03 02:33:16.176052 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:16.175889 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bb3aeb2-84e3-4880-9baa-36abe371f261-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bb3aeb2-84e3-4880-9baa-36abe371f261-maas-api-openshift-ca\") pod \"0bb3aeb2-84e3-4880-9baa-36abe371f261\" (UID: \"0bb3aeb2-84e3-4880-9baa-36abe371f261\") " Nov 03 02:33:16.176119 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:16.176052 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0bb3aeb2-84e3-4880-9baa-36abe371f261-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bb3aeb2-84e3-4880-9baa-36abe371f261-maas-api-xks-ca\") pod \"0bb3aeb2-84e3-4880-9baa-36abe371f261\" (UID: \"0bb3aeb2-84e3-4880-9baa-36abe371f261\") " Nov 03 02:33:16.176300 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:16.176271 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bb3aeb2-84e3-4880-9baa-36abe371f261-maas-api-openshift-ca" pod "0bb3aeb2-84e3-4880-9baa-36abe371f261" (UID: "0bb3aeb2-84e3-4880-9baa-36abe371f261"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:33:16.176549 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:16.176524 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bb3aeb2-84e3-4880-9baa-36abe371f261-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:16.177929 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:16.177906 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bb3aeb2-84e3-4880-9baa-36abe371f261-kube-api-access-m84f9" pod "0bb3aeb2-84e3-4880-9baa-36abe371f261" (UID: "0bb3aeb2-84e3-4880-9baa-36abe371f261"). InnerVolumeSpecName "kube-api-access-m84f9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:33:16.178177 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:16.178153 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bb3aeb2-84e3-4880-9baa-36abe371f261-maas-api-xks-ca" pod "0bb3aeb2-84e3-4880-9baa-36abe371f261" (UID: "0bb3aeb2-84e3-4880-9baa-36abe371f261"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:33:16.277963 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:16.277878 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m84f9\" (UniqueName: \"kubernetes.io/projected/0bb3aeb2-84e3-4880-9baa-36abe371f261-kube-api-access-m84f9\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:16.277963 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:16.277914 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bb3aeb2-84e3-4880-9baa-36abe371f261-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:16.958318 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:16.958282 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a192b197-72s2r" Nov 03 02:33:16.958318 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:16.958303 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a192b197-72s2r" event={"ID":"0bb3aeb2-84e3-4880-9baa-36abe371f261","Type":"ContainerDied","Data":"e64842e406c3cfb33985d1c06794f64bcc03093565cfffc7cc0b2fa5d4c353cf"} Nov 03 02:33:16.958529 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:16.958351 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e64842e406c3cfb33985d1c06794f64bcc03093565cfffc7cc0b2fa5d4c353cf" Nov 03 02:33:20.909536 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:20.909505 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-965a07-64c88dd8c-p8xmd" Nov 03 02:33:23.929667 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:33:23.929628 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:33:27.093089 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:27.093052 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl"] Nov 03 02:33:27.093628 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:27.093608 5605 cpu_manager.go:414] "containerMap: removing container" podUID="0bb3aeb2-84e3-4880-9baa-36abe371f261" containerName="revoke-keys" Nov 03 02:33:27.093707 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:27.093630 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bb3aeb2-84e3-4880-9baa-36abe371f261" containerName="revoke-keys" Nov 03 02:33:27.093771 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:27.093755 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bb3aeb2-84e3-4880-9baa-36abe371f261" containerName="revoke-keys" Nov 03 02:33:27.097084 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:27.097060 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl" Nov 03 02:33:27.155537 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:27.155507 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl"] Nov 03 02:33:27.214609 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:27.214567 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a15ffc44-4fe7-41cf-9370-794bebfda952-credential-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl\" (UID: \"a15ffc44-4fe7-41cf-9370-794bebfda952\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl" Nov 03 02:33:27.214785 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:27.214726 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a15ffc44-4fe7-41cf-9370-794bebfda952-istio-data\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl\" (UID: \"a15ffc44-4fe7-41cf-9370-794bebfda952\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl" Nov 03 02:33:27.214875 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:27.214844 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fr4z\" (UniqueName: \"kubernetes.io/projected/a15ffc44-4fe7-41cf-9370-794bebfda952-kube-api-access-7fr4z\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl\" (UID: \"a15ffc44-4fe7-41cf-9370-794bebfda952\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl" Nov 03 02:33:27.214937 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:27.214887 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a15ffc44-4fe7-41cf-9370-794bebfda952-istio-podinfo\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl\" (UID: \"a15ffc44-4fe7-41cf-9370-794bebfda952\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl" Nov 03 02:33:27.214937 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:27.214914 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a15ffc44-4fe7-41cf-9370-794bebfda952-workload-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl\" (UID: \"a15ffc44-4fe7-41cf-9370-794bebfda952\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl" Nov 03 02:33:27.215050 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:27.215028 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a15ffc44-4fe7-41cf-9370-794bebfda952-istio-envoy\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl\" (UID: \"a15ffc44-4fe7-41cf-9370-794bebfda952\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl" Nov 03 02:33:27.215112 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:27.215068 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a15ffc44-4fe7-41cf-9370-794bebfda952-workload-certs\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl\" (UID: \"a15ffc44-4fe7-41cf-9370-794bebfda952\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl" Nov 03 02:33:27.215178 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:27.215127 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a15ffc44-4fe7-41cf-9370-794bebfda952-istiod-ca-cert\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl\" (UID: \"a15ffc44-4fe7-41cf-9370-794bebfda952\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl" Nov 03 02:33:27.215235 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:27.215187 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a15ffc44-4fe7-41cf-9370-794bebfda952-istio-token\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl\" (UID: \"a15ffc44-4fe7-41cf-9370-794bebfda952\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl" Nov 03 02:33:27.315906 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:27.315873 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a15ffc44-4fe7-41cf-9370-794bebfda952-credential-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl\" (UID: \"a15ffc44-4fe7-41cf-9370-794bebfda952\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl" Nov 03 02:33:27.316156 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:27.315979 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a15ffc44-4fe7-41cf-9370-794bebfda952-istio-data\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl\" (UID: \"a15ffc44-4fe7-41cf-9370-794bebfda952\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl" Nov 03 02:33:27.316156 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:27.316048 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fr4z\" (UniqueName: \"kubernetes.io/projected/a15ffc44-4fe7-41cf-9370-794bebfda952-kube-api-access-7fr4z\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl\" (UID: \"a15ffc44-4fe7-41cf-9370-794bebfda952\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl" Nov 03 02:33:27.316156 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:27.316086 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a15ffc44-4fe7-41cf-9370-794bebfda952-istio-podinfo\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl\" (UID: \"a15ffc44-4fe7-41cf-9370-794bebfda952\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl" Nov 03 02:33:27.316156 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:27.316112 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a15ffc44-4fe7-41cf-9370-794bebfda952-workload-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl\" (UID: \"a15ffc44-4fe7-41cf-9370-794bebfda952\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl" Nov 03 02:33:27.316156 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:27.316152 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a15ffc44-4fe7-41cf-9370-794bebfda952-istio-envoy\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl\" (UID: \"a15ffc44-4fe7-41cf-9370-794bebfda952\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl" Nov 03 02:33:27.316512 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:27.316313 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a15ffc44-4fe7-41cf-9370-794bebfda952-istio-data\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl\" (UID: \"a15ffc44-4fe7-41cf-9370-794bebfda952\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl" Nov 03 02:33:27.316512 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:27.316293 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a15ffc44-4fe7-41cf-9370-794bebfda952-credential-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl\" (UID: \"a15ffc44-4fe7-41cf-9370-794bebfda952\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl" Nov 03 02:33:27.316512 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:27.316343 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a15ffc44-4fe7-41cf-9370-794bebfda952-workload-certs\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl\" (UID: \"a15ffc44-4fe7-41cf-9370-794bebfda952\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl" Nov 03 02:33:27.316512 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:27.316439 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a15ffc44-4fe7-41cf-9370-794bebfda952-istiod-ca-cert\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl\" (UID: \"a15ffc44-4fe7-41cf-9370-794bebfda952\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl" Nov 03 02:33:27.316512 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:27.316506 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a15ffc44-4fe7-41cf-9370-794bebfda952-istio-token\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl\" (UID: \"a15ffc44-4fe7-41cf-9370-794bebfda952\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl" Nov 03 02:33:27.316776 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:27.316646 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a15ffc44-4fe7-41cf-9370-794bebfda952-workload-certs\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl\" (UID: \"a15ffc44-4fe7-41cf-9370-794bebfda952\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl" Nov 03 02:33:27.316871 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:27.316848 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a15ffc44-4fe7-41cf-9370-794bebfda952-workload-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl\" (UID: \"a15ffc44-4fe7-41cf-9370-794bebfda952\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl" Nov 03 02:33:27.317183 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:27.317157 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a15ffc44-4fe7-41cf-9370-794bebfda952-istiod-ca-cert\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl\" (UID: \"a15ffc44-4fe7-41cf-9370-794bebfda952\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl" Nov 03 02:33:27.318577 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:27.318549 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a15ffc44-4fe7-41cf-9370-794bebfda952-istio-envoy\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl\" (UID: \"a15ffc44-4fe7-41cf-9370-794bebfda952\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl" Nov 03 02:33:27.318733 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:27.318715 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a15ffc44-4fe7-41cf-9370-794bebfda952-istio-podinfo\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl\" (UID: \"a15ffc44-4fe7-41cf-9370-794bebfda952\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl" Nov 03 02:33:27.369296 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:27.369238 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a15ffc44-4fe7-41cf-9370-794bebfda952-istio-token\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl\" (UID: \"a15ffc44-4fe7-41cf-9370-794bebfda952\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl" Nov 03 02:33:27.369423 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:27.369349 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fr4z\" (UniqueName: \"kubernetes.io/projected/a15ffc44-4fe7-41cf-9370-794bebfda952-kube-api-access-7fr4z\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl\" (UID: \"a15ffc44-4fe7-41cf-9370-794bebfda952\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl" Nov 03 02:33:27.415929 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:27.415903 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl" Nov 03 02:33:27.839481 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:27.839453 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl"] Nov 03 02:33:27.881228 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:33:27.881192 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda15ffc44_4fe7_41cf_9370_794bebfda952.slice/crio-21956dccaa320af80c052e329052f315339d43e46e6c0bd07265a1f9906be40f WatchSource:0}: Error finding container 21956dccaa320af80c052e329052f315339d43e46e6c0bd07265a1f9906be40f: Status 404 returned error can't find the container with id 21956dccaa320af80c052e329052f315339d43e46e6c0bd07265a1f9906be40f Nov 03 02:33:27.883613 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:27.883580 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:33:27.883688 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:27.883642 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:33:27.883688 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:27.883673 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:33:28.043148 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:28.043112 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl" event={"ID":"a15ffc44-4fe7-41cf-9370-794bebfda952","Type":"ContainerStarted","Data":"c67ce55364d0867af67df425332dc6fc30f1d7b73b673760b0ccfd468ffdd699"} Nov 03 02:33:28.043347 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:28.043168 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl" event={"ID":"a15ffc44-4fe7-41cf-9370-794bebfda952","Type":"ContainerStarted","Data":"21956dccaa320af80c052e329052f315339d43e46e6c0bd07265a1f9906be40f"} Nov 03 02:33:28.155612 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:28.155513 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl" podStartSLOduration=1.155500425 podStartE2EDuration="1.155500425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:33:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:33:28.152688518 +0000 UTC m=+2485.411886673" watchObservedRunningTime="2026-11-03 02:33:28.155500425 +0000 UTC m=+2485.414698578" Nov 03 02:33:28.416651 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:28.416553 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl" Nov 03 02:33:28.418243 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:28.418212 5605 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 03 02:33:28.418435 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:28.418261 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl" podUID="a15ffc44-4fe7-41cf-9370-794bebfda952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 03 02:33:29.416648 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:29.416612 5605 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 03 02:33:29.417066 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:29.416664 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl" podUID="a15ffc44-4fe7-41cf-9370-794bebfda952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 03 02:33:30.416762 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:30.416722 5605 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 03 02:33:30.417424 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:30.417374 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl" podUID="a15ffc44-4fe7-41cf-9370-794bebfda952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 03 02:33:31.417202 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:31.417159 5605 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 03 02:33:31.417735 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:31.417222 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl" podUID="a15ffc44-4fe7-41cf-9370-794bebfda952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 03 02:33:32.416411 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:32.416370 5605 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 03 02:33:32.416585 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:32.416439 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl" podUID="a15ffc44-4fe7-41cf-9370-794bebfda952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 03 02:33:33.204194 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:33.204157 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9aebe8-66bd979655-9t5kx"] Nov 03 02:33:33.209194 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:33.209171 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9aebe8-66bd979655-9t5kx" Nov 03 02:33:33.286203 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:33.286172 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9aebe8-66bd979655-9t5kx"] Nov 03 02:33:33.310569 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:33.310538 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8sm5\" (UniqueName: \"kubernetes.io/projected/00fd0034-3867-4d9e-8f8c-4dd3c0d6fb07-kube-api-access-s8sm5\") pod \"payload-pre-processing-e2e-shared-a-w0-9aebe8-66bd979655-9t5kx\" (UID: \"00fd0034-3867-4d9e-8f8c-4dd3c0d6fb07\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9aebe8-66bd979655-9t5kx" Nov 03 02:33:33.310766 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:33.310592 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00fd0034-3867-4d9e-8f8c-4dd3c0d6fb07-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-9aebe8-66bd979655-9t5kx\" (UID: \"00fd0034-3867-4d9e-8f8c-4dd3c0d6fb07\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9aebe8-66bd979655-9t5kx" Nov 03 02:33:33.334393 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:33.333633 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-9aebe8\"" Nov 03 02:33:33.334393 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:33.333784 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-9aebe8-dockercfg-xthlj\"" Nov 03 02:33:33.410203 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:33.410164 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-9aebe8-9cb69fc69-plhsd"] Nov 03 02:33:33.412209 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:33.412174 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8sm5\" (UniqueName: \"kubernetes.io/projected/00fd0034-3867-4d9e-8f8c-4dd3c0d6fb07-kube-api-access-s8sm5\") pod \"payload-pre-processing-e2e-shared-a-w0-9aebe8-66bd979655-9t5kx\" (UID: \"00fd0034-3867-4d9e-8f8c-4dd3c0d6fb07\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9aebe8-66bd979655-9t5kx" Nov 03 02:33:33.412532 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:33.412511 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00fd0034-3867-4d9e-8f8c-4dd3c0d6fb07-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-9aebe8-66bd979655-9t5kx\" (UID: \"00fd0034-3867-4d9e-8f8c-4dd3c0d6fb07\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9aebe8-66bd979655-9t5kx" Nov 03 02:33:33.413788 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:33.413763 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00fd0034-3867-4d9e-8f8c-4dd3c0d6fb07-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-9aebe8-66bd979655-9t5kx\" (UID: \"00fd0034-3867-4d9e-8f8c-4dd3c0d6fb07\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9aebe8-66bd979655-9t5kx" Nov 03 02:33:33.416021 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:33.415998 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-9aebe8-9cb69fc69-plhsd" Nov 03 02:33:33.418765 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:33.418743 5605 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 03 02:33:33.418941 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:33.418904 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl" podUID="a15ffc44-4fe7-41cf-9370-794bebfda952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 03 02:33:33.442920 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:33.442896 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-9aebe8-9cb69fc69-plhsd"] Nov 03 02:33:33.494601 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:33.494526 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8sm5\" (UniqueName: \"kubernetes.io/projected/00fd0034-3867-4d9e-8f8c-4dd3c0d6fb07-kube-api-access-s8sm5\") pod \"payload-pre-processing-e2e-shared-a-w0-9aebe8-66bd979655-9t5kx\" (UID: \"00fd0034-3867-4d9e-8f8c-4dd3c0d6fb07\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9aebe8-66bd979655-9t5kx" Nov 03 02:33:33.513879 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:33.513850 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f4d6e34-c9a7-491e-a4d7-6bf5d24b28fc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-9aebe8-9cb69fc69-plhsd\" (UID: \"4f4d6e34-c9a7-491e-a4d7-6bf5d24b28fc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-9aebe8-9cb69fc69-plhsd" Nov 03 02:33:33.514003 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:33.513897 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdrdf\" (UniqueName: \"kubernetes.io/projected/4f4d6e34-c9a7-491e-a4d7-6bf5d24b28fc-kube-api-access-vdrdf\") pod \"payload-processing-e2e-shared-a-w0-9aebe8-9cb69fc69-plhsd\" (UID: \"4f4d6e34-c9a7-491e-a4d7-6bf5d24b28fc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-9aebe8-9cb69fc69-plhsd" Nov 03 02:33:33.522873 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:33.522852 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9aebe8-66bd979655-9t5kx" Nov 03 02:33:33.615428 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:33.615389 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f4d6e34-c9a7-491e-a4d7-6bf5d24b28fc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-9aebe8-9cb69fc69-plhsd\" (UID: \"4f4d6e34-c9a7-491e-a4d7-6bf5d24b28fc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-9aebe8-9cb69fc69-plhsd" Nov 03 02:33:33.615563 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:33.615464 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdrdf\" (UniqueName: \"kubernetes.io/projected/4f4d6e34-c9a7-491e-a4d7-6bf5d24b28fc-kube-api-access-vdrdf\") pod \"payload-processing-e2e-shared-a-w0-9aebe8-9cb69fc69-plhsd\" (UID: \"4f4d6e34-c9a7-491e-a4d7-6bf5d24b28fc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-9aebe8-9cb69fc69-plhsd" Nov 03 02:33:33.616703 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:33.616661 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f4d6e34-c9a7-491e-a4d7-6bf5d24b28fc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-9aebe8-9cb69fc69-plhsd\" (UID: \"4f4d6e34-c9a7-491e-a4d7-6bf5d24b28fc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-9aebe8-9cb69fc69-plhsd" Nov 03 02:33:33.682059 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:33.682018 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdrdf\" (UniqueName: \"kubernetes.io/projected/4f4d6e34-c9a7-491e-a4d7-6bf5d24b28fc-kube-api-access-vdrdf\") pod \"payload-processing-e2e-shared-a-w0-9aebe8-9cb69fc69-plhsd\" (UID: \"4f4d6e34-c9a7-491e-a4d7-6bf5d24b28fc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-9aebe8-9cb69fc69-plhsd" Nov 03 02:33:33.731932 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:33.731834 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-9aebe8-9cb69fc69-plhsd" Nov 03 02:33:33.913965 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:33.913930 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9aebe8-66bd979655-9t5kx"] Nov 03 02:33:33.919398 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:33:33.919368 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00fd0034_3867_4d9e_8f8c_4dd3c0d6fb07.slice/crio-35f1054372d1fc1a52762f492d495705fddc9575c9e89c2cdbc9fa34e0315645 WatchSource:0}: Error finding container 35f1054372d1fc1a52762f492d495705fddc9575c9e89c2cdbc9fa34e0315645: Status 404 returned error can't find the container with id 35f1054372d1fc1a52762f492d495705fddc9575c9e89c2cdbc9fa34e0315645 Nov 03 02:33:34.129715 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:34.129683 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-9aebe8-9cb69fc69-plhsd"] Nov 03 02:33:34.134422 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:34.134394 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9aebe8-66bd979655-9t5kx" event={"ID":"00fd0034-3867-4d9e-8f8c-4dd3c0d6fb07","Type":"ContainerStarted","Data":"1df7ee1a916b995d175031814aaa01872c1e3ca8cbc2dc9a37dcb19c586e4f58"} Nov 03 02:33:34.134569 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:34.134439 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9aebe8-66bd979655-9t5kx" event={"ID":"00fd0034-3867-4d9e-8f8c-4dd3c0d6fb07","Type":"ContainerStarted","Data":"35f1054372d1fc1a52762f492d495705fddc9575c9e89c2cdbc9fa34e0315645"} Nov 03 02:33:34.134635 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:34.134610 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9aebe8-66bd979655-9t5kx" Nov 03 02:33:34.139444 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:33:34.139421 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f4d6e34_c9a7_491e_a4d7_6bf5d24b28fc.slice/crio-313d3d7a27549277d5f951aefb3f63cc965767fac194dc32b03192da78488ea1 WatchSource:0}: Error finding container 313d3d7a27549277d5f951aefb3f63cc965767fac194dc32b03192da78488ea1: Status 404 returned error can't find the container with id 313d3d7a27549277d5f951aefb3f63cc965767fac194dc32b03192da78488ea1 Nov 03 02:33:34.245621 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:34.245569 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9aebe8-66bd979655-9t5kx" podStartSLOduration=1.245552246 podStartE2EDuration="1.245552246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:33:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:33:34.242739198 +0000 UTC m=+2491.501937356" watchObservedRunningTime="2026-11-03 02:33:34.245552246 +0000 UTC m=+2491.504750403" Nov 03 02:33:34.416735 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:34.416646 5605 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 03 02:33:34.416735 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:34.416702 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl" podUID="a15ffc44-4fe7-41cf-9370-794bebfda952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 03 02:33:35.142033 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:35.141997 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-9aebe8-9cb69fc69-plhsd" event={"ID":"4f4d6e34-c9a7-491e-a4d7-6bf5d24b28fc","Type":"ContainerStarted","Data":"228a5bc5ec03eba955e0bc916a4d0ddc44bebca1cf5c6d4917a5851f557a16c7"} Nov 03 02:33:35.142220 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:35.142054 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-9aebe8-9cb69fc69-plhsd" event={"ID":"4f4d6e34-c9a7-491e-a4d7-6bf5d24b28fc","Type":"ContainerStarted","Data":"313d3d7a27549277d5f951aefb3f63cc965767fac194dc32b03192da78488ea1"} Nov 03 02:33:35.142455 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:35.142426 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-9aebe8-9cb69fc69-plhsd" Nov 03 02:33:35.249533 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:35.249474 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-9aebe8-9cb69fc69-plhsd" podStartSLOduration=2.249460632 podStartE2EDuration="2.249460632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:33:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:33:35.248100439 +0000 UTC m=+2492.507298594" watchObservedRunningTime="2026-11-03 02:33:35.249460632 +0000 UTC m=+2492.508658786" Nov 03 02:33:35.638792 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:35.637700 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kcdqm"] Nov 03 02:33:35.642963 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:35.642936 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kcdqm" Nov 03 02:33:35.700395 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:35.699596 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 02:33:35.700395 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:35.699670 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 02:33:35.700395 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:35.699717 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 02:33:35.700395 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:35.699811 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-2k54n\"" Nov 03 02:33:35.701221 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:35.701192 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kcdqm"] Nov 03 02:33:35.739267 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:35.739236 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2ad21c46-2e06-4b88-a787-0ca89fde517d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kcdqm\" (UID: \"2ad21c46-2e06-4b88-a787-0ca89fde517d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kcdqm" Nov 03 02:33:35.739487 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:35.739299 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ad21c46-2e06-4b88-a787-0ca89fde517d-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kcdqm\" (UID: \"2ad21c46-2e06-4b88-a787-0ca89fde517d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kcdqm" Nov 03 02:33:35.739487 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:35.739344 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2ad21c46-2e06-4b88-a787-0ca89fde517d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kcdqm\" (UID: \"2ad21c46-2e06-4b88-a787-0ca89fde517d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kcdqm" Nov 03 02:33:35.739487 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:35.739482 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr4qg\" (UniqueName: \"kubernetes.io/projected/2ad21c46-2e06-4b88-a787-0ca89fde517d-kube-api-access-mr4qg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kcdqm\" (UID: \"2ad21c46-2e06-4b88-a787-0ca89fde517d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kcdqm" Nov 03 02:33:35.739681 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:35.739508 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ad21c46-2e06-4b88-a787-0ca89fde517d-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kcdqm\" (UID: \"2ad21c46-2e06-4b88-a787-0ca89fde517d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kcdqm" Nov 03 02:33:35.739681 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:35.739524 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2ad21c46-2e06-4b88-a787-0ca89fde517d-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kcdqm\" (UID: \"2ad21c46-2e06-4b88-a787-0ca89fde517d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kcdqm" Nov 03 02:33:35.739681 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:35.739592 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2ad21c46-2e06-4b88-a787-0ca89fde517d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kcdqm\" (UID: \"2ad21c46-2e06-4b88-a787-0ca89fde517d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kcdqm" Nov 03 02:33:35.841421 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:35.841385 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2ad21c46-2e06-4b88-a787-0ca89fde517d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kcdqm\" (UID: \"2ad21c46-2e06-4b88-a787-0ca89fde517d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kcdqm" Nov 03 02:33:35.841616 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:35.841437 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2ad21c46-2e06-4b88-a787-0ca89fde517d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kcdqm\" (UID: \"2ad21c46-2e06-4b88-a787-0ca89fde517d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kcdqm" Nov 03 02:33:35.841616 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:35.841514 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ad21c46-2e06-4b88-a787-0ca89fde517d-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kcdqm\" (UID: \"2ad21c46-2e06-4b88-a787-0ca89fde517d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kcdqm" Nov 03 02:33:35.841616 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:35.841581 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2ad21c46-2e06-4b88-a787-0ca89fde517d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kcdqm\" (UID: \"2ad21c46-2e06-4b88-a787-0ca89fde517d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kcdqm" Nov 03 02:33:35.841814 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:35.841674 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr4qg\" (UniqueName: \"kubernetes.io/projected/2ad21c46-2e06-4b88-a787-0ca89fde517d-kube-api-access-mr4qg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kcdqm\" (UID: \"2ad21c46-2e06-4b88-a787-0ca89fde517d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kcdqm" Nov 03 02:33:35.841814 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:35.841704 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ad21c46-2e06-4b88-a787-0ca89fde517d-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kcdqm\" (UID: \"2ad21c46-2e06-4b88-a787-0ca89fde517d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kcdqm" Nov 03 02:33:35.841814 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:35.841730 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2ad21c46-2e06-4b88-a787-0ca89fde517d-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kcdqm\" (UID: \"2ad21c46-2e06-4b88-a787-0ca89fde517d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kcdqm" Nov 03 02:33:35.843037 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:35.843005 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2ad21c46-2e06-4b88-a787-0ca89fde517d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kcdqm\" (UID: \"2ad21c46-2e06-4b88-a787-0ca89fde517d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kcdqm" Nov 03 02:33:35.845347 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:35.845314 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2ad21c46-2e06-4b88-a787-0ca89fde517d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kcdqm\" (UID: \"2ad21c46-2e06-4b88-a787-0ca89fde517d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kcdqm" Nov 03 02:33:35.846087 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:35.846042 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2ad21c46-2e06-4b88-a787-0ca89fde517d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kcdqm\" (UID: \"2ad21c46-2e06-4b88-a787-0ca89fde517d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kcdqm" Nov 03 02:33:35.846227 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:35.846204 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2ad21c46-2e06-4b88-a787-0ca89fde517d-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kcdqm\" (UID: \"2ad21c46-2e06-4b88-a787-0ca89fde517d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kcdqm" Nov 03 02:33:35.846769 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:35.846750 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ad21c46-2e06-4b88-a787-0ca89fde517d-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kcdqm\" (UID: \"2ad21c46-2e06-4b88-a787-0ca89fde517d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kcdqm" Nov 03 02:33:35.908464 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:35.908068 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ad21c46-2e06-4b88-a787-0ca89fde517d-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kcdqm\" (UID: \"2ad21c46-2e06-4b88-a787-0ca89fde517d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kcdqm" Nov 03 02:33:35.915913 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:35.911763 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr4qg\" (UniqueName: \"kubernetes.io/projected/2ad21c46-2e06-4b88-a787-0ca89fde517d-kube-api-access-mr4qg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kcdqm\" (UID: \"2ad21c46-2e06-4b88-a787-0ca89fde517d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kcdqm" Nov 03 02:33:35.969434 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:35.969392 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kcdqm" Nov 03 02:33:36.418808 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:36.418667 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kcdqm"] Nov 03 02:33:36.418808 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:36.418680 5605 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:33:36.418808 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:36.418739 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl" podUID="a15ffc44-4fe7-41cf-9370-794bebfda952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:33:36.424348 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:33:36.424319 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ad21c46_2e06_4b88_a787_0ca89fde517d.slice/crio-fa088bb9d9021e5798cc520085f45384d300b5add3a4db59d68387c2f5648749 WatchSource:0}: Error finding container fa088bb9d9021e5798cc520085f45384d300b5add3a4db59d68387c2f5648749: Status 404 returned error can't find the container with id fa088bb9d9021e5798cc520085f45384d300b5add3a4db59d68387c2f5648749 Nov 03 02:33:36.426755 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:36.426722 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:33:36.426850 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:36.426783 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:33:36.544636 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:36.544606 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl" Nov 03 02:33:36.545690 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:36.545664 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl" Nov 03 02:33:36.546345 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:36.546328 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl" Nov 03 02:33:37.157269 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:37.157228 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kcdqm" event={"ID":"2ad21c46-2e06-4b88-a787-0ca89fde517d","Type":"ContainerStarted","Data":"776be96a343db2ef38ca0cbec040649bcd4ae3422810ff3a5ce8fd71aa4bdd1d"} Nov 03 02:33:37.157491 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:37.157298 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kcdqm" Nov 03 02:33:37.157491 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:37.157319 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kcdqm" event={"ID":"2ad21c46-2e06-4b88-a787-0ca89fde517d","Type":"ContainerStarted","Data":"fa088bb9d9021e5798cc520085f45384d300b5add3a4db59d68387c2f5648749"} Nov 03 02:33:37.159226 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:37.159196 5605 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-kcdqm container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 02:33:37.159377 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:37.159241 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kcdqm" podUID="2ad21c46-2e06-4b88-a787-0ca89fde517d" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 02:33:37.272084 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:37.272033 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kcdqm" podStartSLOduration=2.272019987 podStartE2EDuration="2.272019987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:33:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:33:37.270111434 +0000 UTC m=+2494.529309580" watchObservedRunningTime="2026-11-03 02:33:37.272019987 +0000 UTC m=+2494.531218141" Nov 03 02:33:38.199379 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:38.198755 5605 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-kcdqm container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 02:33:38.199379 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:38.198837 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kcdqm" podUID="2ad21c46-2e06-4b88-a787-0ca89fde517d" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 02:33:41.169408 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:41.169372 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kcdqm" Nov 03 02:33:41.469647 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:41.469559 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8"] Nov 03 02:33:41.469833 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:41.469799 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8" podUID="cb319f5e-21f0-486e-b6c0-fe44875c545a" containerName="istio-proxy" containerID="cri-o://f341e6ebb3322bbc49345149b6b05d23926f97cfdff43ec8a771e2dc5e74dd73" gracePeriod=30 Nov 03 02:33:41.493292 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:41.493252 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn"] Nov 03 02:33:41.493624 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:41.493600 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn" podUID="dce11384-8e67-4aef-80ab-3ddd74099bf7" containerName="istio-proxy" containerID="cri-o://dc4083f3aa1d1b5667dd36dbd8d19f11469cd71fa3dd59e23f70c65d26ab6634" gracePeriod=30 Nov 03 02:33:41.497185 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:41.497151 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw"] Nov 03 02:33:41.497406 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:41.497377 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw" podUID="229d9be7-d6b9-4760-80ed-84592e5693d4" containerName="istio-proxy" containerID="cri-o://8694c2b29032070891f4a82edcaa1aa30f11420a997230472ae76da1c7fef1de" gracePeriod=30 Nov 03 02:33:42.221002 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:42.220969 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp"] Nov 03 02:33:42.225181 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:42.225160 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp" Nov 03 02:33:42.281248 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:42.281220 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp"] Nov 03 02:33:42.361334 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:42.361307 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/643d6829-f08f-4bb6-a2e4-d9a9c523224a-istio-token\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp\" (UID: \"643d6829-f08f-4bb6-a2e4-d9a9c523224a\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp" Nov 03 02:33:42.361519 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:42.361346 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/643d6829-f08f-4bb6-a2e4-d9a9c523224a-istio-podinfo\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp\" (UID: \"643d6829-f08f-4bb6-a2e4-d9a9c523224a\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp" Nov 03 02:33:42.361519 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:42.361474 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/643d6829-f08f-4bb6-a2e4-d9a9c523224a-workload-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp\" (UID: \"643d6829-f08f-4bb6-a2e4-d9a9c523224a\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp" Nov 03 02:33:42.361519 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:42.361515 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/643d6829-f08f-4bb6-a2e4-d9a9c523224a-workload-certs\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp\" (UID: \"643d6829-f08f-4bb6-a2e4-d9a9c523224a\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp" Nov 03 02:33:42.361699 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:42.361589 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j7xv\" (UniqueName: \"kubernetes.io/projected/643d6829-f08f-4bb6-a2e4-d9a9c523224a-kube-api-access-5j7xv\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp\" (UID: \"643d6829-f08f-4bb6-a2e4-d9a9c523224a\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp" Nov 03 02:33:42.361699 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:42.361635 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/643d6829-f08f-4bb6-a2e4-d9a9c523224a-istiod-ca-cert\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp\" (UID: \"643d6829-f08f-4bb6-a2e4-d9a9c523224a\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp" Nov 03 02:33:42.361803 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:42.361717 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/643d6829-f08f-4bb6-a2e4-d9a9c523224a-credential-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp\" (UID: \"643d6829-f08f-4bb6-a2e4-d9a9c523224a\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp" Nov 03 02:33:42.361803 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:42.361767 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/643d6829-f08f-4bb6-a2e4-d9a9c523224a-istio-envoy\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp\" (UID: \"643d6829-f08f-4bb6-a2e4-d9a9c523224a\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp" Nov 03 02:33:42.361905 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:42.361799 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/643d6829-f08f-4bb6-a2e4-d9a9c523224a-istio-data\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp\" (UID: \"643d6829-f08f-4bb6-a2e4-d9a9c523224a\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp" Nov 03 02:33:42.462520 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:42.462488 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/643d6829-f08f-4bb6-a2e4-d9a9c523224a-credential-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp\" (UID: \"643d6829-f08f-4bb6-a2e4-d9a9c523224a\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp" Nov 03 02:33:42.462674 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:42.462542 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/643d6829-f08f-4bb6-a2e4-d9a9c523224a-istio-envoy\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp\" (UID: \"643d6829-f08f-4bb6-a2e4-d9a9c523224a\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp" Nov 03 02:33:42.462674 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:42.462569 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/643d6829-f08f-4bb6-a2e4-d9a9c523224a-istio-data\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp\" (UID: \"643d6829-f08f-4bb6-a2e4-d9a9c523224a\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp" Nov 03 02:33:42.462674 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:42.462606 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/643d6829-f08f-4bb6-a2e4-d9a9c523224a-istio-token\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp\" (UID: \"643d6829-f08f-4bb6-a2e4-d9a9c523224a\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp" Nov 03 02:33:42.462674 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:42.462631 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/643d6829-f08f-4bb6-a2e4-d9a9c523224a-istio-podinfo\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp\" (UID: \"643d6829-f08f-4bb6-a2e4-d9a9c523224a\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp" Nov 03 02:33:42.462874 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:42.462766 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/643d6829-f08f-4bb6-a2e4-d9a9c523224a-workload-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp\" (UID: \"643d6829-f08f-4bb6-a2e4-d9a9c523224a\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp" Nov 03 02:33:42.462874 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:42.462821 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/643d6829-f08f-4bb6-a2e4-d9a9c523224a-workload-certs\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp\" (UID: \"643d6829-f08f-4bb6-a2e4-d9a9c523224a\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp" Nov 03 02:33:42.462989 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:42.462871 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j7xv\" (UniqueName: \"kubernetes.io/projected/643d6829-f08f-4bb6-a2e4-d9a9c523224a-kube-api-access-5j7xv\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp\" (UID: \"643d6829-f08f-4bb6-a2e4-d9a9c523224a\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp" Nov 03 02:33:42.462989 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:42.462926 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/643d6829-f08f-4bb6-a2e4-d9a9c523224a-istiod-ca-cert\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp\" (UID: \"643d6829-f08f-4bb6-a2e4-d9a9c523224a\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp" Nov 03 02:33:42.463126 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:42.463101 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/643d6829-f08f-4bb6-a2e4-d9a9c523224a-istio-data\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp\" (UID: \"643d6829-f08f-4bb6-a2e4-d9a9c523224a\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp" Nov 03 02:33:42.463192 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:42.463120 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/643d6829-f08f-4bb6-a2e4-d9a9c523224a-workload-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp\" (UID: \"643d6829-f08f-4bb6-a2e4-d9a9c523224a\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp" Nov 03 02:33:42.463264 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:42.463246 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/643d6829-f08f-4bb6-a2e4-d9a9c523224a-credential-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp\" (UID: \"643d6829-f08f-4bb6-a2e4-d9a9c523224a\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp" Nov 03 02:33:42.463302 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:42.463242 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/643d6829-f08f-4bb6-a2e4-d9a9c523224a-workload-certs\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp\" (UID: \"643d6829-f08f-4bb6-a2e4-d9a9c523224a\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp" Nov 03 02:33:42.463786 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:42.463763 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/643d6829-f08f-4bb6-a2e4-d9a9c523224a-istiod-ca-cert\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp\" (UID: \"643d6829-f08f-4bb6-a2e4-d9a9c523224a\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp" Nov 03 02:33:42.464812 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:42.464793 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/643d6829-f08f-4bb6-a2e4-d9a9c523224a-istio-envoy\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp\" (UID: \"643d6829-f08f-4bb6-a2e4-d9a9c523224a\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp" Nov 03 02:33:42.465246 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:42.465223 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/643d6829-f08f-4bb6-a2e4-d9a9c523224a-istio-podinfo\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp\" (UID: \"643d6829-f08f-4bb6-a2e4-d9a9c523224a\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp" Nov 03 02:33:42.516718 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:42.516662 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/643d6829-f08f-4bb6-a2e4-d9a9c523224a-istio-token\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp\" (UID: \"643d6829-f08f-4bb6-a2e4-d9a9c523224a\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp" Nov 03 02:33:42.517319 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:42.517294 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j7xv\" (UniqueName: \"kubernetes.io/projected/643d6829-f08f-4bb6-a2e4-d9a9c523224a-kube-api-access-5j7xv\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp\" (UID: \"643d6829-f08f-4bb6-a2e4-d9a9c523224a\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp" Nov 03 02:33:42.539148 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:42.539126 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp" Nov 03 02:33:42.962867 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:42.962837 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp"] Nov 03 02:33:42.966966 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:33:42.966937 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod643d6829_f08f_4bb6_a2e4_d9a9c523224a.slice/crio-637cc68be77205bffa1e1f208af471b828f8d0b583ed4a86201ef433e3e17379 WatchSource:0}: Error finding container 637cc68be77205bffa1e1f208af471b828f8d0b583ed4a86201ef433e3e17379: Status 404 returned error can't find the container with id 637cc68be77205bffa1e1f208af471b828f8d0b583ed4a86201ef433e3e17379 Nov 03 02:33:42.969260 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:42.969228 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:33:42.969335 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:42.969295 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:33:42.969335 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:42.969329 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:33:43.200525 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:43.200485 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp" event={"ID":"643d6829-f08f-4bb6-a2e4-d9a9c523224a","Type":"ContainerStarted","Data":"5e8616c2b2ea0e3e7c9941d908c4a4205ac9dd3901935a02c41365eeaec943f8"} Nov 03 02:33:43.200679 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:43.200548 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp" event={"ID":"643d6829-f08f-4bb6-a2e4-d9a9c523224a","Type":"ContainerStarted","Data":"637cc68be77205bffa1e1f208af471b828f8d0b583ed4a86201ef433e3e17379"} Nov 03 02:33:43.312828 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:43.312771 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp" podStartSLOduration=1.3127528640000001 podStartE2EDuration="1.312752864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:33:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:33:43.310755364 +0000 UTC m=+2500.569953532" watchObservedRunningTime="2026-11-03 02:33:43.312752864 +0000 UTC m=+2500.571951018" Nov 03 02:33:43.540292 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:43.540256 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp" Nov 03 02:33:44.541521 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:44.541481 5605 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 03 02:33:44.541992 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:44.541531 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp" podUID="643d6829-f08f-4bb6-a2e4-d9a9c523224a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 03 02:33:45.145497 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:45.145462 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9aebe8-66bd979655-9t5kx" Nov 03 02:33:45.539616 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:45.539541 5605 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 03 02:33:45.539616 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:45.539593 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp" podUID="643d6829-f08f-4bb6-a2e4-d9a9c523224a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 03 02:33:46.151221 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.151191 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-9aebe8-9cb69fc69-plhsd" Nov 03 02:33:46.539525 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.539489 5605 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 03 02:33:46.539653 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.539544 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp" podUID="643d6829-f08f-4bb6-a2e4-d9a9c523224a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 03 02:33:46.816595 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.816570 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn" Nov 03 02:33:46.821517 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.821452 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8" Nov 03 02:33:46.843555 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.843459 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb319f5e-21f0-486e-b6c0-fe44875c545a-istio-token\" (UniqueName: \"kubernetes.io/projected/cb319f5e-21f0-486e-b6c0-fe44875c545a-istio-token\") pod \"cb319f5e-21f0-486e-b6c0-fe44875c545a\" (UID: \"cb319f5e-21f0-486e-b6c0-fe44875c545a\") " Nov 03 02:33:46.843555 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.843512 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dce11384-8e67-4aef-80ab-3ddd74099bf7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dce11384-8e67-4aef-80ab-3ddd74099bf7-istio-envoy\") pod \"dce11384-8e67-4aef-80ab-3ddd74099bf7\" (UID: \"dce11384-8e67-4aef-80ab-3ddd74099bf7\") " Nov 03 02:33:46.843952 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.843923 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dce11384-8e67-4aef-80ab-3ddd74099bf7-kube-api-access-jmbt8\" (UniqueName: \"kubernetes.io/projected/dce11384-8e67-4aef-80ab-3ddd74099bf7-kube-api-access-jmbt8\") pod \"dce11384-8e67-4aef-80ab-3ddd74099bf7\" (UID: \"dce11384-8e67-4aef-80ab-3ddd74099bf7\") " Nov 03 02:33:46.844040 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.843984 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb319f5e-21f0-486e-b6c0-fe44875c545a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb319f5e-21f0-486e-b6c0-fe44875c545a-workload-certs\") pod \"cb319f5e-21f0-486e-b6c0-fe44875c545a\" (UID: \"cb319f5e-21f0-486e-b6c0-fe44875c545a\") " Nov 03 02:33:46.844040 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.844013 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dce11384-8e67-4aef-80ab-3ddd74099bf7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dce11384-8e67-4aef-80ab-3ddd74099bf7-istiod-ca-cert\") pod \"dce11384-8e67-4aef-80ab-3ddd74099bf7\" (UID: \"dce11384-8e67-4aef-80ab-3ddd74099bf7\") " Nov 03 02:33:46.844160 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.844045 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dce11384-8e67-4aef-80ab-3ddd74099bf7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce11384-8e67-4aef-80ab-3ddd74099bf7-credential-socket\") pod \"dce11384-8e67-4aef-80ab-3ddd74099bf7\" (UID: \"dce11384-8e67-4aef-80ab-3ddd74099bf7\") " Nov 03 02:33:46.844160 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.844074 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dce11384-8e67-4aef-80ab-3ddd74099bf7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dce11384-8e67-4aef-80ab-3ddd74099bf7-workload-certs\") pod \"dce11384-8e67-4aef-80ab-3ddd74099bf7\" (UID: \"dce11384-8e67-4aef-80ab-3ddd74099bf7\") " Nov 03 02:33:46.844160 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.844105 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dce11384-8e67-4aef-80ab-3ddd74099bf7-istio-token\" (UniqueName: \"kubernetes.io/projected/dce11384-8e67-4aef-80ab-3ddd74099bf7-istio-token\") pod \"dce11384-8e67-4aef-80ab-3ddd74099bf7\" (UID: \"dce11384-8e67-4aef-80ab-3ddd74099bf7\") " Nov 03 02:33:46.844160 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.844137 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb319f5e-21f0-486e-b6c0-fe44875c545a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb319f5e-21f0-486e-b6c0-fe44875c545a-istiod-ca-cert\") pod \"cb319f5e-21f0-486e-b6c0-fe44875c545a\" (UID: \"cb319f5e-21f0-486e-b6c0-fe44875c545a\") " Nov 03 02:33:46.844375 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.844185 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb319f5e-21f0-486e-b6c0-fe44875c545a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb319f5e-21f0-486e-b6c0-fe44875c545a-workload-socket\") pod \"cb319f5e-21f0-486e-b6c0-fe44875c545a\" (UID: \"cb319f5e-21f0-486e-b6c0-fe44875c545a\") " Nov 03 02:33:46.844375 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.844249 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb319f5e-21f0-486e-b6c0-fe44875c545a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb319f5e-21f0-486e-b6c0-fe44875c545a-istio-envoy\") pod \"cb319f5e-21f0-486e-b6c0-fe44875c545a\" (UID: \"cb319f5e-21f0-486e-b6c0-fe44875c545a\") " Nov 03 02:33:46.844503 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.844404 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dce11384-8e67-4aef-80ab-3ddd74099bf7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dce11384-8e67-4aef-80ab-3ddd74099bf7-istio-data\") pod \"dce11384-8e67-4aef-80ab-3ddd74099bf7\" (UID: \"dce11384-8e67-4aef-80ab-3ddd74099bf7\") " Nov 03 02:33:46.844503 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.844444 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb319f5e-21f0-486e-b6c0-fe44875c545a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb319f5e-21f0-486e-b6c0-fe44875c545a-istio-data\") pod \"cb319f5e-21f0-486e-b6c0-fe44875c545a\" (UID: \"cb319f5e-21f0-486e-b6c0-fe44875c545a\") " Nov 03 02:33:46.844615 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.844493 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb319f5e-21f0-486e-b6c0-fe44875c545a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb319f5e-21f0-486e-b6c0-fe44875c545a-istio-podinfo\") pod \"cb319f5e-21f0-486e-b6c0-fe44875c545a\" (UID: \"cb319f5e-21f0-486e-b6c0-fe44875c545a\") " Nov 03 02:33:46.844615 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.844545 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb319f5e-21f0-486e-b6c0-fe44875c545a-kube-api-access-dhc4f\" (UniqueName: \"kubernetes.io/projected/cb319f5e-21f0-486e-b6c0-fe44875c545a-kube-api-access-dhc4f\") pod \"cb319f5e-21f0-486e-b6c0-fe44875c545a\" (UID: \"cb319f5e-21f0-486e-b6c0-fe44875c545a\") " Nov 03 02:33:46.844734 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.844654 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dce11384-8e67-4aef-80ab-3ddd74099bf7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce11384-8e67-4aef-80ab-3ddd74099bf7-workload-socket\") pod \"dce11384-8e67-4aef-80ab-3ddd74099bf7\" (UID: \"dce11384-8e67-4aef-80ab-3ddd74099bf7\") " Nov 03 02:33:46.844734 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.844687 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dce11384-8e67-4aef-80ab-3ddd74099bf7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dce11384-8e67-4aef-80ab-3ddd74099bf7-istio-podinfo\") pod \"dce11384-8e67-4aef-80ab-3ddd74099bf7\" (UID: \"dce11384-8e67-4aef-80ab-3ddd74099bf7\") " Nov 03 02:33:46.844838 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.844745 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb319f5e-21f0-486e-b6c0-fe44875c545a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb319f5e-21f0-486e-b6c0-fe44875c545a-credential-socket\") pod \"cb319f5e-21f0-486e-b6c0-fe44875c545a\" (UID: \"cb319f5e-21f0-486e-b6c0-fe44875c545a\") " Nov 03 02:33:46.846554 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.845808 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce11384-8e67-4aef-80ab-3ddd74099bf7-istio-data" pod "dce11384-8e67-4aef-80ab-3ddd74099bf7" (UID: "dce11384-8e67-4aef-80ab-3ddd74099bf7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:33:46.846554 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.845820 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb319f5e-21f0-486e-b6c0-fe44875c545a-credential-socket" pod "cb319f5e-21f0-486e-b6c0-fe44875c545a" (UID: "cb319f5e-21f0-486e-b6c0-fe44875c545a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:33:46.846554 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.846110 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb319f5e-21f0-486e-b6c0-fe44875c545a-workload-socket" pod "cb319f5e-21f0-486e-b6c0-fe44875c545a" (UID: "cb319f5e-21f0-486e-b6c0-fe44875c545a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:33:46.846554 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.846381 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb319f5e-21f0-486e-b6c0-fe44875c545a-workload-certs" pod "cb319f5e-21f0-486e-b6c0-fe44875c545a" (UID: "cb319f5e-21f0-486e-b6c0-fe44875c545a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:33:46.846824 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.846690 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dce11384-8e67-4aef-80ab-3ddd74099bf7-kube-api-access-jmbt8" pod "dce11384-8e67-4aef-80ab-3ddd74099bf7" (UID: "dce11384-8e67-4aef-80ab-3ddd74099bf7"). InnerVolumeSpecName "kube-api-access-jmbt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:33:46.847498 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.846909 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce11384-8e67-4aef-80ab-3ddd74099bf7-workload-socket" pod "dce11384-8e67-4aef-80ab-3ddd74099bf7" (UID: "dce11384-8e67-4aef-80ab-3ddd74099bf7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:33:46.847498 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.847033 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dce11384-8e67-4aef-80ab-3ddd74099bf7-istiod-ca-cert" pod "dce11384-8e67-4aef-80ab-3ddd74099bf7" (UID: "dce11384-8e67-4aef-80ab-3ddd74099bf7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:33:46.847498 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.847127 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce11384-8e67-4aef-80ab-3ddd74099bf7-workload-certs" pod "dce11384-8e67-4aef-80ab-3ddd74099bf7" (UID: "dce11384-8e67-4aef-80ab-3ddd74099bf7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:33:46.847498 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.847299 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce11384-8e67-4aef-80ab-3ddd74099bf7-credential-socket" pod "dce11384-8e67-4aef-80ab-3ddd74099bf7" (UID: "dce11384-8e67-4aef-80ab-3ddd74099bf7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:33:46.847771 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.847542 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb319f5e-21f0-486e-b6c0-fe44875c545a-istio-data" pod "cb319f5e-21f0-486e-b6c0-fe44875c545a" (UID: "cb319f5e-21f0-486e-b6c0-fe44875c545a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:33:46.847771 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.847667 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb319f5e-21f0-486e-b6c0-fe44875c545a-istiod-ca-cert" pod "cb319f5e-21f0-486e-b6c0-fe44875c545a" (UID: "cb319f5e-21f0-486e-b6c0-fe44875c545a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:33:46.848191 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.848168 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb319f5e-21f0-486e-b6c0-fe44875c545a-istio-podinfo" pod "cb319f5e-21f0-486e-b6c0-fe44875c545a" (UID: "cb319f5e-21f0-486e-b6c0-fe44875c545a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:33:46.851455 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.850277 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb319f5e-21f0-486e-b6c0-fe44875c545a-kube-api-access-dhc4f" pod "cb319f5e-21f0-486e-b6c0-fe44875c545a" (UID: "cb319f5e-21f0-486e-b6c0-fe44875c545a"). InnerVolumeSpecName "kube-api-access-dhc4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:33:46.851455 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.850352 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb319f5e-21f0-486e-b6c0-fe44875c545a-istio-token" pod "cb319f5e-21f0-486e-b6c0-fe44875c545a" (UID: "cb319f5e-21f0-486e-b6c0-fe44875c545a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:33:46.851455 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.850395 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dce11384-8e67-4aef-80ab-3ddd74099bf7-istio-podinfo" pod "dce11384-8e67-4aef-80ab-3ddd74099bf7" (UID: "dce11384-8e67-4aef-80ab-3ddd74099bf7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:33:46.851455 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.850438 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce11384-8e67-4aef-80ab-3ddd74099bf7-istio-envoy" pod "dce11384-8e67-4aef-80ab-3ddd74099bf7" (UID: "dce11384-8e67-4aef-80ab-3ddd74099bf7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:33:46.851455 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.851177 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dce11384-8e67-4aef-80ab-3ddd74099bf7-istio-token" pod "dce11384-8e67-4aef-80ab-3ddd74099bf7" (UID: "dce11384-8e67-4aef-80ab-3ddd74099bf7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:33:46.851455 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.851429 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb319f5e-21f0-486e-b6c0-fe44875c545a-istio-envoy" pod "cb319f5e-21f0-486e-b6c0-fe44875c545a" (UID: "cb319f5e-21f0-486e-b6c0-fe44875c545a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:33:46.899524 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.899502 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw" Nov 03 02:33:46.946398 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.946349 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/229d9be7-d6b9-4760-80ed-84592e5693d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/229d9be7-d6b9-4760-80ed-84592e5693d4-istio-podinfo\") pod \"229d9be7-d6b9-4760-80ed-84592e5693d4\" (UID: \"229d9be7-d6b9-4760-80ed-84592e5693d4\") " Nov 03 02:33:46.946499 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.946473 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/229d9be7-d6b9-4760-80ed-84592e5693d4-kube-api-access-qpdmk\" (UniqueName: \"kubernetes.io/projected/229d9be7-d6b9-4760-80ed-84592e5693d4-kube-api-access-qpdmk\") pod \"229d9be7-d6b9-4760-80ed-84592e5693d4\" (UID: \"229d9be7-d6b9-4760-80ed-84592e5693d4\") " Nov 03 02:33:46.946543 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.946510 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/229d9be7-d6b9-4760-80ed-84592e5693d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/229d9be7-d6b9-4760-80ed-84592e5693d4-credential-socket\") pod \"229d9be7-d6b9-4760-80ed-84592e5693d4\" (UID: \"229d9be7-d6b9-4760-80ed-84592e5693d4\") " Nov 03 02:33:46.946608 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.946589 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/229d9be7-d6b9-4760-80ed-84592e5693d4-istio-token\" (UniqueName: \"kubernetes.io/projected/229d9be7-d6b9-4760-80ed-84592e5693d4-istio-token\") pod \"229d9be7-d6b9-4760-80ed-84592e5693d4\" (UID: \"229d9be7-d6b9-4760-80ed-84592e5693d4\") " Nov 03 02:33:46.946659 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.946638 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/229d9be7-d6b9-4760-80ed-84592e5693d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/229d9be7-d6b9-4760-80ed-84592e5693d4-istio-envoy\") pod \"229d9be7-d6b9-4760-80ed-84592e5693d4\" (UID: \"229d9be7-d6b9-4760-80ed-84592e5693d4\") " Nov 03 02:33:46.946712 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.946676 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/229d9be7-d6b9-4760-80ed-84592e5693d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/229d9be7-d6b9-4760-80ed-84592e5693d4-workload-certs\") pod \"229d9be7-d6b9-4760-80ed-84592e5693d4\" (UID: \"229d9be7-d6b9-4760-80ed-84592e5693d4\") " Nov 03 02:33:46.946770 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.946723 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/229d9be7-d6b9-4760-80ed-84592e5693d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/229d9be7-d6b9-4760-80ed-84592e5693d4-istio-data\") pod \"229d9be7-d6b9-4760-80ed-84592e5693d4\" (UID: \"229d9be7-d6b9-4760-80ed-84592e5693d4\") " Nov 03 02:33:46.946770 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.946754 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/229d9be7-d6b9-4760-80ed-84592e5693d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/229d9be7-d6b9-4760-80ed-84592e5693d4-istiod-ca-cert\") pod \"229d9be7-d6b9-4760-80ed-84592e5693d4\" (UID: \"229d9be7-d6b9-4760-80ed-84592e5693d4\") " Nov 03 02:33:46.946871 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.946800 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/229d9be7-d6b9-4760-80ed-84592e5693d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/229d9be7-d6b9-4760-80ed-84592e5693d4-workload-socket\") pod \"229d9be7-d6b9-4760-80ed-84592e5693d4\" (UID: \"229d9be7-d6b9-4760-80ed-84592e5693d4\") " Nov 03 02:33:46.946923 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.946869 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/229d9be7-d6b9-4760-80ed-84592e5693d4-credential-socket" pod "229d9be7-d6b9-4760-80ed-84592e5693d4" (UID: "229d9be7-d6b9-4760-80ed-84592e5693d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:33:46.947082 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.946995 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/229d9be7-d6b9-4760-80ed-84592e5693d4-workload-certs" pod "229d9be7-d6b9-4760-80ed-84592e5693d4" (UID: "229d9be7-d6b9-4760-80ed-84592e5693d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:33:46.947187 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.947165 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/229d9be7-d6b9-4760-80ed-84592e5693d4-istio-data" pod "229d9be7-d6b9-4760-80ed-84592e5693d4" (UID: "229d9be7-d6b9-4760-80ed-84592e5693d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:33:46.947545 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.947500 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/229d9be7-d6b9-4760-80ed-84592e5693d4-workload-socket" pod "229d9be7-d6b9-4760-80ed-84592e5693d4" (UID: "229d9be7-d6b9-4760-80ed-84592e5693d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:33:46.947856 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.947830 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/229d9be7-d6b9-4760-80ed-84592e5693d4-istiod-ca-cert" pod "229d9be7-d6b9-4760-80ed-84592e5693d4" (UID: "229d9be7-d6b9-4760-80ed-84592e5693d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:33:46.948868 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.948062 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmbt8\" (UniqueName: \"kubernetes.io/projected/dce11384-8e67-4aef-80ab-3ddd74099bf7-kube-api-access-jmbt8\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:46.948868 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.948630 5605 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb319f5e-21f0-486e-b6c0-fe44875c545a-workload-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:46.948868 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.948649 5605 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dce11384-8e67-4aef-80ab-3ddd74099bf7-istiod-ca-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:46.948868 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.948662 5605 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce11384-8e67-4aef-80ab-3ddd74099bf7-credential-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:46.948868 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.948675 5605 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dce11384-8e67-4aef-80ab-3ddd74099bf7-workload-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:46.948868 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.948687 5605 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dce11384-8e67-4aef-80ab-3ddd74099bf7-istio-token\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:46.948868 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.948700 5605 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb319f5e-21f0-486e-b6c0-fe44875c545a-istiod-ca-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:46.948868 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.948712 5605 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb319f5e-21f0-486e-b6c0-fe44875c545a-workload-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:46.948868 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.948724 5605 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb319f5e-21f0-486e-b6c0-fe44875c545a-istio-envoy\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:46.948868 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.948737 5605 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/229d9be7-d6b9-4760-80ed-84592e5693d4-credential-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:46.948868 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.948751 5605 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dce11384-8e67-4aef-80ab-3ddd74099bf7-istio-data\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:46.948868 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.948763 5605 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb319f5e-21f0-486e-b6c0-fe44875c545a-istio-data\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:46.948868 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.948776 5605 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb319f5e-21f0-486e-b6c0-fe44875c545a-istio-podinfo\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:46.948868 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.948789 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhc4f\" (UniqueName: \"kubernetes.io/projected/cb319f5e-21f0-486e-b6c0-fe44875c545a-kube-api-access-dhc4f\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:46.948868 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.948804 5605 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/229d9be7-d6b9-4760-80ed-84592e5693d4-workload-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:46.948868 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.948816 5605 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce11384-8e67-4aef-80ab-3ddd74099bf7-workload-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:46.948868 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.948829 5605 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dce11384-8e67-4aef-80ab-3ddd74099bf7-istio-podinfo\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:46.948868 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.948843 5605 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/229d9be7-d6b9-4760-80ed-84592e5693d4-istio-data\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:46.948868 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.948856 5605 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb319f5e-21f0-486e-b6c0-fe44875c545a-credential-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:46.948868 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.948869 5605 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb319f5e-21f0-486e-b6c0-fe44875c545a-istio-token\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:46.948868 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.948881 5605 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dce11384-8e67-4aef-80ab-3ddd74099bf7-istio-envoy\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:46.950188 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.949092 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/229d9be7-d6b9-4760-80ed-84592e5693d4-istio-podinfo" pod "229d9be7-d6b9-4760-80ed-84592e5693d4" (UID: "229d9be7-d6b9-4760-80ed-84592e5693d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:33:46.950188 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.949185 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/229d9be7-d6b9-4760-80ed-84592e5693d4-kube-api-access-qpdmk" pod "229d9be7-d6b9-4760-80ed-84592e5693d4" (UID: "229d9be7-d6b9-4760-80ed-84592e5693d4"). InnerVolumeSpecName "kube-api-access-qpdmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:33:46.950188 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.950057 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/229d9be7-d6b9-4760-80ed-84592e5693d4-istio-envoy" pod "229d9be7-d6b9-4760-80ed-84592e5693d4" (UID: "229d9be7-d6b9-4760-80ed-84592e5693d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:33:46.950836 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:46.950808 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/229d9be7-d6b9-4760-80ed-84592e5693d4-istio-token" pod "229d9be7-d6b9-4760-80ed-84592e5693d4" (UID: "229d9be7-d6b9-4760-80ed-84592e5693d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:33:47.049759 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:47.049729 5605 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/229d9be7-d6b9-4760-80ed-84592e5693d4-istio-token\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:47.049894 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:47.049769 5605 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/229d9be7-d6b9-4760-80ed-84592e5693d4-istio-envoy\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:47.049894 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:47.049778 5605 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/229d9be7-d6b9-4760-80ed-84592e5693d4-istiod-ca-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:47.049894 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:47.049787 5605 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/229d9be7-d6b9-4760-80ed-84592e5693d4-workload-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:47.049894 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:47.049796 5605 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/229d9be7-d6b9-4760-80ed-84592e5693d4-istio-podinfo\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:47.049894 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:47.049805 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpdmk\" (UniqueName: \"kubernetes.io/projected/229d9be7-d6b9-4760-80ed-84592e5693d4-kube-api-access-qpdmk\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:47.231604 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:47.231565 5605 generic.go:356] "Generic (PLEG): container finished" podID="dce11384-8e67-4aef-80ab-3ddd74099bf7" containerID="dc4083f3aa1d1b5667dd36dbd8d19f11469cd71fa3dd59e23f70c65d26ab6634" exitCode=0 Nov 03 02:33:47.232038 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:47.231612 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn" Nov 03 02:33:47.232038 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:47.231647 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn" event={"ID":"dce11384-8e67-4aef-80ab-3ddd74099bf7","Type":"ContainerDied","Data":"dc4083f3aa1d1b5667dd36dbd8d19f11469cd71fa3dd59e23f70c65d26ab6634"} Nov 03 02:33:47.232038 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:47.231700 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn" event={"ID":"dce11384-8e67-4aef-80ab-3ddd74099bf7","Type":"ContainerDied","Data":"fa2026ec6b02569ae04f0e8d3bfff77c96e98756657ff6fa668c8b4f61f00ab6"} Nov 03 02:33:47.232038 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:47.231732 5605 scope.go:122] "RemoveContainer" containerID="dc4083f3aa1d1b5667dd36dbd8d19f11469cd71fa3dd59e23f70c65d26ab6634" Nov 03 02:33:47.234017 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:47.233917 5605 generic.go:356] "Generic (PLEG): container finished" podID="229d9be7-d6b9-4760-80ed-84592e5693d4" containerID="8694c2b29032070891f4a82edcaa1aa30f11420a997230472ae76da1c7fef1de" exitCode=0 Nov 03 02:33:47.234940 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:47.234916 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw" Nov 03 02:33:47.236510 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:47.236486 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw" event={"ID":"229d9be7-d6b9-4760-80ed-84592e5693d4","Type":"ContainerDied","Data":"8694c2b29032070891f4a82edcaa1aa30f11420a997230472ae76da1c7fef1de"} Nov 03 02:33:47.237247 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:47.237221 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw" event={"ID":"229d9be7-d6b9-4760-80ed-84592e5693d4","Type":"ContainerDied","Data":"c8d278eb10ec880ee9f6bb68a35b283602dfc606c71d0b594fdf7d31d1316571"} Nov 03 02:33:47.239657 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:47.239633 5605 generic.go:356] "Generic (PLEG): container finished" podID="cb319f5e-21f0-486e-b6c0-fe44875c545a" containerID="f341e6ebb3322bbc49345149b6b05d23926f97cfdff43ec8a771e2dc5e74dd73" exitCode=0 Nov 03 02:33:47.239759 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:47.239745 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8" event={"ID":"cb319f5e-21f0-486e-b6c0-fe44875c545a","Type":"ContainerDied","Data":"f341e6ebb3322bbc49345149b6b05d23926f97cfdff43ec8a771e2dc5e74dd73"} Nov 03 02:33:47.239811 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:47.239783 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8" event={"ID":"cb319f5e-21f0-486e-b6c0-fe44875c545a","Type":"ContainerDied","Data":"324611e88fd2a33c2e3f6eb5c57fd417267feadb69f9ee34697dcc4f9ce72f7b"} Nov 03 02:33:47.239864 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:47.239852 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8" Nov 03 02:33:47.283582 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:47.283550 5605 scope.go:122] "RemoveContainer" containerID="dc4083f3aa1d1b5667dd36dbd8d19f11469cd71fa3dd59e23f70c65d26ab6634" Nov 03 02:33:47.284032 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:33:47.284003 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc4083f3aa1d1b5667dd36dbd8d19f11469cd71fa3dd59e23f70c65d26ab6634\": container with ID starting with dc4083f3aa1d1b5667dd36dbd8d19f11469cd71fa3dd59e23f70c65d26ab6634 not found: ID does not exist" containerID="dc4083f3aa1d1b5667dd36dbd8d19f11469cd71fa3dd59e23f70c65d26ab6634" Nov 03 02:33:47.284134 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:47.284046 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc4083f3aa1d1b5667dd36dbd8d19f11469cd71fa3dd59e23f70c65d26ab6634"} err="failed to get container status \"dc4083f3aa1d1b5667dd36dbd8d19f11469cd71fa3dd59e23f70c65d26ab6634\": rpc error: code = NotFound desc = could not find container \"dc4083f3aa1d1b5667dd36dbd8d19f11469cd71fa3dd59e23f70c65d26ab6634\": container with ID starting with dc4083f3aa1d1b5667dd36dbd8d19f11469cd71fa3dd59e23f70c65d26ab6634 not found: ID does not exist" Nov 03 02:33:47.284134 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:47.284072 5605 scope.go:122] "RemoveContainer" containerID="8694c2b29032070891f4a82edcaa1aa30f11420a997230472ae76da1c7fef1de" Nov 03 02:33:47.287647 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:47.287622 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:33:47.295016 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:47.294995 5605 scope.go:122] "RemoveContainer" containerID="8694c2b29032070891f4a82edcaa1aa30f11420a997230472ae76da1c7fef1de" Nov 03 02:33:47.295273 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:33:47.295251 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8694c2b29032070891f4a82edcaa1aa30f11420a997230472ae76da1c7fef1de\": container with ID starting with 8694c2b29032070891f4a82edcaa1aa30f11420a997230472ae76da1c7fef1de not found: ID does not exist" containerID="8694c2b29032070891f4a82edcaa1aa30f11420a997230472ae76da1c7fef1de" Nov 03 02:33:47.295341 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:47.295283 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8694c2b29032070891f4a82edcaa1aa30f11420a997230472ae76da1c7fef1de"} err="failed to get container status \"8694c2b29032070891f4a82edcaa1aa30f11420a997230472ae76da1c7fef1de\": rpc error: code = NotFound desc = could not find container \"8694c2b29032070891f4a82edcaa1aa30f11420a997230472ae76da1c7fef1de\": container with ID starting with 8694c2b29032070891f4a82edcaa1aa30f11420a997230472ae76da1c7fef1de not found: ID does not exist" Nov 03 02:33:47.295341 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:47.295300 5605 scope.go:122] "RemoveContainer" containerID="f341e6ebb3322bbc49345149b6b05d23926f97cfdff43ec8a771e2dc5e74dd73" Nov 03 02:33:47.304938 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:47.304917 5605 scope.go:122] "RemoveContainer" containerID="f341e6ebb3322bbc49345149b6b05d23926f97cfdff43ec8a771e2dc5e74dd73" Nov 03 02:33:47.305263 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:33:47.305238 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f341e6ebb3322bbc49345149b6b05d23926f97cfdff43ec8a771e2dc5e74dd73\": container with ID starting with f341e6ebb3322bbc49345149b6b05d23926f97cfdff43ec8a771e2dc5e74dd73 not found: ID does not exist" containerID="f341e6ebb3322bbc49345149b6b05d23926f97cfdff43ec8a771e2dc5e74dd73" Nov 03 02:33:47.305388 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:47.305270 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f341e6ebb3322bbc49345149b6b05d23926f97cfdff43ec8a771e2dc5e74dd73"} err="failed to get container status \"f341e6ebb3322bbc49345149b6b05d23926f97cfdff43ec8a771e2dc5e74dd73\": rpc error: code = NotFound desc = could not find container \"f341e6ebb3322bbc49345149b6b05d23926f97cfdff43ec8a771e2dc5e74dd73\": container with ID starting with f341e6ebb3322bbc49345149b6b05d23926f97cfdff43ec8a771e2dc5e74dd73 not found: ID does not exist" Nov 03 02:33:47.393621 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:47.393560 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8"] Nov 03 02:33:47.396769 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:47.396743 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-qj7j8"] Nov 03 02:33:47.498442 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:47.498419 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn"] Nov 03 02:33:47.503504 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:47.503480 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-b5zfn"] Nov 03 02:33:47.539527 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:47.539502 5605 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 03 02:33:47.539615 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:47.539554 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp" podUID="643d6829-f08f-4bb6-a2e4-d9a9c523224a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 03 02:33:47.600915 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:47.600839 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw"] Nov 03 02:33:47.605323 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:47.605298 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-jbkzw"] Nov 03 02:33:48.540425 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:48.540386 5605 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 03 02:33:48.540806 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:48.540445 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp" podUID="643d6829-f08f-4bb6-a2e4-d9a9c523224a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 03 02:33:48.850401 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:48.850344 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="229d9be7-d6b9-4760-80ed-84592e5693d4" path="/var/lib/kubelet/pods/229d9be7-d6b9-4760-80ed-84592e5693d4/volumes" Nov 03 02:33:48.850864 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:48.850849 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb319f5e-21f0-486e-b6c0-fe44875c545a" path="/var/lib/kubelet/pods/cb319f5e-21f0-486e-b6c0-fe44875c545a/volumes" Nov 03 02:33:48.851263 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:48.851251 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dce11384-8e67-4aef-80ab-3ddd74099bf7" path="/var/lib/kubelet/pods/dce11384-8e67-4aef-80ab-3ddd74099bf7/volumes" Nov 03 02:33:49.540529 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:49.540493 5605 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 03 02:33:49.540888 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:49.540563 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp" podUID="643d6829-f08f-4bb6-a2e4-d9a9c523224a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 03 02:33:50.543667 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:50.543637 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp" Nov 03 02:33:50.544072 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:50.544048 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp" Nov 03 02:33:50.544751 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:33:50.544729 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp" Nov 03 02:33:53.915495 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:33:53.915461 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:34:09.972601 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:09.972558 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:34:21.487923 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:21.487885 5605 generic.go:356] "Generic (PLEG): container finished" podID="804673d5-10dd-463e-9c52-fc67bd1c307a" containerID="ad496f17e0211a23c7841cc6aeee7430bf3e2835b24fdb916a66b3ee65b60328" exitCode=0 Nov 03 02:34:21.488486 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:21.487958 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl" event={"ID":"804673d5-10dd-463e-9c52-fc67bd1c307a","Type":"ContainerDied","Data":"ad496f17e0211a23c7841cc6aeee7430bf3e2835b24fdb916a66b3ee65b60328"} Nov 03 02:34:21.488486 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:21.488421 5605 scope.go:122] "RemoveContainer" containerID="ad496f17e0211a23c7841cc6aeee7430bf3e2835b24fdb916a66b3ee65b60328" Nov 03 02:34:21.492907 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:21.490440 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:34:21.492907 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:21.490535 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:34:21.492907 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:21.490577 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:34:22.496709 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:22.496662 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl" event={"ID":"804673d5-10dd-463e-9c52-fc67bd1c307a","Type":"ContainerStarted","Data":"82c22ebb2f60b02952e61747fb938df5e0ac16162973a3a42a9ee177bfedc8bf"} Nov 03 02:34:22.802246 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:22.802216 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl" Nov 03 02:34:23.333715 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:23.333678 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl" Nov 03 02:34:23.335105 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:23.335074 5605 patch_prober.go:29] interesting pod/e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 03 02:34:23.335262 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:23.335133 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl" podUID="804673d5-10dd-463e-9c52-fc67bd1c307a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 03 02:34:23.916465 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:34:23.916406 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:34:24.333776 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:24.333742 5605 patch_prober.go:29] interesting pod/e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 03 02:34:24.333969 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:24.333803 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl" podUID="804673d5-10dd-463e-9c52-fc67bd1c307a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 03 02:34:25.334383 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:25.334318 5605 patch_prober.go:29] interesting pod/e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 03 02:34:25.334809 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:25.334416 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl" podUID="804673d5-10dd-463e-9c52-fc67bd1c307a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 03 02:34:26.333871 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:26.333835 5605 patch_prober.go:29] interesting pod/e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 03 02:34:26.334059 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:26.333893 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl" podUID="804673d5-10dd-463e-9c52-fc67bd1c307a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 03 02:34:27.333653 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:27.333599 5605 patch_prober.go:29] interesting pod/e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 03 02:34:27.334083 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:27.333683 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl" podUID="804673d5-10dd-463e-9c52-fc67bd1c307a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 03 02:34:28.338729 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:28.338691 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl" Nov 03 02:34:28.339768 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:28.339744 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl" Nov 03 02:34:31.049965 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:31.049920 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e013133c-f9hgs"] Nov 03 02:34:31.096680 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:31.096633 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e013133c-f9hgs"] Nov 03 02:34:32.851719 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:32.851685 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13192b33-c9a3-422f-a2cd-79c52dd55715" path="/var/lib/kubelet/pods/13192b33-c9a3-422f-a2cd-79c52dd55715/volumes" Nov 03 02:34:40.454873 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:40.454829 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb"] Nov 03 02:34:40.455383 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:40.455305 5605 cpu_manager.go:414] "containerMap: removing container" podUID="cb319f5e-21f0-486e-b6c0-fe44875c545a" containerName="istio-proxy" Nov 03 02:34:40.455383 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:40.455317 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb319f5e-21f0-486e-b6c0-fe44875c545a" containerName="istio-proxy" Nov 03 02:34:40.455383 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:40.455350 5605 cpu_manager.go:414] "containerMap: removing container" podUID="229d9be7-d6b9-4760-80ed-84592e5693d4" containerName="istio-proxy" Nov 03 02:34:40.455383 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:40.455371 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="229d9be7-d6b9-4760-80ed-84592e5693d4" containerName="istio-proxy" Nov 03 02:34:40.455383 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:40.455384 5605 cpu_manager.go:414] "containerMap: removing container" podUID="dce11384-8e67-4aef-80ab-3ddd74099bf7" containerName="istio-proxy" Nov 03 02:34:40.455557 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:40.455390 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dce11384-8e67-4aef-80ab-3ddd74099bf7" containerName="istio-proxy" Nov 03 02:34:40.455557 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:40.455461 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb319f5e-21f0-486e-b6c0-fe44875c545a" containerName="istio-proxy" Nov 03 02:34:40.455557 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:40.455473 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="229d9be7-d6b9-4760-80ed-84592e5693d4" containerName="istio-proxy" Nov 03 02:34:40.455557 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:40.455481 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="dce11384-8e67-4aef-80ab-3ddd74099bf7" containerName="istio-proxy" Nov 03 02:34:40.459156 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:40.459132 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb" Nov 03 02:34:40.507840 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:40.507797 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-f735f4-openshift-default-dockercfg-4w55v\"" Nov 03 02:34:40.558514 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:40.558474 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb"] Nov 03 02:34:40.578316 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:40.578272 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e3a3bd1-724d-454c-9e5e-615b71be6ab4-istiod-ca-cert\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb\" (UID: \"2e3a3bd1-724d-454c-9e5e-615b71be6ab4\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb" Nov 03 02:34:40.578532 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:40.578348 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e3a3bd1-724d-454c-9e5e-615b71be6ab4-istio-token\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb\" (UID: \"2e3a3bd1-724d-454c-9e5e-615b71be6ab4\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb" Nov 03 02:34:40.578532 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:40.578435 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5l6k\" (UniqueName: \"kubernetes.io/projected/2e3a3bd1-724d-454c-9e5e-615b71be6ab4-kube-api-access-z5l6k\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb\" (UID: \"2e3a3bd1-724d-454c-9e5e-615b71be6ab4\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb" Nov 03 02:34:40.578532 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:40.578515 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e3a3bd1-724d-454c-9e5e-615b71be6ab4-istio-data\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb\" (UID: \"2e3a3bd1-724d-454c-9e5e-615b71be6ab4\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb" Nov 03 02:34:40.578645 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:40.578566 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e3a3bd1-724d-454c-9e5e-615b71be6ab4-workload-socket\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb\" (UID: \"2e3a3bd1-724d-454c-9e5e-615b71be6ab4\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb" Nov 03 02:34:40.578737 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:40.578706 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e3a3bd1-724d-454c-9e5e-615b71be6ab4-istio-podinfo\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb\" (UID: \"2e3a3bd1-724d-454c-9e5e-615b71be6ab4\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb" Nov 03 02:34:40.578793 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:40.578765 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e3a3bd1-724d-454c-9e5e-615b71be6ab4-workload-certs\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb\" (UID: \"2e3a3bd1-724d-454c-9e5e-615b71be6ab4\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb" Nov 03 02:34:40.578901 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:40.578876 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e3a3bd1-724d-454c-9e5e-615b71be6ab4-credential-socket\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb\" (UID: \"2e3a3bd1-724d-454c-9e5e-615b71be6ab4\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb" Nov 03 02:34:40.578985 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:40.578968 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e3a3bd1-724d-454c-9e5e-615b71be6ab4-istio-envoy\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb\" (UID: \"2e3a3bd1-724d-454c-9e5e-615b71be6ab4\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb" Nov 03 02:34:40.680007 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:40.679958 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e3a3bd1-724d-454c-9e5e-615b71be6ab4-istio-podinfo\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb\" (UID: \"2e3a3bd1-724d-454c-9e5e-615b71be6ab4\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb" Nov 03 02:34:40.680007 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:40.680002 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e3a3bd1-724d-454c-9e5e-615b71be6ab4-workload-certs\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb\" (UID: \"2e3a3bd1-724d-454c-9e5e-615b71be6ab4\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb" Nov 03 02:34:40.680285 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:40.680045 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e3a3bd1-724d-454c-9e5e-615b71be6ab4-credential-socket\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb\" (UID: \"2e3a3bd1-724d-454c-9e5e-615b71be6ab4\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb" Nov 03 02:34:40.680285 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:40.680083 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e3a3bd1-724d-454c-9e5e-615b71be6ab4-istio-envoy\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb\" (UID: \"2e3a3bd1-724d-454c-9e5e-615b71be6ab4\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb" Nov 03 02:34:40.680285 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:40.680156 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e3a3bd1-724d-454c-9e5e-615b71be6ab4-istiod-ca-cert\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb\" (UID: \"2e3a3bd1-724d-454c-9e5e-615b71be6ab4\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb" Nov 03 02:34:40.680285 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:40.680198 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e3a3bd1-724d-454c-9e5e-615b71be6ab4-istio-token\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb\" (UID: \"2e3a3bd1-724d-454c-9e5e-615b71be6ab4\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb" Nov 03 02:34:40.680285 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:40.680243 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5l6k\" (UniqueName: \"kubernetes.io/projected/2e3a3bd1-724d-454c-9e5e-615b71be6ab4-kube-api-access-z5l6k\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb\" (UID: \"2e3a3bd1-724d-454c-9e5e-615b71be6ab4\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb" Nov 03 02:34:40.680642 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:40.680559 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e3a3bd1-724d-454c-9e5e-615b71be6ab4-workload-certs\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb\" (UID: \"2e3a3bd1-724d-454c-9e5e-615b71be6ab4\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb" Nov 03 02:34:40.680642 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:40.680278 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e3a3bd1-724d-454c-9e5e-615b71be6ab4-istio-data\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb\" (UID: \"2e3a3bd1-724d-454c-9e5e-615b71be6ab4\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb" Nov 03 02:34:40.680754 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:40.680714 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e3a3bd1-724d-454c-9e5e-615b71be6ab4-workload-socket\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb\" (UID: \"2e3a3bd1-724d-454c-9e5e-615b71be6ab4\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb" Nov 03 02:34:40.680951 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:40.680921 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e3a3bd1-724d-454c-9e5e-615b71be6ab4-istio-data\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb\" (UID: \"2e3a3bd1-724d-454c-9e5e-615b71be6ab4\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb" Nov 03 02:34:40.681080 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:40.681059 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e3a3bd1-724d-454c-9e5e-615b71be6ab4-workload-socket\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb\" (UID: \"2e3a3bd1-724d-454c-9e5e-615b71be6ab4\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb" Nov 03 02:34:40.681148 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:40.681058 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e3a3bd1-724d-454c-9e5e-615b71be6ab4-istiod-ca-cert\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb\" (UID: \"2e3a3bd1-724d-454c-9e5e-615b71be6ab4\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb" Nov 03 02:34:40.681208 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:40.681157 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e3a3bd1-724d-454c-9e5e-615b71be6ab4-credential-socket\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb\" (UID: \"2e3a3bd1-724d-454c-9e5e-615b71be6ab4\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb" Nov 03 02:34:40.682599 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:40.682570 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e3a3bd1-724d-454c-9e5e-615b71be6ab4-istio-envoy\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb\" (UID: \"2e3a3bd1-724d-454c-9e5e-615b71be6ab4\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb" Nov 03 02:34:40.682833 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:40.682811 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e3a3bd1-724d-454c-9e5e-615b71be6ab4-istio-podinfo\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb\" (UID: \"2e3a3bd1-724d-454c-9e5e-615b71be6ab4\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb" Nov 03 02:34:40.734194 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:40.734118 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e3a3bd1-724d-454c-9e5e-615b71be6ab4-istio-token\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb\" (UID: \"2e3a3bd1-724d-454c-9e5e-615b71be6ab4\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb" Nov 03 02:34:40.734194 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:40.734185 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5l6k\" (UniqueName: \"kubernetes.io/projected/2e3a3bd1-724d-454c-9e5e-615b71be6ab4-kube-api-access-z5l6k\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb\" (UID: \"2e3a3bd1-724d-454c-9e5e-615b71be6ab4\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb" Nov 03 02:34:40.773233 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:40.773195 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb" Nov 03 02:34:41.206482 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:41.206392 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb"] Nov 03 02:34:41.208991 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:34:41.208961 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e3a3bd1_724d_454c_9e5e_615b71be6ab4.slice/crio-ae549b6b84c9129da7277df47b4495170f03ae5f6da64e2bfae555ec4b70462a WatchSource:0}: Error finding container ae549b6b84c9129da7277df47b4495170f03ae5f6da64e2bfae555ec4b70462a: Status 404 returned error can't find the container with id ae549b6b84c9129da7277df47b4495170f03ae5f6da64e2bfae555ec4b70462a Nov 03 02:34:41.211747 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:41.211708 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:34:41.211828 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:41.211787 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:34:41.211874 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:41.211823 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:34:41.626415 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:41.626346 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb" event={"ID":"2e3a3bd1-724d-454c-9e5e-615b71be6ab4","Type":"ContainerStarted","Data":"a1825dc55be22a1bf31e31e56f7d8d3e130f679efd8f44aebc0c230faa5da06c"} Nov 03 02:34:41.626819 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:41.626430 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb" event={"ID":"2e3a3bd1-724d-454c-9e5e-615b71be6ab4","Type":"ContainerStarted","Data":"ae549b6b84c9129da7277df47b4495170f03ae5f6da64e2bfae555ec4b70462a"} Nov 03 02:34:41.774277 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:41.774242 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb" Nov 03 02:34:41.774670 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:41.774622 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb" podStartSLOduration=1.774609532 podStartE2EDuration="1.774609532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:34:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:34:41.734872083 +0000 UTC m=+2558.994070227" watchObservedRunningTime="2026-11-03 02:34:41.774609532 +0000 UTC m=+2559.033807685" Nov 03 02:34:42.778382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:42.778293 5605 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Nov 03 02:34:42.778946 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:42.778392 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb" podUID="2e3a3bd1-724d-454c-9e5e-615b71be6ab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Nov 03 02:34:43.778382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:43.777864 5605 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Nov 03 02:34:43.778382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:43.777926 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb" podUID="2e3a3bd1-724d-454c-9e5e-615b71be6ab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Nov 03 02:34:44.777791 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:44.777752 5605 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Nov 03 02:34:44.778072 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:44.778039 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb" podUID="2e3a3bd1-724d-454c-9e5e-615b71be6ab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Nov 03 02:34:45.776383 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:45.774856 5605 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Nov 03 02:34:45.776383 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:45.774929 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb" podUID="2e3a3bd1-724d-454c-9e5e-615b71be6ab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Nov 03 02:34:46.775381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:46.774870 5605 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Nov 03 02:34:46.775381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:46.774934 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb" podUID="2e3a3bd1-724d-454c-9e5e-615b71be6ab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Nov 03 02:34:47.781021 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:47.780991 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb" Nov 03 02:34:47.781678 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:47.781626 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb" Nov 03 02:34:47.783302 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:47.783280 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb" Nov 03 02:34:48.049185 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:48.049151 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1db7f710-z7ksr"] Nov 03 02:34:48.055651 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:48.055622 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1db7f710-z7ksr"] Nov 03 02:34:48.850709 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:48.850669 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d2a3dba-562a-4ee3-b9a1-d6845a4d93b3" path="/var/lib/kubelet/pods/2d2a3dba-562a-4ee3-b9a1-d6845a4d93b3/volumes" Nov 03 02:34:49.046617 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:49.046576 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7a55cf2-pb6hs"] Nov 03 02:34:49.049994 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:49.049960 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7a55cf2-pb6hs"] Nov 03 02:34:50.850334 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:50.850295 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2cd0a0a-e029-426f-b25a-8fdb92bd10ea" path="/var/lib/kubelet/pods/c2cd0a0a-e029-426f-b25a-8fdb92bd10ea/volumes" Nov 03 02:34:53.923144 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:34:53.923097 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:34:57.736929 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:34:57.736885 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:35:01.093536 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:01.093502 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-78b5c5-58dd96d55c-5jz2j"] Nov 03 02:35:01.109280 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:01.109242 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-78b5c5-58dd96d55c-5jz2j" Nov 03 02:35:01.132024 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:01.131991 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-78b5c5-58dd96d55c-5jz2j"] Nov 03 02:35:01.172902 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:01.172859 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/754ba27d-cd77-4f65-a165-bb6f79b49664-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-78b5c5-58dd96d55c-5jz2j\" (UID: \"754ba27d-cd77-4f65-a165-bb6f79b49664\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-78b5c5-58dd96d55c-5jz2j" Nov 03 02:35:01.173246 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:01.173208 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfhdf\" (UniqueName: \"kubernetes.io/projected/754ba27d-cd77-4f65-a165-bb6f79b49664-kube-api-access-rfhdf\") pod \"payload-processing-e2e-shared-b-w1-78b5c5-58dd96d55c-5jz2j\" (UID: \"754ba27d-cd77-4f65-a165-bb6f79b49664\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-78b5c5-58dd96d55c-5jz2j" Nov 03 02:35:01.177396 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:01.174818 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-78b5c5\"" Nov 03 02:35:01.177396 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:01.174878 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-78b5c5-dockercfg-6pxlq\"" Nov 03 02:35:01.274206 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:01.274171 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfhdf\" (UniqueName: \"kubernetes.io/projected/754ba27d-cd77-4f65-a165-bb6f79b49664-kube-api-access-rfhdf\") pod \"payload-processing-e2e-shared-b-w1-78b5c5-58dd96d55c-5jz2j\" (UID: \"754ba27d-cd77-4f65-a165-bb6f79b49664\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-78b5c5-58dd96d55c-5jz2j" Nov 03 02:35:01.274206 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:01.274217 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/754ba27d-cd77-4f65-a165-bb6f79b49664-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-78b5c5-58dd96d55c-5jz2j\" (UID: \"754ba27d-cd77-4f65-a165-bb6f79b49664\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-78b5c5-58dd96d55c-5jz2j" Nov 03 02:35:01.274995 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:01.274974 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/754ba27d-cd77-4f65-a165-bb6f79b49664-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-78b5c5-58dd96d55c-5jz2j\" (UID: \"754ba27d-cd77-4f65-a165-bb6f79b49664\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-78b5c5-58dd96d55c-5jz2j" Nov 03 02:35:01.328758 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:01.328729 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfhdf\" (UniqueName: \"kubernetes.io/projected/754ba27d-cd77-4f65-a165-bb6f79b49664-kube-api-access-rfhdf\") pod \"payload-processing-e2e-shared-b-w1-78b5c5-58dd96d55c-5jz2j\" (UID: \"754ba27d-cd77-4f65-a165-bb6f79b49664\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-78b5c5-58dd96d55c-5jz2j" Nov 03 02:35:01.425024 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:01.424937 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-78b5c5-58dd96d55c-5jz2j" Nov 03 02:35:01.795440 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:01.795337 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-78b5c5-58dd96d55c-5jz2j"] Nov 03 02:35:01.800591 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:35:01.800558 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod754ba27d_cd77_4f65_a165_bb6f79b49664.slice/crio-0d2f38bb3720741095e8b9bfb6406a71350eb0d9dbd4c70a95b8b12274895f1f WatchSource:0}: Error finding container 0d2f38bb3720741095e8b9bfb6406a71350eb0d9dbd4c70a95b8b12274895f1f: Status 404 returned error can't find the container with id 0d2f38bb3720741095e8b9bfb6406a71350eb0d9dbd4c70a95b8b12274895f1f Nov 03 02:35:02.790433 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:02.790392 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-78b5c5-58dd96d55c-5jz2j" event={"ID":"754ba27d-cd77-4f65-a165-bb6f79b49664","Type":"ContainerStarted","Data":"4d38a44a7076ec4c8b85134bab414ba905ad81e0f639a1bdd3ff20265224513e"} Nov 03 02:35:02.790901 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:02.790454 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-78b5c5-58dd96d55c-5jz2j" event={"ID":"754ba27d-cd77-4f65-a165-bb6f79b49664","Type":"ContainerStarted","Data":"0d2f38bb3720741095e8b9bfb6406a71350eb0d9dbd4c70a95b8b12274895f1f"} Nov 03 02:35:02.790901 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:02.790605 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-78b5c5-58dd96d55c-5jz2j" Nov 03 02:35:02.898768 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:02.898710 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-78b5c5-58dd96d55c-5jz2j" podStartSLOduration=1.898690891 podStartE2EDuration="1.898690891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:35:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:35:02.895730376 +0000 UTC m=+2580.154928530" watchObservedRunningTime="2026-11-03 02:35:02.898690891 +0000 UTC m=+2580.157889048" Nov 03 02:35:10.673411 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:10.673375 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm"] Nov 03 02:35:10.677461 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:10.677440 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm" Nov 03 02:35:10.720521 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:10.720491 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45"] Nov 03 02:35:10.724402 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:10.724377 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45" Nov 03 02:35:10.775101 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:10.775071 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm"] Nov 03 02:35:10.794972 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:10.794927 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4-istio-token\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm\" (UID: \"dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm" Nov 03 02:35:10.795152 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:10.794995 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4-istio-data\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm\" (UID: \"dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm" Nov 03 02:35:10.795152 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:10.795110 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4-workload-socket\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm\" (UID: \"dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm" Nov 03 02:35:10.795279 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:10.795170 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4-credential-socket\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm\" (UID: \"dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm" Nov 03 02:35:10.795279 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:10.795194 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4-workload-certs\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm\" (UID: \"dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm" Nov 03 02:35:10.795279 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:10.795209 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgvxz\" (UniqueName: \"kubernetes.io/projected/dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4-kube-api-access-jgvxz\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm\" (UID: \"dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm" Nov 03 02:35:10.795474 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:10.795303 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4-istiod-ca-cert\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm\" (UID: \"dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm" Nov 03 02:35:10.795474 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:10.795322 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4-istio-podinfo\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm\" (UID: \"dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm" Nov 03 02:35:10.795474 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:10.795398 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4-istio-envoy\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm\" (UID: \"dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm" Nov 03 02:35:10.871776 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:10.871740 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45"] Nov 03 02:35:10.897254 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:10.897215 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed134ef4-d99b-4f5d-abf4-3b04723c2d99-credential-socket\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45\" (UID: \"ed134ef4-d99b-4f5d-abf4-3b04723c2d99\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45" Nov 03 02:35:10.897458 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:10.897262 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed134ef4-d99b-4f5d-abf4-3b04723c2d99-istio-podinfo\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45\" (UID: \"ed134ef4-d99b-4f5d-abf4-3b04723c2d99\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45" Nov 03 02:35:10.897458 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:10.897337 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed134ef4-d99b-4f5d-abf4-3b04723c2d99-workload-certs\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45\" (UID: \"ed134ef4-d99b-4f5d-abf4-3b04723c2d99\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45" Nov 03 02:35:10.897458 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:10.897413 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed134ef4-d99b-4f5d-abf4-3b04723c2d99-istio-data\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45\" (UID: \"ed134ef4-d99b-4f5d-abf4-3b04723c2d99\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45" Nov 03 02:35:10.897653 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:10.897508 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4-istiod-ca-cert\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm\" (UID: \"dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm" Nov 03 02:35:10.897653 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:10.897542 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4-istio-podinfo\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm\" (UID: \"dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm" Nov 03 02:35:10.897653 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:10.897580 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed134ef4-d99b-4f5d-abf4-3b04723c2d99-istiod-ca-cert\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45\" (UID: \"ed134ef4-d99b-4f5d-abf4-3b04723c2d99\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45" Nov 03 02:35:10.897653 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:10.897612 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4-istio-envoy\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm\" (UID: \"dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm" Nov 03 02:35:10.897870 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:10.897697 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjhrh\" (UniqueName: \"kubernetes.io/projected/ed134ef4-d99b-4f5d-abf4-3b04723c2d99-kube-api-access-mjhrh\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45\" (UID: \"ed134ef4-d99b-4f5d-abf4-3b04723c2d99\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45" Nov 03 02:35:10.897870 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:10.897813 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed134ef4-d99b-4f5d-abf4-3b04723c2d99-workload-socket\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45\" (UID: \"ed134ef4-d99b-4f5d-abf4-3b04723c2d99\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45" Nov 03 02:35:10.897870 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:10.897856 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4-istio-token\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm\" (UID: \"dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm" Nov 03 02:35:10.898032 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:10.897898 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed134ef4-d99b-4f5d-abf4-3b04723c2d99-istio-envoy\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45\" (UID: \"ed134ef4-d99b-4f5d-abf4-3b04723c2d99\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45" Nov 03 02:35:10.898032 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:10.898012 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4-istio-data\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm\" (UID: \"dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm" Nov 03 02:35:10.898157 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:10.898117 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4-workload-socket\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm\" (UID: \"dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm" Nov 03 02:35:10.898215 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:10.898186 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed134ef4-d99b-4f5d-abf4-3b04723c2d99-istio-token\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45\" (UID: \"ed134ef4-d99b-4f5d-abf4-3b04723c2d99\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45" Nov 03 02:35:10.898270 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:10.898254 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4-credential-socket\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm\" (UID: \"dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm" Nov 03 02:35:10.898339 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:10.898278 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4-istiod-ca-cert\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm\" (UID: \"dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm" Nov 03 02:35:10.898339 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:10.898286 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4-workload-certs\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm\" (UID: \"dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm" Nov 03 02:35:10.898475 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:10.898334 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgvxz\" (UniqueName: \"kubernetes.io/projected/dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4-kube-api-access-jgvxz\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm\" (UID: \"dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm" Nov 03 02:35:10.898475 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:10.898460 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4-workload-socket\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm\" (UID: \"dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm" Nov 03 02:35:10.898731 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:10.898711 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4-istio-data\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm\" (UID: \"dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm" Nov 03 02:35:10.898854 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:10.898792 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4-workload-certs\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm\" (UID: \"dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm" Nov 03 02:35:10.898916 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:10.898858 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4-credential-socket\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm\" (UID: \"dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm" Nov 03 02:35:10.900575 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:10.900548 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4-istio-envoy\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm\" (UID: \"dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm" Nov 03 02:35:10.901204 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:10.901183 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4-istio-podinfo\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm\" (UID: \"dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm" Nov 03 02:35:10.951570 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:10.951504 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4-istio-token\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm\" (UID: \"dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm" Nov 03 02:35:10.953784 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:10.953759 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgvxz\" (UniqueName: \"kubernetes.io/projected/dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4-kube-api-access-jgvxz\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm\" (UID: \"dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm" Nov 03 02:35:10.990033 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:10.990013 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm" Nov 03 02:35:10.999445 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:10.999410 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed134ef4-d99b-4f5d-abf4-3b04723c2d99-istio-token\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45\" (UID: \"ed134ef4-d99b-4f5d-abf4-3b04723c2d99\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45" Nov 03 02:35:10.999548 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:10.999475 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed134ef4-d99b-4f5d-abf4-3b04723c2d99-credential-socket\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45\" (UID: \"ed134ef4-d99b-4f5d-abf4-3b04723c2d99\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45" Nov 03 02:35:10.999548 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:10.999495 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed134ef4-d99b-4f5d-abf4-3b04723c2d99-istio-podinfo\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45\" (UID: \"ed134ef4-d99b-4f5d-abf4-3b04723c2d99\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45" Nov 03 02:35:10.999647 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:10.999549 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed134ef4-d99b-4f5d-abf4-3b04723c2d99-workload-certs\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45\" (UID: \"ed134ef4-d99b-4f5d-abf4-3b04723c2d99\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45" Nov 03 02:35:10.999647 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:10.999579 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed134ef4-d99b-4f5d-abf4-3b04723c2d99-istio-data\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45\" (UID: \"ed134ef4-d99b-4f5d-abf4-3b04723c2d99\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45" Nov 03 02:35:11.000476 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:10.999647 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed134ef4-d99b-4f5d-abf4-3b04723c2d99-istiod-ca-cert\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45\" (UID: \"ed134ef4-d99b-4f5d-abf4-3b04723c2d99\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45" Nov 03 02:35:11.000476 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:10.999769 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjhrh\" (UniqueName: \"kubernetes.io/projected/ed134ef4-d99b-4f5d-abf4-3b04723c2d99-kube-api-access-mjhrh\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45\" (UID: \"ed134ef4-d99b-4f5d-abf4-3b04723c2d99\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45" Nov 03 02:35:11.000476 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:10.999861 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed134ef4-d99b-4f5d-abf4-3b04723c2d99-credential-socket\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45\" (UID: \"ed134ef4-d99b-4f5d-abf4-3b04723c2d99\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45" Nov 03 02:35:11.000476 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:10.999865 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed134ef4-d99b-4f5d-abf4-3b04723c2d99-workload-socket\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45\" (UID: \"ed134ef4-d99b-4f5d-abf4-3b04723c2d99\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45" Nov 03 02:35:11.000476 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:10.999934 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed134ef4-d99b-4f5d-abf4-3b04723c2d99-workload-certs\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45\" (UID: \"ed134ef4-d99b-4f5d-abf4-3b04723c2d99\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45" Nov 03 02:35:11.000476 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:10.999954 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed134ef4-d99b-4f5d-abf4-3b04723c2d99-istio-envoy\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45\" (UID: \"ed134ef4-d99b-4f5d-abf4-3b04723c2d99\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45" Nov 03 02:35:11.000476 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:11.000319 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed134ef4-d99b-4f5d-abf4-3b04723c2d99-istiod-ca-cert\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45\" (UID: \"ed134ef4-d99b-4f5d-abf4-3b04723c2d99\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45" Nov 03 02:35:11.000912 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:11.000650 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed134ef4-d99b-4f5d-abf4-3b04723c2d99-istio-data\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45\" (UID: \"ed134ef4-d99b-4f5d-abf4-3b04723c2d99\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45" Nov 03 02:35:11.000912 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:11.000706 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed134ef4-d99b-4f5d-abf4-3b04723c2d99-workload-socket\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45\" (UID: \"ed134ef4-d99b-4f5d-abf4-3b04723c2d99\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45" Nov 03 02:35:11.002215 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:11.002191 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed134ef4-d99b-4f5d-abf4-3b04723c2d99-istio-podinfo\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45\" (UID: \"ed134ef4-d99b-4f5d-abf4-3b04723c2d99\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45" Nov 03 02:35:11.002462 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:11.002443 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed134ef4-d99b-4f5d-abf4-3b04723c2d99-istio-envoy\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45\" (UID: \"ed134ef4-d99b-4f5d-abf4-3b04723c2d99\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45" Nov 03 02:35:11.066182 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:11.057403 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjhrh\" (UniqueName: \"kubernetes.io/projected/ed134ef4-d99b-4f5d-abf4-3b04723c2d99-kube-api-access-mjhrh\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45\" (UID: \"ed134ef4-d99b-4f5d-abf4-3b04723c2d99\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45" Nov 03 02:35:11.066182 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:11.061412 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed134ef4-d99b-4f5d-abf4-3b04723c2d99-istio-token\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45\" (UID: \"ed134ef4-d99b-4f5d-abf4-3b04723c2d99\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45" Nov 03 02:35:11.337775 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:11.337741 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45" Nov 03 02:35:11.419794 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:11.419692 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm"] Nov 03 02:35:11.422328 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:35:11.422299 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcc23e04_d38a_4ca6_a3b2_67aa5495a9b4.slice/crio-a8a6822d64209b19c2f4b3d90937469c30a73f29f7111a28c80d7ad5caa3e068 WatchSource:0}: Error finding container a8a6822d64209b19c2f4b3d90937469c30a73f29f7111a28c80d7ad5caa3e068: Status 404 returned error can't find the container with id a8a6822d64209b19c2f4b3d90937469c30a73f29f7111a28c80d7ad5caa3e068 Nov 03 02:35:11.425825 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:11.425784 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:35:11.425941 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:11.425875 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:35:11.425941 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:11.425928 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:35:11.711929 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:11.711828 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45"] Nov 03 02:35:11.717549 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:35:11.717511 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded134ef4_d99b_4f5d_abf4_3b04723c2d99.slice/crio-c826ef085227cb2f5b4131b063692c7be34278d27e13b9374d6908ba615c4244 WatchSource:0}: Error finding container c826ef085227cb2f5b4131b063692c7be34278d27e13b9374d6908ba615c4244: Status 404 returned error can't find the container with id c826ef085227cb2f5b4131b063692c7be34278d27e13b9374d6908ba615c4244 Nov 03 02:35:11.720121 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:11.720084 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:35:11.720231 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:11.720167 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:35:11.720231 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:11.720212 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:35:11.881011 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:11.880971 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm" event={"ID":"dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4","Type":"ContainerStarted","Data":"b6c9856b6582019ea7bf17dc916bb9619be30a2c1c0269bdc75343d7e83f5516"} Nov 03 02:35:11.881347 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:11.881324 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm" event={"ID":"dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4","Type":"ContainerStarted","Data":"a8a6822d64209b19c2f4b3d90937469c30a73f29f7111a28c80d7ad5caa3e068"} Nov 03 02:35:11.887121 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:11.887086 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45" event={"ID":"ed134ef4-d99b-4f5d-abf4-3b04723c2d99","Type":"ContainerStarted","Data":"5341ae2a71e228242acc3c5f105558ae8cdcbc5216497959a44b153cdd416170"} Nov 03 02:35:11.887309 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:11.887293 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45" event={"ID":"ed134ef4-d99b-4f5d-abf4-3b04723c2d99","Type":"ContainerStarted","Data":"c826ef085227cb2f5b4131b063692c7be34278d27e13b9374d6908ba615c4244"} Nov 03 02:35:11.999546 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:11.999461 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm" Nov 03 02:35:12.000063 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:12.000009 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm" podStartSLOduration=1.999992647 podStartE2EDuration="1.999992647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:35:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:35:11.994318208 +0000 UTC m=+2589.253516378" watchObservedRunningTime="2026-11-03 02:35:11.999992647 +0000 UTC m=+2589.259190802" Nov 03 02:35:12.253986 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:12.253855 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45" podStartSLOduration=2.253831407 podStartE2EDuration="2.253831407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:35:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:35:12.201856666 +0000 UTC m=+2589.461054821" watchObservedRunningTime="2026-11-03 02:35:12.253831407 +0000 UTC m=+2589.513029562" Nov 03 02:35:12.338759 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:12.338724 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45" Nov 03 02:35:12.340184 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:12.340158 5605 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 03 02:35:12.340308 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:12.340211 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45" podUID="ed134ef4-d99b-4f5d-abf4-3b04723c2d99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 03 02:35:12.992389 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:12.992070 5605 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Nov 03 02:35:12.992389 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:12.992137 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm" podUID="dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Nov 03 02:35:13.338791 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:13.338756 5605 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 03 02:35:13.338966 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:13.338808 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45" podUID="ed134ef4-d99b-4f5d-abf4-3b04723c2d99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 03 02:35:13.807378 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:13.807333 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-78b5c5-58dd96d55c-5jz2j" Nov 03 02:35:13.994381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:13.993715 5605 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Nov 03 02:35:13.994381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:13.993776 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm" podUID="dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Nov 03 02:35:14.338679 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:14.338639 5605 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 03 02:35:14.338897 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:14.338701 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45" podUID="ed134ef4-d99b-4f5d-abf4-3b04723c2d99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 03 02:35:14.996389 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:14.992715 5605 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Nov 03 02:35:14.996389 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:14.992774 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm" podUID="dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Nov 03 02:35:15.338169 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:15.338128 5605 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 03 02:35:15.338332 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:15.338201 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45" podUID="ed134ef4-d99b-4f5d-abf4-3b04723c2d99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 03 02:35:15.993382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:15.992496 5605 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Nov 03 02:35:15.993382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:15.992559 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm" podUID="dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Nov 03 02:35:16.338749 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:16.338717 5605 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 03 02:35:16.339171 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:16.338776 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45" podUID="ed134ef4-d99b-4f5d-abf4-3b04723c2d99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 03 02:35:16.992742 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:16.992704 5605 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Nov 03 02:35:16.993019 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:16.992992 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm" podUID="dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Nov 03 02:35:17.339201 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:17.339036 5605 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 03 02:35:17.339201 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:17.339099 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45" podUID="ed134ef4-d99b-4f5d-abf4-3b04723c2d99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 03 02:35:18.256089 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:18.256052 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm" Nov 03 02:35:18.256769 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:18.256729 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm" Nov 03 02:35:18.258055 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:18.258030 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm" Nov 03 02:35:18.617863 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:18.617821 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45" Nov 03 02:35:18.618760 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:18.618676 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45" Nov 03 02:35:18.619828 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:18.619805 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45" Nov 03 02:35:23.915122 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:35:23.915082 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:35:25.804255 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:25.804220 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc"] Nov 03 02:35:25.809646 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:25.809627 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc" Nov 03 02:35:25.887465 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:25.887423 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a2dba2d-d002-4f4e-ab37-298b6dadc320-istio-envoy\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc\" (UID: \"4a2dba2d-d002-4f4e-ab37-298b6dadc320\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc" Nov 03 02:35:25.887663 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:25.887475 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a2dba2d-d002-4f4e-ab37-298b6dadc320-istio-data\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc\" (UID: \"4a2dba2d-d002-4f4e-ab37-298b6dadc320\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc" Nov 03 02:35:25.887663 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:25.887522 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a2dba2d-d002-4f4e-ab37-298b6dadc320-istio-podinfo\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc\" (UID: \"4a2dba2d-d002-4f4e-ab37-298b6dadc320\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc" Nov 03 02:35:25.887663 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:25.887612 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a2dba2d-d002-4f4e-ab37-298b6dadc320-workload-certs\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc\" (UID: \"4a2dba2d-d002-4f4e-ab37-298b6dadc320\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc" Nov 03 02:35:25.887663 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:25.887645 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a2dba2d-d002-4f4e-ab37-298b6dadc320-istiod-ca-cert\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc\" (UID: \"4a2dba2d-d002-4f4e-ab37-298b6dadc320\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc" Nov 03 02:35:25.887890 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:25.887767 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a2dba2d-d002-4f4e-ab37-298b6dadc320-workload-socket\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc\" (UID: \"4a2dba2d-d002-4f4e-ab37-298b6dadc320\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc" Nov 03 02:35:25.887890 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:25.887804 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sknjj\" (UniqueName: \"kubernetes.io/projected/4a2dba2d-d002-4f4e-ab37-298b6dadc320-kube-api-access-sknjj\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc\" (UID: \"4a2dba2d-d002-4f4e-ab37-298b6dadc320\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc" Nov 03 02:35:25.887890 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:25.887844 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a2dba2d-d002-4f4e-ab37-298b6dadc320-credential-socket\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc\" (UID: \"4a2dba2d-d002-4f4e-ab37-298b6dadc320\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc" Nov 03 02:35:25.887890 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:25.887865 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a2dba2d-d002-4f4e-ab37-298b6dadc320-istio-token\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc\" (UID: \"4a2dba2d-d002-4f4e-ab37-298b6dadc320\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc" Nov 03 02:35:25.905967 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:25.905917 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc"] Nov 03 02:35:25.989163 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:25.989119 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a2dba2d-d002-4f4e-ab37-298b6dadc320-workload-socket\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc\" (UID: \"4a2dba2d-d002-4f4e-ab37-298b6dadc320\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc" Nov 03 02:35:25.989391 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:25.989178 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sknjj\" (UniqueName: \"kubernetes.io/projected/4a2dba2d-d002-4f4e-ab37-298b6dadc320-kube-api-access-sknjj\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc\" (UID: \"4a2dba2d-d002-4f4e-ab37-298b6dadc320\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc" Nov 03 02:35:25.989391 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:25.989232 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a2dba2d-d002-4f4e-ab37-298b6dadc320-credential-socket\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc\" (UID: \"4a2dba2d-d002-4f4e-ab37-298b6dadc320\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc" Nov 03 02:35:25.989391 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:25.989257 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a2dba2d-d002-4f4e-ab37-298b6dadc320-istio-token\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc\" (UID: \"4a2dba2d-d002-4f4e-ab37-298b6dadc320\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc" Nov 03 02:35:25.989596 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:25.989402 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a2dba2d-d002-4f4e-ab37-298b6dadc320-istio-envoy\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc\" (UID: \"4a2dba2d-d002-4f4e-ab37-298b6dadc320\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc" Nov 03 02:35:25.989596 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:25.989428 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a2dba2d-d002-4f4e-ab37-298b6dadc320-istio-data\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc\" (UID: \"4a2dba2d-d002-4f4e-ab37-298b6dadc320\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc" Nov 03 02:35:25.989596 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:25.989464 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a2dba2d-d002-4f4e-ab37-298b6dadc320-istio-podinfo\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc\" (UID: \"4a2dba2d-d002-4f4e-ab37-298b6dadc320\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc" Nov 03 02:35:25.989596 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:25.989535 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a2dba2d-d002-4f4e-ab37-298b6dadc320-workload-certs\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc\" (UID: \"4a2dba2d-d002-4f4e-ab37-298b6dadc320\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc" Nov 03 02:35:25.989596 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:25.989561 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a2dba2d-d002-4f4e-ab37-298b6dadc320-istiod-ca-cert\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc\" (UID: \"4a2dba2d-d002-4f4e-ab37-298b6dadc320\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc" Nov 03 02:35:25.989827 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:25.989619 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a2dba2d-d002-4f4e-ab37-298b6dadc320-workload-socket\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc\" (UID: \"4a2dba2d-d002-4f4e-ab37-298b6dadc320\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc" Nov 03 02:35:25.989827 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:25.989663 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a2dba2d-d002-4f4e-ab37-298b6dadc320-credential-socket\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc\" (UID: \"4a2dba2d-d002-4f4e-ab37-298b6dadc320\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc" Nov 03 02:35:25.990116 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:25.990092 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a2dba2d-d002-4f4e-ab37-298b6dadc320-workload-certs\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc\" (UID: \"4a2dba2d-d002-4f4e-ab37-298b6dadc320\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc" Nov 03 02:35:25.990298 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:25.990107 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a2dba2d-d002-4f4e-ab37-298b6dadc320-istio-data\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc\" (UID: \"4a2dba2d-d002-4f4e-ab37-298b6dadc320\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc" Nov 03 02:35:25.990446 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:25.990345 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a2dba2d-d002-4f4e-ab37-298b6dadc320-istiod-ca-cert\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc\" (UID: \"4a2dba2d-d002-4f4e-ab37-298b6dadc320\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc" Nov 03 02:35:25.991932 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:25.991905 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a2dba2d-d002-4f4e-ab37-298b6dadc320-istio-envoy\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc\" (UID: \"4a2dba2d-d002-4f4e-ab37-298b6dadc320\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc" Nov 03 02:35:25.992139 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:25.992121 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a2dba2d-d002-4f4e-ab37-298b6dadc320-istio-podinfo\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc\" (UID: \"4a2dba2d-d002-4f4e-ab37-298b6dadc320\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc" Nov 03 02:35:26.042585 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:26.042558 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a2dba2d-d002-4f4e-ab37-298b6dadc320-istio-token\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc\" (UID: \"4a2dba2d-d002-4f4e-ab37-298b6dadc320\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc" Nov 03 02:35:26.042745 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:26.042604 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sknjj\" (UniqueName: \"kubernetes.io/projected/4a2dba2d-d002-4f4e-ab37-298b6dadc320-kube-api-access-sknjj\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc\" (UID: \"4a2dba2d-d002-4f4e-ab37-298b6dadc320\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc" Nov 03 02:35:26.140442 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:26.126448 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc" Nov 03 02:35:26.203458 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:26.201879 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-957b33-54c6fcf67c-td8mv"] Nov 03 02:35:26.223082 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:26.223044 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-957b33-54c6fcf67c-td8mv"] Nov 03 02:35:26.223275 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:26.223165 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-957b33-54c6fcf67c-td8mv" Nov 03 02:35:26.273691 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:26.273653 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-957b33\"" Nov 03 02:35:26.273919 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:26.273898 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-957b33-dockercfg-vsvsx\"" Nov 03 02:35:26.299604 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:26.299565 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wbhb\" (UniqueName: \"kubernetes.io/projected/cb7f4ac2-ff63-48b0-ab55-74bf25ffbb6a-kube-api-access-9wbhb\") pod \"payload-processing-e2e-shared-b-w2-957b33-54c6fcf67c-td8mv\" (UID: \"cb7f4ac2-ff63-48b0-ab55-74bf25ffbb6a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-957b33-54c6fcf67c-td8mv" Nov 03 02:35:26.299793 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:26.299689 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb7f4ac2-ff63-48b0-ab55-74bf25ffbb6a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-957b33-54c6fcf67c-td8mv\" (UID: \"cb7f4ac2-ff63-48b0-ab55-74bf25ffbb6a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-957b33-54c6fcf67c-td8mv" Nov 03 02:35:26.400744 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:26.400649 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb7f4ac2-ff63-48b0-ab55-74bf25ffbb6a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-957b33-54c6fcf67c-td8mv\" (UID: \"cb7f4ac2-ff63-48b0-ab55-74bf25ffbb6a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-957b33-54c6fcf67c-td8mv" Nov 03 02:35:26.400917 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:26.400833 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wbhb\" (UniqueName: \"kubernetes.io/projected/cb7f4ac2-ff63-48b0-ab55-74bf25ffbb6a-kube-api-access-9wbhb\") pod \"payload-processing-e2e-shared-b-w2-957b33-54c6fcf67c-td8mv\" (UID: \"cb7f4ac2-ff63-48b0-ab55-74bf25ffbb6a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-957b33-54c6fcf67c-td8mv" Nov 03 02:35:26.401495 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:26.401473 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb7f4ac2-ff63-48b0-ab55-74bf25ffbb6a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-957b33-54c6fcf67c-td8mv\" (UID: \"cb7f4ac2-ff63-48b0-ab55-74bf25ffbb6a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-957b33-54c6fcf67c-td8mv" Nov 03 02:35:26.454626 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:26.454593 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wbhb\" (UniqueName: \"kubernetes.io/projected/cb7f4ac2-ff63-48b0-ab55-74bf25ffbb6a-kube-api-access-9wbhb\") pod \"payload-processing-e2e-shared-b-w2-957b33-54c6fcf67c-td8mv\" (UID: \"cb7f4ac2-ff63-48b0-ab55-74bf25ffbb6a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-957b33-54c6fcf67c-td8mv" Nov 03 02:35:26.545080 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:26.545053 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-957b33-54c6fcf67c-td8mv" Nov 03 02:35:26.623629 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:26.622609 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc"] Nov 03 02:35:26.628174 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:35:26.628140 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a2dba2d_d002_4f4e_ab37_298b6dadc320.slice/crio-a586e3ec1d9e39cff747925eca283d7507b8b47727f0ce9f0aa16e7e6c2a08ed WatchSource:0}: Error finding container a586e3ec1d9e39cff747925eca283d7507b8b47727f0ce9f0aa16e7e6c2a08ed: Status 404 returned error can't find the container with id a586e3ec1d9e39cff747925eca283d7507b8b47727f0ce9f0aa16e7e6c2a08ed Nov 03 02:35:26.631075 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:26.631041 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:35:26.631175 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:26.631111 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:35:26.631175 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:26.631139 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:35:26.917568 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:26.917487 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-957b33-54c6fcf67c-td8mv"] Nov 03 02:35:26.921226 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:35:26.921202 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb7f4ac2_ff63_48b0_ab55_74bf25ffbb6a.slice/crio-144e6723be0c9f8b70ee7c32971a268f29eddfdc2cb0da04022922b4f35d6c79 WatchSource:0}: Error finding container 144e6723be0c9f8b70ee7c32971a268f29eddfdc2cb0da04022922b4f35d6c79: Status 404 returned error can't find the container with id 144e6723be0c9f8b70ee7c32971a268f29eddfdc2cb0da04022922b4f35d6c79 Nov 03 02:35:27.032603 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:27.032561 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc" event={"ID":"4a2dba2d-d002-4f4e-ab37-298b6dadc320","Type":"ContainerStarted","Data":"50aecd8f9fadd74fbe86eb1b7501f9d14698eceff2b1c005436503b39b1af3c6"} Nov 03 02:35:27.032764 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:27.032625 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc" event={"ID":"4a2dba2d-d002-4f4e-ab37-298b6dadc320","Type":"ContainerStarted","Data":"a586e3ec1d9e39cff747925eca283d7507b8b47727f0ce9f0aa16e7e6c2a08ed"} Nov 03 02:35:27.034596 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:27.034560 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-957b33-54c6fcf67c-td8mv" event={"ID":"cb7f4ac2-ff63-48b0-ab55-74bf25ffbb6a","Type":"ContainerStarted","Data":"7c6c22377f9718b4f4aa017a0b52be07ac5c57f59c30e4a8a3721cb6c6d0f280"} Nov 03 02:35:27.034737 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:27.034619 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-957b33-54c6fcf67c-td8mv" Nov 03 02:35:27.034737 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:27.034636 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-957b33-54c6fcf67c-td8mv" event={"ID":"cb7f4ac2-ff63-48b0-ab55-74bf25ffbb6a","Type":"ContainerStarted","Data":"144e6723be0c9f8b70ee7c32971a268f29eddfdc2cb0da04022922b4f35d6c79"} Nov 03 02:35:27.130715 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:27.130681 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc" Nov 03 02:35:27.132039 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:27.132012 5605 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Nov 03 02:35:27.132177 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:27.132115 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc" podUID="4a2dba2d-d002-4f4e-ab37-298b6dadc320" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Nov 03 02:35:27.146455 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:27.146414 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc" podStartSLOduration=2.14640023 podStartE2EDuration="2.14640023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:35:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:35:27.141251996 +0000 UTC m=+2604.400450150" watchObservedRunningTime="2026-11-03 02:35:27.14640023 +0000 UTC m=+2604.405598383" Nov 03 02:35:27.249003 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:27.248898 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-957b33-54c6fcf67c-td8mv" podStartSLOduration=1.248879696 podStartE2EDuration="1.248879696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:35:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:35:27.246328669 +0000 UTC m=+2604.505526820" watchObservedRunningTime="2026-11-03 02:35:27.248879696 +0000 UTC m=+2604.508077851" Nov 03 02:35:28.131738 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:28.131707 5605 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Nov 03 02:35:28.132144 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:28.131756 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc" podUID="4a2dba2d-d002-4f4e-ab37-298b6dadc320" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Nov 03 02:35:29.131648 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:29.131613 5605 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Nov 03 02:35:29.131830 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:29.131666 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc" podUID="4a2dba2d-d002-4f4e-ab37-298b6dadc320" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Nov 03 02:35:30.131351 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:30.131319 5605 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Nov 03 02:35:30.131556 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:30.131397 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc" podUID="4a2dba2d-d002-4f4e-ab37-298b6dadc320" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Nov 03 02:35:31.131036 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:31.131004 5605 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Nov 03 02:35:31.131467 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:31.131059 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc" podUID="4a2dba2d-d002-4f4e-ab37-298b6dadc320" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Nov 03 02:35:32.131104 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:32.131070 5605 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Nov 03 02:35:32.131594 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:32.131123 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc" podUID="4a2dba2d-d002-4f4e-ab37-298b6dadc320" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Nov 03 02:35:33.135377 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:33.135335 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc" Nov 03 02:35:33.135847 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:33.135823 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc" Nov 03 02:35:33.136549 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:33.136530 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc" Nov 03 02:35:38.044292 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:38.044256 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7ba433a-b5pwq"] Nov 03 02:35:38.045111 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:38.045092 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-957b33-54c6fcf67c-td8mv" Nov 03 02:35:38.091453 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:38.091425 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7ba433a-b5pwq"] Nov 03 02:35:38.850394 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:38.850342 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3807ac44-ce6e-407f-a37f-77f9afb60819" path="/var/lib/kubelet/pods/3807ac44-ce6e-407f-a37f-77f9afb60819/volumes" Nov 03 02:35:49.045532 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:49.045427 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bdc2839-dhzqw"] Nov 03 02:35:49.091800 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:49.091766 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bdc2839-dhzqw"] Nov 03 02:35:50.850000 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:50.849968 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed4768a9-da35-463c-8001-4f8ed903a7cb" path="/var/lib/kubelet/pods/ed4768a9-da35-463c-8001-4f8ed903a7cb/volumes" Nov 03 02:35:53.914636 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:35:53.914597 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:35:56.390593 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:56.390411 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-4616ec-84b7c6449f-xf5bh"] Nov 03 02:35:56.396229 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:56.396200 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-4616ec-84b7c6449f-xf5bh" Nov 03 02:35:56.513414 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:56.475923 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-4616ec-84b7c6449f-xf5bh"] Nov 03 02:35:56.539533 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:56.517182 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrchx\" (UniqueName: \"kubernetes.io/projected/f7535c5c-da75-4fb2-8c06-94b2821a4f8b-kube-api-access-hrchx\") pod \"payload-processing-e2e-praxis-b-4616ec-84b7c6449f-xf5bh\" (UID: \"f7535c5c-da75-4fb2-8c06-94b2821a4f8b\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-4616ec-84b7c6449f-xf5bh" Nov 03 02:35:56.544687 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:56.544656 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7535c5c-da75-4fb2-8c06-94b2821a4f8b-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-4616ec-84b7c6449f-xf5bh\" (UID: \"f7535c5c-da75-4fb2-8c06-94b2821a4f8b\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-4616ec-84b7c6449f-xf5bh" Nov 03 02:35:56.547537 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:56.547507 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-4616ec\"" Nov 03 02:35:56.547737 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:56.547664 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-4616ec-dockercfg-pnqvm\"" Nov 03 02:35:56.571941 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:56.571905 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4616ec-944fbf57f-n5n9d"] Nov 03 02:35:56.576194 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:56.576172 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4616ec-944fbf57f-n5n9d" Nov 03 02:35:56.620069 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:56.620028 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4616ec-944fbf57f-n5n9d"] Nov 03 02:35:56.646386 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:56.646276 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7535c5c-da75-4fb2-8c06-94b2821a4f8b-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-4616ec-84b7c6449f-xf5bh\" (UID: \"f7535c5c-da75-4fb2-8c06-94b2821a4f8b\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-4616ec-84b7c6449f-xf5bh" Nov 03 02:35:56.646578 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:56.646519 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrchx\" (UniqueName: \"kubernetes.io/projected/f7535c5c-da75-4fb2-8c06-94b2821a4f8b-kube-api-access-hrchx\") pod \"payload-processing-e2e-praxis-b-4616ec-84b7c6449f-xf5bh\" (UID: \"f7535c5c-da75-4fb2-8c06-94b2821a4f8b\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-4616ec-84b7c6449f-xf5bh" Nov 03 02:35:56.647141 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:56.647113 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7535c5c-da75-4fb2-8c06-94b2821a4f8b-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-4616ec-84b7c6449f-xf5bh\" (UID: \"f7535c5c-da75-4fb2-8c06-94b2821a4f8b\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-4616ec-84b7c6449f-xf5bh" Nov 03 02:35:56.699988 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:56.699956 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrchx\" (UniqueName: \"kubernetes.io/projected/f7535c5c-da75-4fb2-8c06-94b2821a4f8b-kube-api-access-hrchx\") pod \"payload-processing-e2e-praxis-b-4616ec-84b7c6449f-xf5bh\" (UID: \"f7535c5c-da75-4fb2-8c06-94b2821a4f8b\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-4616ec-84b7c6449f-xf5bh" Nov 03 02:35:56.748467 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:56.748425 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00c96087-f0d3-434e-bceb-658d0f62c4e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-4616ec-944fbf57f-n5n9d\" (UID: \"00c96087-f0d3-434e-bceb-658d0f62c4e3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4616ec-944fbf57f-n5n9d" Nov 03 02:35:56.748644 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:56.748607 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wggzx\" (UniqueName: \"kubernetes.io/projected/00c96087-f0d3-434e-bceb-658d0f62c4e3-kube-api-access-wggzx\") pod \"payload-pre-processing-e2e-praxis-b-4616ec-944fbf57f-n5n9d\" (UID: \"00c96087-f0d3-434e-bceb-658d0f62c4e3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4616ec-944fbf57f-n5n9d" Nov 03 02:35:56.773269 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:56.773237 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-4616ec-84b7c6449f-xf5bh" Nov 03 02:35:56.851453 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:56.851415 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00c96087-f0d3-434e-bceb-658d0f62c4e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-4616ec-944fbf57f-n5n9d\" (UID: \"00c96087-f0d3-434e-bceb-658d0f62c4e3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4616ec-944fbf57f-n5n9d" Nov 03 02:35:56.851613 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:56.851533 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wggzx\" (UniqueName: \"kubernetes.io/projected/00c96087-f0d3-434e-bceb-658d0f62c4e3-kube-api-access-wggzx\") pod \"payload-pre-processing-e2e-praxis-b-4616ec-944fbf57f-n5n9d\" (UID: \"00c96087-f0d3-434e-bceb-658d0f62c4e3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4616ec-944fbf57f-n5n9d" Nov 03 02:35:56.852723 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:56.852698 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00c96087-f0d3-434e-bceb-658d0f62c4e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-4616ec-944fbf57f-n5n9d\" (UID: \"00c96087-f0d3-434e-bceb-658d0f62c4e3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4616ec-944fbf57f-n5n9d" Nov 03 02:35:56.905421 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:56.905310 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wggzx\" (UniqueName: \"kubernetes.io/projected/00c96087-f0d3-434e-bceb-658d0f62c4e3-kube-api-access-wggzx\") pod \"payload-pre-processing-e2e-praxis-b-4616ec-944fbf57f-n5n9d\" (UID: \"00c96087-f0d3-434e-bceb-658d0f62c4e3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4616ec-944fbf57f-n5n9d" Nov 03 02:35:57.166671 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:57.166585 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-4616ec-84b7c6449f-xf5bh"] Nov 03 02:35:57.190857 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:57.190831 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4616ec-944fbf57f-n5n9d" Nov 03 02:35:57.267419 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:57.267351 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-4616ec-84b7c6449f-xf5bh" event={"ID":"f7535c5c-da75-4fb2-8c06-94b2821a4f8b","Type":"ContainerStarted","Data":"e251ddd8ef1b64097ac587c573960d1e51c8e35c5a8a5d3c2b5fc0342a0d74e9"} Nov 03 02:35:57.267571 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:57.267467 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-4616ec-84b7c6449f-xf5bh" event={"ID":"f7535c5c-da75-4fb2-8c06-94b2821a4f8b","Type":"ContainerStarted","Data":"a8f96955b04d5519d6c1e5ced00496062a38b637e5d2b787d8d115fee1e13fe8"} Nov 03 02:35:57.268326 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:57.268304 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-4616ec-84b7c6449f-xf5bh" Nov 03 02:35:57.374327 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:57.374265 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-4616ec-84b7c6449f-xf5bh" podStartSLOduration=1.374251835 podStartE2EDuration="1.374251835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:35:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:35:57.373213 +0000 UTC m=+2634.632411154" watchObservedRunningTime="2026-11-03 02:35:57.374251835 +0000 UTC m=+2634.633449989" Nov 03 02:35:57.568449 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:57.568408 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4616ec-944fbf57f-n5n9d"] Nov 03 02:35:57.572611 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:35:57.572585 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00c96087_f0d3_434e_bceb_658d0f62c4e3.slice/crio-c534b62bc9e8f1d6f317934e7fc8e7eae2673271244d2a891acd5ffefbea4011 WatchSource:0}: Error finding container c534b62bc9e8f1d6f317934e7fc8e7eae2673271244d2a891acd5ffefbea4011: Status 404 returned error can't find the container with id c534b62bc9e8f1d6f317934e7fc8e7eae2673271244d2a891acd5ffefbea4011 Nov 03 02:35:58.047439 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:58.047318 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:35:58.275949 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:58.275910 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4616ec-944fbf57f-n5n9d" event={"ID":"00c96087-f0d3-434e-bceb-658d0f62c4e3","Type":"ContainerStarted","Data":"c15c4314627be745281a349061e5298f190263be5c69f9acb9bf83e6928c1877"} Nov 03 02:35:58.276174 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:58.275970 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4616ec-944fbf57f-n5n9d" event={"ID":"00c96087-f0d3-434e-bceb-658d0f62c4e3","Type":"ContainerStarted","Data":"c534b62bc9e8f1d6f317934e7fc8e7eae2673271244d2a891acd5ffefbea4011"} Nov 03 02:35:58.384130 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:58.384076 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4616ec-944fbf57f-n5n9d" podStartSLOduration=2.384059564 podStartE2EDuration="2.384059564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:35:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:35:58.381841496 +0000 UTC m=+2635.641039652" watchObservedRunningTime="2026-11-03 02:35:58.384059564 +0000 UTC m=+2635.643257719" Nov 03 02:35:59.283775 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:35:59.283734 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4616ec-944fbf57f-n5n9d" Nov 03 02:36:03.793667 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:36:03.793635 5605 scope.go:122] "RemoveContainer" containerID="38a5a3375eddf1d2d3b677b9b5e36be53bea8cc0fc7fd2498336f03c7658c04c" Nov 03 02:36:03.804499 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:36:03.804475 5605 scope.go:122] "RemoveContainer" containerID="c53f58cbde646c8d93835f6ea2e451b0a6a32cf8bfd1d9c4e8cfc45b78f08bf2" Nov 03 02:36:03.813535 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:36:03.813513 5605 scope.go:122] "RemoveContainer" containerID="98dcc5b6508cb9db19f18809b31532088bc00726d8e2ceaa49c41ebea04ebf56" Nov 03 02:36:05.050246 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:36:05.050216 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8566dce2-ttts6"] Nov 03 02:36:05.096652 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:36:05.096624 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8566dce2-ttts6"] Nov 03 02:36:06.850076 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:36:06.850032 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab77c343-4072-41bc-9dac-384db1bd6a81" path="/var/lib/kubelet/pods/ab77c343-4072-41bc-9dac-384db1bd6a81/volumes" Nov 03 02:36:09.286939 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:36:09.286910 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-4616ec-84b7c6449f-xf5bh" Nov 03 02:36:10.292862 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:36:10.292828 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4616ec-944fbf57f-n5n9d" Nov 03 02:36:21.044207 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:36:21.044171 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e4b6450-ddp67"] Nov 03 02:36:21.091139 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:36:21.091101 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e4b6450-ddp67"] Nov 03 02:36:22.850498 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:36:22.850464 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86089afc-62e2-4be9-b8a8-9d4926e68a11" path="/var/lib/kubelet/pods/86089afc-62e2-4be9-b8a8-9d4926e68a11/volumes" Nov 03 02:36:23.913508 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:36:23.913464 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:36:25.981462 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:36:25.981419 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:36:41.044830 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:36:41.044794 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a035c29d-gphwj"] Nov 03 02:36:41.091570 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:36:41.091539 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a035c29d-gphwj"] Nov 03 02:36:42.850642 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:36:42.850614 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8f69569-01e2-4398-9ba7-de111b956ea0" path="/var/lib/kubelet/pods/d8f69569-01e2-4398-9ba7-de111b956ea0/volumes" Nov 03 02:36:44.044741 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:36:44.044708 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-976aed18-vqpgt"] Nov 03 02:36:44.091122 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:36:44.091087 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-976aed18-vqpgt"] Nov 03 02:36:44.851049 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:36:44.851016 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a98cc88-77ca-4387-9d4c-62d907bdf21f" path="/var/lib/kubelet/pods/8a98cc88-77ca-4387-9d4c-62d907bdf21f/volumes" Nov 03 02:36:52.043349 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:36:52.043316 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff928274-bshwh"] Nov 03 02:36:52.090234 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:36:52.090198 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff928274-bshwh"] Nov 03 02:36:52.850621 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:36:52.850590 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e55dcb1-6255-441c-8f77-85b05e14727d" path="/var/lib/kubelet/pods/5e55dcb1-6255-441c-8f77-85b05e14727d/volumes" Nov 03 02:36:53.914432 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:36:53.914398 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:36:55.043069 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:36:55.043033 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae8e330a-lvnwg"] Nov 03 02:36:55.090078 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:36:55.090052 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae8e330a-lvnwg"] Nov 03 02:36:56.850602 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:36:56.850569 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85aa8ff4-f330-45e1-bb65-58ea54bbf0df" path="/var/lib/kubelet/pods/85aa8ff4-f330-45e1-bb65-58ea54bbf0df/volumes" Nov 03 02:37:03.849188 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:37:03.849155 5605 scope.go:122] "RemoveContainer" containerID="72f2ba1030ea9a18bf019b07b1ffc0667976818ab4c60be7d805ca3cb5ad87cc" Nov 03 02:37:03.858487 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:37:03.858462 5605 scope.go:122] "RemoveContainer" containerID="bdf5054506d4bed72e674803dae3525dea06367732baf478c4ac56cb392c73a2" Nov 03 02:37:03.869980 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:37:03.869954 5605 scope.go:122] "RemoveContainer" containerID="e719f55515439e7fec00322737013fb550ecacb400b1df8192200222303a55ab" Nov 03 02:37:05.229799 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:37:05.229763 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-965a07/auth-test-model-965a07-kserve-86544f48df-b4jcp"] Nov 03 02:37:05.235747 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:37:05.235722 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-965a07/auth-test-model-965a07-kserve-86544f48df-b4jcp" Nov 03 02:37:05.286691 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:37:05.286656 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/233b378d-0c02-4452-854f-563f18c1a66e-home\") pod \"auth-test-model-965a07-kserve-86544f48df-b4jcp\" (UID: \"233b378d-0c02-4452-854f-563f18c1a66e\") " pod="ai-tenant-e2e-shared-a-w3-965a07/auth-test-model-965a07-kserve-86544f48df-b4jcp" Nov 03 02:37:05.288181 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:37:05.286866 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/233b378d-0c02-4452-854f-563f18c1a66e-tmp-dir\") pod \"auth-test-model-965a07-kserve-86544f48df-b4jcp\" (UID: \"233b378d-0c02-4452-854f-563f18c1a66e\") " pod="ai-tenant-e2e-shared-a-w3-965a07/auth-test-model-965a07-kserve-86544f48df-b4jcp" Nov 03 02:37:05.288181 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:37:05.286944 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/233b378d-0c02-4452-854f-563f18c1a66e-tls-certs\") pod \"auth-test-model-965a07-kserve-86544f48df-b4jcp\" (UID: \"233b378d-0c02-4452-854f-563f18c1a66e\") " pod="ai-tenant-e2e-shared-a-w3-965a07/auth-test-model-965a07-kserve-86544f48df-b4jcp" Nov 03 02:37:05.288181 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:37:05.287070 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/233b378d-0c02-4452-854f-563f18c1a66e-model-cache\") pod \"auth-test-model-965a07-kserve-86544f48df-b4jcp\" (UID: \"233b378d-0c02-4452-854f-563f18c1a66e\") " pod="ai-tenant-e2e-shared-a-w3-965a07/auth-test-model-965a07-kserve-86544f48df-b4jcp" Nov 03 02:37:05.288181 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:37:05.287132 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-942zn\" (UniqueName: \"kubernetes.io/projected/233b378d-0c02-4452-854f-563f18c1a66e-kube-api-access-942zn\") pod \"auth-test-model-965a07-kserve-86544f48df-b4jcp\" (UID: \"233b378d-0c02-4452-854f-563f18c1a66e\") " pod="ai-tenant-e2e-shared-a-w3-965a07/auth-test-model-965a07-kserve-86544f48df-b4jcp" Nov 03 02:37:05.288181 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:37:05.287249 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/233b378d-0c02-4452-854f-563f18c1a66e-dshm\") pod \"auth-test-model-965a07-kserve-86544f48df-b4jcp\" (UID: \"233b378d-0c02-4452-854f-563f18c1a66e\") " pod="ai-tenant-e2e-shared-a-w3-965a07/auth-test-model-965a07-kserve-86544f48df-b4jcp" Nov 03 02:37:05.332961 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:37:05.332918 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-965a07\"/\"openshift-service-ca.crt\"" Nov 03 02:37:05.332961 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:37:05.332946 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-965a07\"/\"kube-root-ca.crt\"" Nov 03 02:37:05.333188 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:37:05.332966 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-965a07\"/\"auth-test-model-965a07-kserve-self-signed-certs\"" Nov 03 02:37:05.333188 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:37:05.333024 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-965a07\"/\"default-dockercfg-xptq8\"" Nov 03 02:37:05.335865 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:37:05.335844 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-965a07/auth-test-model-965a07-kserve-86544f48df-b4jcp"] Nov 03 02:37:05.396422 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:37:05.396312 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/233b378d-0c02-4452-854f-563f18c1a66e-home\") pod \"auth-test-model-965a07-kserve-86544f48df-b4jcp\" (UID: \"233b378d-0c02-4452-854f-563f18c1a66e\") " pod="ai-tenant-e2e-shared-a-w3-965a07/auth-test-model-965a07-kserve-86544f48df-b4jcp" Nov 03 02:37:05.396670 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:37:05.396499 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/233b378d-0c02-4452-854f-563f18c1a66e-tmp-dir\") pod \"auth-test-model-965a07-kserve-86544f48df-b4jcp\" (UID: \"233b378d-0c02-4452-854f-563f18c1a66e\") " pod="ai-tenant-e2e-shared-a-w3-965a07/auth-test-model-965a07-kserve-86544f48df-b4jcp" Nov 03 02:37:05.396670 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:37:05.396568 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/233b378d-0c02-4452-854f-563f18c1a66e-tls-certs\") pod \"auth-test-model-965a07-kserve-86544f48df-b4jcp\" (UID: \"233b378d-0c02-4452-854f-563f18c1a66e\") " pod="ai-tenant-e2e-shared-a-w3-965a07/auth-test-model-965a07-kserve-86544f48df-b4jcp" Nov 03 02:37:05.396670 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:37:05.396638 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/233b378d-0c02-4452-854f-563f18c1a66e-model-cache\") pod \"auth-test-model-965a07-kserve-86544f48df-b4jcp\" (UID: \"233b378d-0c02-4452-854f-563f18c1a66e\") " pod="ai-tenant-e2e-shared-a-w3-965a07/auth-test-model-965a07-kserve-86544f48df-b4jcp" Nov 03 02:37:05.396845 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:37:05.396690 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-942zn\" (UniqueName: \"kubernetes.io/projected/233b378d-0c02-4452-854f-563f18c1a66e-kube-api-access-942zn\") pod \"auth-test-model-965a07-kserve-86544f48df-b4jcp\" (UID: \"233b378d-0c02-4452-854f-563f18c1a66e\") " pod="ai-tenant-e2e-shared-a-w3-965a07/auth-test-model-965a07-kserve-86544f48df-b4jcp" Nov 03 02:37:05.396845 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:37:05.396808 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/233b378d-0c02-4452-854f-563f18c1a66e-dshm\") pod \"auth-test-model-965a07-kserve-86544f48df-b4jcp\" (UID: \"233b378d-0c02-4452-854f-563f18c1a66e\") " pod="ai-tenant-e2e-shared-a-w3-965a07/auth-test-model-965a07-kserve-86544f48df-b4jcp" Nov 03 02:37:05.403460 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:37:05.401588 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/233b378d-0c02-4452-854f-563f18c1a66e-dshm\") pod \"auth-test-model-965a07-kserve-86544f48df-b4jcp\" (UID: \"233b378d-0c02-4452-854f-563f18c1a66e\") " pod="ai-tenant-e2e-shared-a-w3-965a07/auth-test-model-965a07-kserve-86544f48df-b4jcp" Nov 03 02:37:05.403460 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:37:05.401848 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/233b378d-0c02-4452-854f-563f18c1a66e-model-cache\") pod \"auth-test-model-965a07-kserve-86544f48df-b4jcp\" (UID: \"233b378d-0c02-4452-854f-563f18c1a66e\") " pod="ai-tenant-e2e-shared-a-w3-965a07/auth-test-model-965a07-kserve-86544f48df-b4jcp" Nov 03 02:37:05.403460 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:37:05.402470 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/233b378d-0c02-4452-854f-563f18c1a66e-tmp-dir\") pod \"auth-test-model-965a07-kserve-86544f48df-b4jcp\" (UID: \"233b378d-0c02-4452-854f-563f18c1a66e\") " pod="ai-tenant-e2e-shared-a-w3-965a07/auth-test-model-965a07-kserve-86544f48df-b4jcp" Nov 03 02:37:05.403746 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:37:05.403581 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/233b378d-0c02-4452-854f-563f18c1a66e-home\") pod \"auth-test-model-965a07-kserve-86544f48df-b4jcp\" (UID: \"233b378d-0c02-4452-854f-563f18c1a66e\") " pod="ai-tenant-e2e-shared-a-w3-965a07/auth-test-model-965a07-kserve-86544f48df-b4jcp" Nov 03 02:37:05.411737 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:37:05.411706 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/233b378d-0c02-4452-854f-563f18c1a66e-tls-certs\") pod \"auth-test-model-965a07-kserve-86544f48df-b4jcp\" (UID: \"233b378d-0c02-4452-854f-563f18c1a66e\") " pod="ai-tenant-e2e-shared-a-w3-965a07/auth-test-model-965a07-kserve-86544f48df-b4jcp" Nov 03 02:37:05.457213 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:37:05.457175 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-942zn\" (UniqueName: \"kubernetes.io/projected/233b378d-0c02-4452-854f-563f18c1a66e-kube-api-access-942zn\") pod \"auth-test-model-965a07-kserve-86544f48df-b4jcp\" (UID: \"233b378d-0c02-4452-854f-563f18c1a66e\") " pod="ai-tenant-e2e-shared-a-w3-965a07/auth-test-model-965a07-kserve-86544f48df-b4jcp" Nov 03 02:37:05.548100 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:37:05.548020 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-965a07/auth-test-model-965a07-kserve-86544f48df-b4jcp" Nov 03 02:37:05.930316 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:37:05.930279 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-965a07/auth-test-model-965a07-kserve-86544f48df-b4jcp"] Nov 03 02:37:05.935008 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:37:05.934978 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod233b378d_0c02_4452_854f_563f18c1a66e.slice/crio-d3418640b162ee95be6a2f9ba431ada80412b1764cd93f6e38ef33ee3a0ff44e WatchSource:0}: Error finding container d3418640b162ee95be6a2f9ba431ada80412b1764cd93f6e38ef33ee3a0ff44e: Status 404 returned error can't find the container with id d3418640b162ee95be6a2f9ba431ada80412b1764cd93f6e38ef33ee3a0ff44e Nov 03 02:37:06.765496 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:37:06.765445 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-965a07/auth-test-model-965a07-kserve-86544f48df-b4jcp" event={"ID":"233b378d-0c02-4452-854f-563f18c1a66e","Type":"ContainerStarted","Data":"51d5b4295c7ee99d5179b9cbc26e620e326e7409082b13abe8b761c8554e5dc1"} Nov 03 02:37:06.766096 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:37:06.765519 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-965a07/auth-test-model-965a07-kserve-86544f48df-b4jcp" event={"ID":"233b378d-0c02-4452-854f-563f18c1a66e","Type":"ContainerStarted","Data":"d3418640b162ee95be6a2f9ba431ada80412b1764cd93f6e38ef33ee3a0ff44e"} Nov 03 02:37:06.914770 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:37:06.914716 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-965a07/auth-test-model-965a07-kserve-86544f48df-b4jcp" podStartSLOduration=1.914702955 podStartE2EDuration="1.914702955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:37:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:37:06.871767045 +0000 UTC m=+2704.130965202" watchObservedRunningTime="2026-11-03 02:37:06.914702955 +0000 UTC m=+2704.173901141" Nov 03 02:37:07.548788 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:37:07.548752 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-965a07/auth-test-model-965a07-kserve-86544f48df-b4jcp" Nov 03 02:37:14.045174 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:37:14.045085 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36dca76a-75qw9"] Nov 03 02:37:14.091843 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:37:14.091809 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36dca76a-75qw9"] Nov 03 02:37:14.850695 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:37:14.850658 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3641616a-f449-4f07-876c-f2c1710e4edd" path="/var/lib/kubelet/pods/3641616a-f449-4f07-876c-f2c1710e4edd/volumes" Nov 03 02:37:15.549141 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:37:15.549112 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-965a07/auth-test-model-965a07-kserve-86544f48df-b4jcp" Nov 03 02:37:15.561922 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:37:15.561899 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-965a07/auth-test-model-965a07-kserve-86544f48df-b4jcp" Nov 03 02:37:15.838956 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:37:15.838877 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-965a07/auth-test-model-965a07-kserve-86544f48df-b4jcp" Nov 03 02:37:19.044034 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:37:19.043992 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7183b7ee-b8bn5"] Nov 03 02:37:19.090338 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:37:19.090306 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7183b7ee-b8bn5"] Nov 03 02:37:20.850998 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:37:20.850960 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98b93b8e-c7eb-40c5-896d-bb6d93d6df21" path="/var/lib/kubelet/pods/98b93b8e-c7eb-40c5-896d-bb6d93d6df21/volumes" Nov 03 02:37:21.043335 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:37:21.043303 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-777defe4-4dcnd"] Nov 03 02:37:21.089900 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:37:21.089872 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-777defe4-4dcnd"] Nov 03 02:37:21.535234 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:37:21.535200 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:37:22.853216 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:37:22.853182 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a47a1d4-8b2c-4edd-9c89-6ca01b06bc76" path="/var/lib/kubelet/pods/2a47a1d4-8b2c-4edd-9c89-6ca01b06bc76/volumes" Nov 03 02:37:23.915618 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:37:23.915576 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:37:37.046331 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:37:37.046296 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cda482ac-rxtqh"] Nov 03 02:37:37.092994 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:37:37.092971 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c5a3177-kx7m8"] Nov 03 02:37:37.095394 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:37:37.095364 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cda482ac-rxtqh"] Nov 03 02:37:37.097673 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:37:37.097649 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c5a3177-kx7m8"] Nov 03 02:37:38.851202 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:37:38.851169 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63fc50d9-d0a7-428a-8dcc-894bafea8987" path="/var/lib/kubelet/pods/63fc50d9-d0a7-428a-8dcc-894bafea8987/volumes" Nov 03 02:37:38.851622 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:37:38.851606 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6a49f49-9af3-4c84-8a68-5ff152f7154e" path="/var/lib/kubelet/pods/b6a49f49-9af3-4c84-8a68-5ff152f7154e/volumes" Nov 03 02:37:42.043225 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:37:42.043184 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-484e0158-5f5v7"] Nov 03 02:37:42.089736 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:37:42.089706 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-484e0158-5f5v7"] Nov 03 02:37:42.852421 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:37:42.852384 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7d44d42-0cf3-4195-8e69-73bcd264619a" path="/var/lib/kubelet/pods/e7d44d42-0cf3-4195-8e69-73bcd264619a/volumes" Nov 03 02:37:48.046098 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:37:48.046067 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1caa6e5-5t6j8"] Nov 03 02:37:48.092516 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:37:48.092487 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1caa6e5-5t6j8"] Nov 03 02:37:48.849756 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:37:48.849713 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4748e14a-d17c-43ab-8951-0a01f40b7512" path="/var/lib/kubelet/pods/4748e14a-d17c-43ab-8951-0a01f40b7512/volumes" Nov 03 02:37:53.915163 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:37:53.915124 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:37:56.044069 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:37:56.044035 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e16e9de-292gr"] Nov 03 02:37:56.090590 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:37:56.090561 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e16e9de-292gr"] Nov 03 02:37:56.851797 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:37:56.851766 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72f7eed5-46c0-4c61-bb2e-7cfe945667c9" path="/var/lib/kubelet/pods/72f7eed5-46c0-4c61-bb2e-7cfe945667c9/volumes" Nov 03 02:38:00.044956 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:00.044909 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1df32a7a-nq7p2"] Nov 03 02:38:00.091637 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:00.091608 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1df32a7a-nq7p2"] Nov 03 02:38:00.850191 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:00.850157 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9269e01-6892-4797-815a-b46f93340eae" path="/var/lib/kubelet/pods/d9269e01-6892-4797-815a-b46f93340eae/volumes" Nov 03 02:38:03.931588 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:03.931556 5605 scope.go:122] "RemoveContainer" containerID="39f3f8edca2048dc9d94968d6b58451bccb2c6950435296ae2bd87cb2f51b88e" Nov 03 02:38:03.941793 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:03.941773 5605 scope.go:122] "RemoveContainer" containerID="5bfbd14e91f5154f28a4a9ba1ab9b1ad252553ae81038091f2d26e9735661cec" Nov 03 02:38:03.950649 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:03.950620 5605 scope.go:122] "RemoveContainer" containerID="10c1eb9e29a74a9065672924559b5dc1dad22509031dfc9d55f54feb9b236d98" Nov 03 02:38:03.958688 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:03.958664 5605 scope.go:122] "RemoveContainer" containerID="e1ed4d531758e6eb40403e9c999c8d7c35f1d8524f11f6d623f87f42788a9c5a" Nov 03 02:38:03.966307 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:03.966291 5605 scope.go:122] "RemoveContainer" containerID="1ef1c328f11169a92e3a3ad2a339b66b2f4173c733fdf688c75a1c5768797b00" Nov 03 02:38:16.132666 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:16.132634 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts"] Nov 03 02:38:16.133053 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:16.132851 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts" podUID="77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7" containerName="istio-proxy" containerID="cri-o://bb5d9985b1995763ff8eae2b182a6162980bbb304b9f543a73f56c1b6244cac2" gracePeriod=30 Nov 03 02:38:17.088658 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:17.088627 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a192b197-72s2r"] Nov 03 02:38:17.090844 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:17.090823 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a192b197-72s2r"] Nov 03 02:38:18.850417 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:18.850386 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bb3aeb2-84e3-4880-9baa-36abe371f261" path="/var/lib/kubelet/pods/0bb3aeb2-84e3-4880-9baa-36abe371f261/volumes" Nov 03 02:38:21.011828 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:21.011794 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:38:21.294888 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:21.294804 5605 generic.go:356] "Generic (PLEG): container finished" podID="77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7" containerID="bb5d9985b1995763ff8eae2b182a6162980bbb304b9f543a73f56c1b6244cac2" exitCode=0 Nov 03 02:38:21.294888 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:21.294874 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts" event={"ID":"77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7","Type":"ContainerDied","Data":"bb5d9985b1995763ff8eae2b182a6162980bbb304b9f543a73f56c1b6244cac2"} Nov 03 02:38:21.397772 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:21.397743 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts" Nov 03 02:38:21.494772 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:21.493271 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7-istio-data\") pod \"77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7\" (UID: \"77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7\") " Nov 03 02:38:21.494772 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:21.493370 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7-workload-socket\") pod \"77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7\" (UID: \"77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7\") " Nov 03 02:38:21.494772 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:21.493428 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7-credential-socket\") pod \"77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7\" (UID: \"77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7\") " Nov 03 02:38:21.494772 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:21.493490 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7-kube-api-access-x8lck\" (UniqueName: \"kubernetes.io/projected/77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7-kube-api-access-x8lck\") pod \"77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7\" (UID: \"77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7\") " Nov 03 02:38:21.494772 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:21.493543 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7-istio-token\" (UniqueName: \"kubernetes.io/projected/77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7-istio-token\") pod \"77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7\" (UID: \"77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7\") " Nov 03 02:38:21.494772 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:21.493572 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7-istio-podinfo\") pod \"77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7\" (UID: \"77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7\") " Nov 03 02:38:21.494772 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:21.493611 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7-istio-envoy\") pod \"77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7\" (UID: \"77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7\") " Nov 03 02:38:21.494772 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:21.493652 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7-workload-certs\") pod \"77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7\" (UID: \"77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7\") " Nov 03 02:38:21.494772 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:21.493735 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7-istiod-ca-cert\") pod \"77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7\" (UID: \"77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7\") " Nov 03 02:38:21.494772 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:21.493823 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7-workload-socket" pod "77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7" (UID: "77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:38:21.494772 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:21.494042 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7-istio-data" pod "77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7" (UID: "77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:38:21.494772 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:21.494468 5605 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7-istio-data\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:21.494772 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:21.494492 5605 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7-workload-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:21.496190 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:21.495912 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7-workload-certs" pod "77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7" (UID: "77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:38:21.497110 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:21.497073 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7-istiod-ca-cert" pod "77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7" (UID: "77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:38:21.497515 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:21.497492 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7-credential-socket" pod "77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7" (UID: "77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:38:21.507480 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:21.507434 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7-istio-envoy" pod "77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7" (UID: "77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:38:21.508170 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:21.508135 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7-istio-token" pod "77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7" (UID: "77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:38:21.508469 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:21.508437 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7-istio-podinfo" pod "77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7" (UID: "77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:38:21.508685 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:21.508656 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7-kube-api-access-x8lck" pod "77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7" (UID: "77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7"). InnerVolumeSpecName "kube-api-access-x8lck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:38:21.595282 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:21.595248 5605 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7-credential-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:21.595282 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:21.595279 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8lck\" (UniqueName: \"kubernetes.io/projected/77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7-kube-api-access-x8lck\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:21.595481 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:21.595292 5605 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7-istio-token\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:21.595481 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:21.595302 5605 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7-istio-podinfo\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:21.595481 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:21.595312 5605 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7-istio-envoy\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:21.595481 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:21.595322 5605 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7-workload-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:21.595481 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:21.595331 5605 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7-istiod-ca-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:22.303188 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:22.303157 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts" Nov 03 02:38:22.303629 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:22.303192 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts" event={"ID":"77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7","Type":"ContainerDied","Data":"5aa35fee679478965d913aba4c113b27629234208d27d5466e74df37f243cf0c"} Nov 03 02:38:22.303629 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:22.303253 5605 scope.go:122] "RemoveContainer" containerID="bb5d9985b1995763ff8eae2b182a6162980bbb304b9f543a73f56c1b6244cac2" Nov 03 02:38:22.457929 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:22.457893 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts"] Nov 03 02:38:22.463847 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:22.463818 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gfvts"] Nov 03 02:38:22.852599 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:22.852567 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7" path="/var/lib/kubelet/pods/77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7/volumes" Nov 03 02:38:23.917115 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:38:23.917079 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:38:29.230590 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:29.230556 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl"] Nov 03 02:38:29.231074 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:29.230761 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl" podUID="a15ffc44-4fe7-41cf-9370-794bebfda952" containerName="istio-proxy" containerID="cri-o://c67ce55364d0867af67df425332dc6fc30f1d7b73b673760b0ccfd468ffdd699" gracePeriod=30 Nov 03 02:38:30.780520 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:30.780486 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-9aebe8/auth-test-model-9aebe8-kserve-949d578b5-l8chz"] Nov 03 02:38:30.781010 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:30.780996 5605 cpu_manager.go:414] "containerMap: removing container" podUID="77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7" containerName="istio-proxy" Nov 03 02:38:30.781074 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:30.781011 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7" containerName="istio-proxy" Nov 03 02:38:30.781110 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:30.781093 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="77ce2f96-bb53-40d1-bfd4-a0c5b7ca75f7" containerName="istio-proxy" Nov 03 02:38:30.784429 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:30.784398 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-9aebe8/auth-test-model-9aebe8-kserve-949d578b5-l8chz" Nov 03 02:38:30.875496 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:30.875467 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-9aebe8\"/\"default-dockercfg-nvn92\"" Nov 03 02:38:30.882571 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:30.882542 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-9aebe8\"/\"openshift-service-ca.crt\"" Nov 03 02:38:30.882720 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:30.882573 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-9aebe8\"/\"kube-root-ca.crt\"" Nov 03 02:38:30.882720 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:30.882597 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-9aebe8\"/\"auth-test-model-9aebe8-kserve-self-signed-certs\"" Nov 03 02:38:30.887705 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:30.887674 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-9aebe8/auth-test-model-9aebe8-kserve-949d578b5-l8chz"] Nov 03 02:38:30.908212 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:30.908180 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgjdw\" (UniqueName: \"kubernetes.io/projected/d715d6df-5d86-45da-80bd-bcdd72a73770-kube-api-access-cgjdw\") pod \"auth-test-model-9aebe8-kserve-949d578b5-l8chz\" (UID: \"d715d6df-5d86-45da-80bd-bcdd72a73770\") " pod="ai-tenant-e2e-shared-a-w0-9aebe8/auth-test-model-9aebe8-kserve-949d578b5-l8chz" Nov 03 02:38:30.908346 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:30.908230 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d715d6df-5d86-45da-80bd-bcdd72a73770-home\") pod \"auth-test-model-9aebe8-kserve-949d578b5-l8chz\" (UID: \"d715d6df-5d86-45da-80bd-bcdd72a73770\") " pod="ai-tenant-e2e-shared-a-w0-9aebe8/auth-test-model-9aebe8-kserve-949d578b5-l8chz" Nov 03 02:38:30.908415 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:30.908334 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d715d6df-5d86-45da-80bd-bcdd72a73770-dshm\") pod \"auth-test-model-9aebe8-kserve-949d578b5-l8chz\" (UID: \"d715d6df-5d86-45da-80bd-bcdd72a73770\") " pod="ai-tenant-e2e-shared-a-w0-9aebe8/auth-test-model-9aebe8-kserve-949d578b5-l8chz" Nov 03 02:38:30.908519 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:30.908502 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d715d6df-5d86-45da-80bd-bcdd72a73770-tmp-dir\") pod \"auth-test-model-9aebe8-kserve-949d578b5-l8chz\" (UID: \"d715d6df-5d86-45da-80bd-bcdd72a73770\") " pod="ai-tenant-e2e-shared-a-w0-9aebe8/auth-test-model-9aebe8-kserve-949d578b5-l8chz" Nov 03 02:38:30.908575 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:30.908550 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d715d6df-5d86-45da-80bd-bcdd72a73770-tls-certs\") pod \"auth-test-model-9aebe8-kserve-949d578b5-l8chz\" (UID: \"d715d6df-5d86-45da-80bd-bcdd72a73770\") " pod="ai-tenant-e2e-shared-a-w0-9aebe8/auth-test-model-9aebe8-kserve-949d578b5-l8chz" Nov 03 02:38:30.908649 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:30.908636 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d715d6df-5d86-45da-80bd-bcdd72a73770-model-cache\") pod \"auth-test-model-9aebe8-kserve-949d578b5-l8chz\" (UID: \"d715d6df-5d86-45da-80bd-bcdd72a73770\") " pod="ai-tenant-e2e-shared-a-w0-9aebe8/auth-test-model-9aebe8-kserve-949d578b5-l8chz" Nov 03 02:38:31.009932 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:31.009900 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d715d6df-5d86-45da-80bd-bcdd72a73770-tmp-dir\") pod \"auth-test-model-9aebe8-kserve-949d578b5-l8chz\" (UID: \"d715d6df-5d86-45da-80bd-bcdd72a73770\") " pod="ai-tenant-e2e-shared-a-w0-9aebe8/auth-test-model-9aebe8-kserve-949d578b5-l8chz" Nov 03 02:38:31.010114 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:31.009948 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d715d6df-5d86-45da-80bd-bcdd72a73770-tls-certs\") pod \"auth-test-model-9aebe8-kserve-949d578b5-l8chz\" (UID: \"d715d6df-5d86-45da-80bd-bcdd72a73770\") " pod="ai-tenant-e2e-shared-a-w0-9aebe8/auth-test-model-9aebe8-kserve-949d578b5-l8chz" Nov 03 02:38:31.010114 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:31.010008 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d715d6df-5d86-45da-80bd-bcdd72a73770-model-cache\") pod \"auth-test-model-9aebe8-kserve-949d578b5-l8chz\" (UID: \"d715d6df-5d86-45da-80bd-bcdd72a73770\") " pod="ai-tenant-e2e-shared-a-w0-9aebe8/auth-test-model-9aebe8-kserve-949d578b5-l8chz" Nov 03 02:38:31.010235 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:31.010205 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgjdw\" (UniqueName: \"kubernetes.io/projected/d715d6df-5d86-45da-80bd-bcdd72a73770-kube-api-access-cgjdw\") pod \"auth-test-model-9aebe8-kserve-949d578b5-l8chz\" (UID: \"d715d6df-5d86-45da-80bd-bcdd72a73770\") " pod="ai-tenant-e2e-shared-a-w0-9aebe8/auth-test-model-9aebe8-kserve-949d578b5-l8chz" Nov 03 02:38:31.010301 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:31.010252 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d715d6df-5d86-45da-80bd-bcdd72a73770-home\") pod \"auth-test-model-9aebe8-kserve-949d578b5-l8chz\" (UID: \"d715d6df-5d86-45da-80bd-bcdd72a73770\") " pod="ai-tenant-e2e-shared-a-w0-9aebe8/auth-test-model-9aebe8-kserve-949d578b5-l8chz" Nov 03 02:38:31.010384 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:31.010295 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d715d6df-5d86-45da-80bd-bcdd72a73770-dshm\") pod \"auth-test-model-9aebe8-kserve-949d578b5-l8chz\" (UID: \"d715d6df-5d86-45da-80bd-bcdd72a73770\") " pod="ai-tenant-e2e-shared-a-w0-9aebe8/auth-test-model-9aebe8-kserve-949d578b5-l8chz" Nov 03 02:38:31.010475 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:31.010453 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d715d6df-5d86-45da-80bd-bcdd72a73770-tmp-dir\") pod \"auth-test-model-9aebe8-kserve-949d578b5-l8chz\" (UID: \"d715d6df-5d86-45da-80bd-bcdd72a73770\") " pod="ai-tenant-e2e-shared-a-w0-9aebe8/auth-test-model-9aebe8-kserve-949d578b5-l8chz" Nov 03 02:38:31.010817 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:31.010720 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d715d6df-5d86-45da-80bd-bcdd72a73770-home\") pod \"auth-test-model-9aebe8-kserve-949d578b5-l8chz\" (UID: \"d715d6df-5d86-45da-80bd-bcdd72a73770\") " pod="ai-tenant-e2e-shared-a-w0-9aebe8/auth-test-model-9aebe8-kserve-949d578b5-l8chz" Nov 03 02:38:31.010817 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:31.010763 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d715d6df-5d86-45da-80bd-bcdd72a73770-model-cache\") pod \"auth-test-model-9aebe8-kserve-949d578b5-l8chz\" (UID: \"d715d6df-5d86-45da-80bd-bcdd72a73770\") " pod="ai-tenant-e2e-shared-a-w0-9aebe8/auth-test-model-9aebe8-kserve-949d578b5-l8chz" Nov 03 02:38:31.012809 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:31.012779 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d715d6df-5d86-45da-80bd-bcdd72a73770-dshm\") pod \"auth-test-model-9aebe8-kserve-949d578b5-l8chz\" (UID: \"d715d6df-5d86-45da-80bd-bcdd72a73770\") " pod="ai-tenant-e2e-shared-a-w0-9aebe8/auth-test-model-9aebe8-kserve-949d578b5-l8chz" Nov 03 02:38:31.013021 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:31.013004 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d715d6df-5d86-45da-80bd-bcdd72a73770-tls-certs\") pod \"auth-test-model-9aebe8-kserve-949d578b5-l8chz\" (UID: \"d715d6df-5d86-45da-80bd-bcdd72a73770\") " pod="ai-tenant-e2e-shared-a-w0-9aebe8/auth-test-model-9aebe8-kserve-949d578b5-l8chz" Nov 03 02:38:31.064350 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:31.064322 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgjdw\" (UniqueName: \"kubernetes.io/projected/d715d6df-5d86-45da-80bd-bcdd72a73770-kube-api-access-cgjdw\") pod \"auth-test-model-9aebe8-kserve-949d578b5-l8chz\" (UID: \"d715d6df-5d86-45da-80bd-bcdd72a73770\") " pod="ai-tenant-e2e-shared-a-w0-9aebe8/auth-test-model-9aebe8-kserve-949d578b5-l8chz" Nov 03 02:38:31.095944 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:31.095908 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-9aebe8/auth-test-model-9aebe8-kserve-949d578b5-l8chz" Nov 03 02:38:31.470489 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:31.470412 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-9aebe8/auth-test-model-9aebe8-kserve-949d578b5-l8chz"] Nov 03 02:38:31.476276 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:38:31.476247 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd715d6df_5d86_45da_80bd_bcdd72a73770.slice/crio-a5831fec3572096f8d31b357152497a401e9c3d87ef710176dd2e4808e4be48d WatchSource:0}: Error finding container a5831fec3572096f8d31b357152497a401e9c3d87ef710176dd2e4808e4be48d: Status 404 returned error can't find the container with id a5831fec3572096f8d31b357152497a401e9c3d87ef710176dd2e4808e4be48d Nov 03 02:38:32.371391 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:32.371347 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-9aebe8/auth-test-model-9aebe8-kserve-949d578b5-l8chz" event={"ID":"d715d6df-5d86-45da-80bd-bcdd72a73770","Type":"ContainerStarted","Data":"0fdc89f54fd70c209c6d1c18c1d520a7900fa0e3d73654294ede449d764192f6"} Nov 03 02:38:32.371777 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:32.371399 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-9aebe8/auth-test-model-9aebe8-kserve-949d578b5-l8chz" event={"ID":"d715d6df-5d86-45da-80bd-bcdd72a73770","Type":"ContainerStarted","Data":"a5831fec3572096f8d31b357152497a401e9c3d87ef710176dd2e4808e4be48d"} Nov 03 02:38:33.096763 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:33.096725 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-9aebe8/auth-test-model-9aebe8-kserve-949d578b5-l8chz" Nov 03 02:38:34.387217 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:34.387186 5605 generic.go:356] "Generic (PLEG): container finished" podID="a15ffc44-4fe7-41cf-9370-794bebfda952" containerID="c67ce55364d0867af67df425332dc6fc30f1d7b73b673760b0ccfd468ffdd699" exitCode=0 Nov 03 02:38:34.387644 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:34.387264 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl" event={"ID":"a15ffc44-4fe7-41cf-9370-794bebfda952","Type":"ContainerDied","Data":"c67ce55364d0867af67df425332dc6fc30f1d7b73b673760b0ccfd468ffdd699"} Nov 03 02:38:34.488278 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:34.488254 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl" Nov 03 02:38:34.644070 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:34.644019 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-9aebe8/auth-test-model-9aebe8-kserve-949d578b5-l8chz" podStartSLOduration=4.643993146 podStartE2EDuration="4.643993146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:38:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:38:32.482187159 +0000 UTC m=+2789.741385311" watchObservedRunningTime="2026-11-03 02:38:34.643993146 +0000 UTC m=+2791.903191295" Nov 03 02:38:34.657874 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:34.657839 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a15ffc44-4fe7-41cf-9370-794bebfda952-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a15ffc44-4fe7-41cf-9370-794bebfda952-istio-envoy\") pod \"a15ffc44-4fe7-41cf-9370-794bebfda952\" (UID: \"a15ffc44-4fe7-41cf-9370-794bebfda952\") " Nov 03 02:38:34.658036 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:34.657895 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a15ffc44-4fe7-41cf-9370-794bebfda952-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a15ffc44-4fe7-41cf-9370-794bebfda952-istio-data\") pod \"a15ffc44-4fe7-41cf-9370-794bebfda952\" (UID: \"a15ffc44-4fe7-41cf-9370-794bebfda952\") " Nov 03 02:38:34.658036 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:34.657929 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a15ffc44-4fe7-41cf-9370-794bebfda952-istio-token\" (UniqueName: \"kubernetes.io/projected/a15ffc44-4fe7-41cf-9370-794bebfda952-istio-token\") pod \"a15ffc44-4fe7-41cf-9370-794bebfda952\" (UID: \"a15ffc44-4fe7-41cf-9370-794bebfda952\") " Nov 03 02:38:34.658163 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:34.658046 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a15ffc44-4fe7-41cf-9370-794bebfda952-kube-api-access-7fr4z\" (UniqueName: \"kubernetes.io/projected/a15ffc44-4fe7-41cf-9370-794bebfda952-kube-api-access-7fr4z\") pod \"a15ffc44-4fe7-41cf-9370-794bebfda952\" (UID: \"a15ffc44-4fe7-41cf-9370-794bebfda952\") " Nov 03 02:38:34.658163 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:34.658082 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a15ffc44-4fe7-41cf-9370-794bebfda952-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a15ffc44-4fe7-41cf-9370-794bebfda952-credential-socket\") pod \"a15ffc44-4fe7-41cf-9370-794bebfda952\" (UID: \"a15ffc44-4fe7-41cf-9370-794bebfda952\") " Nov 03 02:38:34.658163 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:34.658118 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a15ffc44-4fe7-41cf-9370-794bebfda952-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a15ffc44-4fe7-41cf-9370-794bebfda952-istio-podinfo\") pod \"a15ffc44-4fe7-41cf-9370-794bebfda952\" (UID: \"a15ffc44-4fe7-41cf-9370-794bebfda952\") " Nov 03 02:38:34.658163 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:34.658132 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a15ffc44-4fe7-41cf-9370-794bebfda952-istio-data" pod "a15ffc44-4fe7-41cf-9370-794bebfda952" (UID: "a15ffc44-4fe7-41cf-9370-794bebfda952"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:38:34.658163 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:34.658153 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a15ffc44-4fe7-41cf-9370-794bebfda952-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a15ffc44-4fe7-41cf-9370-794bebfda952-istiod-ca-cert\") pod \"a15ffc44-4fe7-41cf-9370-794bebfda952\" (UID: \"a15ffc44-4fe7-41cf-9370-794bebfda952\") " Nov 03 02:38:34.658592 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:34.658231 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a15ffc44-4fe7-41cf-9370-794bebfda952-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a15ffc44-4fe7-41cf-9370-794bebfda952-workload-certs\") pod \"a15ffc44-4fe7-41cf-9370-794bebfda952\" (UID: \"a15ffc44-4fe7-41cf-9370-794bebfda952\") " Nov 03 02:38:34.658592 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:34.658268 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a15ffc44-4fe7-41cf-9370-794bebfda952-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a15ffc44-4fe7-41cf-9370-794bebfda952-workload-socket\") pod \"a15ffc44-4fe7-41cf-9370-794bebfda952\" (UID: \"a15ffc44-4fe7-41cf-9370-794bebfda952\") " Nov 03 02:38:34.658592 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:34.658376 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a15ffc44-4fe7-41cf-9370-794bebfda952-credential-socket" pod "a15ffc44-4fe7-41cf-9370-794bebfda952" (UID: "a15ffc44-4fe7-41cf-9370-794bebfda952"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:38:34.658592 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:34.658549 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a15ffc44-4fe7-41cf-9370-794bebfda952-workload-certs" pod "a15ffc44-4fe7-41cf-9370-794bebfda952" (UID: "a15ffc44-4fe7-41cf-9370-794bebfda952"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:38:34.658855 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:34.658767 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a15ffc44-4fe7-41cf-9370-794bebfda952-workload-socket" pod "a15ffc44-4fe7-41cf-9370-794bebfda952" (UID: "a15ffc44-4fe7-41cf-9370-794bebfda952"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:38:34.658913 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:34.658862 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a15ffc44-4fe7-41cf-9370-794bebfda952-istiod-ca-cert" pod "a15ffc44-4fe7-41cf-9370-794bebfda952" (UID: "a15ffc44-4fe7-41cf-9370-794bebfda952"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:38:34.658985 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:34.658961 5605 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a15ffc44-4fe7-41cf-9370-794bebfda952-istio-data\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:34.658985 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:34.658987 5605 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a15ffc44-4fe7-41cf-9370-794bebfda952-credential-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:34.659184 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:34.659003 5605 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a15ffc44-4fe7-41cf-9370-794bebfda952-workload-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:34.659184 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:34.659022 5605 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a15ffc44-4fe7-41cf-9370-794bebfda952-workload-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:34.660607 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:34.660583 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a15ffc44-4fe7-41cf-9370-794bebfda952-istio-envoy" pod "a15ffc44-4fe7-41cf-9370-794bebfda952" (UID: "a15ffc44-4fe7-41cf-9370-794bebfda952"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:38:34.660718 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:34.660617 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a15ffc44-4fe7-41cf-9370-794bebfda952-istio-token" pod "a15ffc44-4fe7-41cf-9370-794bebfda952" (UID: "a15ffc44-4fe7-41cf-9370-794bebfda952"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:38:34.661120 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:34.661099 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a15ffc44-4fe7-41cf-9370-794bebfda952-istio-podinfo" pod "a15ffc44-4fe7-41cf-9370-794bebfda952" (UID: "a15ffc44-4fe7-41cf-9370-794bebfda952"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:38:34.661207 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:34.661116 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a15ffc44-4fe7-41cf-9370-794bebfda952-kube-api-access-7fr4z" pod "a15ffc44-4fe7-41cf-9370-794bebfda952" (UID: "a15ffc44-4fe7-41cf-9370-794bebfda952"). InnerVolumeSpecName "kube-api-access-7fr4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:38:34.760507 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:34.760480 5605 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a15ffc44-4fe7-41cf-9370-794bebfda952-istio-envoy\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:34.760507 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:34.760506 5605 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a15ffc44-4fe7-41cf-9370-794bebfda952-istio-token\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:34.760668 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:34.760518 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fr4z\" (UniqueName: \"kubernetes.io/projected/a15ffc44-4fe7-41cf-9370-794bebfda952-kube-api-access-7fr4z\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:34.760668 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:34.760528 5605 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a15ffc44-4fe7-41cf-9370-794bebfda952-istio-podinfo\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:34.760668 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:34.760539 5605 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a15ffc44-4fe7-41cf-9370-794bebfda952-istiod-ca-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:35.395506 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:35.395472 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl" Nov 03 02:38:35.395506 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:35.395489 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl" event={"ID":"a15ffc44-4fe7-41cf-9370-794bebfda952","Type":"ContainerDied","Data":"21956dccaa320af80c052e329052f315339d43e46e6c0bd07265a1f9906be40f"} Nov 03 02:38:35.396000 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:35.395556 5605 scope.go:122] "RemoveContainer" containerID="c67ce55364d0867af67df425332dc6fc30f1d7b73b673760b0ccfd468ffdd699" Nov 03 02:38:35.595587 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:35.595556 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl"] Nov 03 02:38:35.642412 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:35.642382 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vqcvl"] Nov 03 02:38:36.850772 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:36.850743 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a15ffc44-4fe7-41cf-9370-794bebfda952" path="/var/lib/kubelet/pods/a15ffc44-4fe7-41cf-9370-794bebfda952/volumes" Nov 03 02:38:41.096720 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:41.096637 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-9aebe8/auth-test-model-9aebe8-kserve-949d578b5-l8chz" Nov 03 02:38:41.109081 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:41.109054 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-9aebe8/auth-test-model-9aebe8-kserve-949d578b5-l8chz" Nov 03 02:38:41.443838 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:41.443753 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-9aebe8/auth-test-model-9aebe8-kserve-949d578b5-l8chz" Nov 03 02:38:46.352426 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:46.352379 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9"] Nov 03 02:38:46.352952 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:46.352600 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9" podUID="f8c546e7-4bc7-459f-8fa8-65e18c1585b3" containerName="istio-proxy" containerID="cri-o://3295455d507f7dd60fb02ea4f8e29dbf7f192fed0c1f023c1da904a6fd24c8b7" gracePeriod=30 Nov 03 02:38:51.499344 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:51.499307 5605 generic.go:356] "Generic (PLEG): container finished" podID="f8c546e7-4bc7-459f-8fa8-65e18c1585b3" containerID="3295455d507f7dd60fb02ea4f8e29dbf7f192fed0c1f023c1da904a6fd24c8b7" exitCode=0 Nov 03 02:38:51.499788 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:51.499388 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9" event={"ID":"f8c546e7-4bc7-459f-8fa8-65e18c1585b3","Type":"ContainerDied","Data":"3295455d507f7dd60fb02ea4f8e29dbf7f192fed0c1f023c1da904a6fd24c8b7"} Nov 03 02:38:51.623942 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:51.623874 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9" Nov 03 02:38:51.750682 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:51.750651 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8c546e7-4bc7-459f-8fa8-65e18c1585b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c546e7-4bc7-459f-8fa8-65e18c1585b3-workload-socket\") pod \"f8c546e7-4bc7-459f-8fa8-65e18c1585b3\" (UID: \"f8c546e7-4bc7-459f-8fa8-65e18c1585b3\") " Nov 03 02:38:51.750863 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:51.750708 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8c546e7-4bc7-459f-8fa8-65e18c1585b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8c546e7-4bc7-459f-8fa8-65e18c1585b3-istio-envoy\") pod \"f8c546e7-4bc7-459f-8fa8-65e18c1585b3\" (UID: \"f8c546e7-4bc7-459f-8fa8-65e18c1585b3\") " Nov 03 02:38:51.750863 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:51.750771 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8c546e7-4bc7-459f-8fa8-65e18c1585b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8c546e7-4bc7-459f-8fa8-65e18c1585b3-workload-certs\") pod \"f8c546e7-4bc7-459f-8fa8-65e18c1585b3\" (UID: \"f8c546e7-4bc7-459f-8fa8-65e18c1585b3\") " Nov 03 02:38:51.750989 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:51.750864 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8c546e7-4bc7-459f-8fa8-65e18c1585b3-kube-api-access-h8fps\" (UniqueName: \"kubernetes.io/projected/f8c546e7-4bc7-459f-8fa8-65e18c1585b3-kube-api-access-h8fps\") pod \"f8c546e7-4bc7-459f-8fa8-65e18c1585b3\" (UID: \"f8c546e7-4bc7-459f-8fa8-65e18c1585b3\") " Nov 03 02:38:51.750989 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:51.750897 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8c546e7-4bc7-459f-8fa8-65e18c1585b3-istio-token\" (UniqueName: \"kubernetes.io/projected/f8c546e7-4bc7-459f-8fa8-65e18c1585b3-istio-token\") pod \"f8c546e7-4bc7-459f-8fa8-65e18c1585b3\" (UID: \"f8c546e7-4bc7-459f-8fa8-65e18c1585b3\") " Nov 03 02:38:51.750989 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:51.750964 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8c546e7-4bc7-459f-8fa8-65e18c1585b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8c546e7-4bc7-459f-8fa8-65e18c1585b3-istiod-ca-cert\") pod \"f8c546e7-4bc7-459f-8fa8-65e18c1585b3\" (UID: \"f8c546e7-4bc7-459f-8fa8-65e18c1585b3\") " Nov 03 02:38:51.751155 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:51.751002 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8c546e7-4bc7-459f-8fa8-65e18c1585b3-workload-socket" pod "f8c546e7-4bc7-459f-8fa8-65e18c1585b3" (UID: "f8c546e7-4bc7-459f-8fa8-65e18c1585b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:38:51.751155 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:51.751067 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f8c546e7-4bc7-459f-8fa8-65e18c1585b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8c546e7-4bc7-459f-8fa8-65e18c1585b3-istio-podinfo\") pod \"f8c546e7-4bc7-459f-8fa8-65e18c1585b3\" (UID: \"f8c546e7-4bc7-459f-8fa8-65e18c1585b3\") " Nov 03 02:38:51.751155 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:51.751119 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8c546e7-4bc7-459f-8fa8-65e18c1585b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8c546e7-4bc7-459f-8fa8-65e18c1585b3-istio-data\") pod \"f8c546e7-4bc7-459f-8fa8-65e18c1585b3\" (UID: \"f8c546e7-4bc7-459f-8fa8-65e18c1585b3\") " Nov 03 02:38:51.751155 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:51.751121 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8c546e7-4bc7-459f-8fa8-65e18c1585b3-workload-certs" pod "f8c546e7-4bc7-459f-8fa8-65e18c1585b3" (UID: "f8c546e7-4bc7-459f-8fa8-65e18c1585b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:38:51.751155 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:51.751151 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8c546e7-4bc7-459f-8fa8-65e18c1585b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c546e7-4bc7-459f-8fa8-65e18c1585b3-credential-socket\") pod \"f8c546e7-4bc7-459f-8fa8-65e18c1585b3\" (UID: \"f8c546e7-4bc7-459f-8fa8-65e18c1585b3\") " Nov 03 02:38:51.752023 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:51.751675 5605 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c546e7-4bc7-459f-8fa8-65e18c1585b3-workload-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:51.752023 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:51.751700 5605 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8c546e7-4bc7-459f-8fa8-65e18c1585b3-workload-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:51.752023 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:51.751906 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8c546e7-4bc7-459f-8fa8-65e18c1585b3-credential-socket" pod "f8c546e7-4bc7-459f-8fa8-65e18c1585b3" (UID: "f8c546e7-4bc7-459f-8fa8-65e18c1585b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:38:51.752023 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:51.751927 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8c546e7-4bc7-459f-8fa8-65e18c1585b3-istio-data" pod "f8c546e7-4bc7-459f-8fa8-65e18c1585b3" (UID: "f8c546e7-4bc7-459f-8fa8-65e18c1585b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:38:51.752323 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:51.752099 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8c546e7-4bc7-459f-8fa8-65e18c1585b3-istiod-ca-cert" pod "f8c546e7-4bc7-459f-8fa8-65e18c1585b3" (UID: "f8c546e7-4bc7-459f-8fa8-65e18c1585b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:38:51.753260 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:51.753230 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8c546e7-4bc7-459f-8fa8-65e18c1585b3-istio-envoy" pod "f8c546e7-4bc7-459f-8fa8-65e18c1585b3" (UID: "f8c546e7-4bc7-459f-8fa8-65e18c1585b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:38:51.753351 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:51.753325 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8c546e7-4bc7-459f-8fa8-65e18c1585b3-kube-api-access-h8fps" pod "f8c546e7-4bc7-459f-8fa8-65e18c1585b3" (UID: "f8c546e7-4bc7-459f-8fa8-65e18c1585b3"). InnerVolumeSpecName "kube-api-access-h8fps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:38:51.753475 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:51.753457 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8c546e7-4bc7-459f-8fa8-65e18c1585b3-istio-podinfo" pod "f8c546e7-4bc7-459f-8fa8-65e18c1585b3" (UID: "f8c546e7-4bc7-459f-8fa8-65e18c1585b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:38:51.753535 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:51.753514 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8c546e7-4bc7-459f-8fa8-65e18c1585b3-istio-token" pod "f8c546e7-4bc7-459f-8fa8-65e18c1585b3" (UID: "f8c546e7-4bc7-459f-8fa8-65e18c1585b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:38:51.853105 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:51.853067 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8fps\" (UniqueName: \"kubernetes.io/projected/f8c546e7-4bc7-459f-8fa8-65e18c1585b3-kube-api-access-h8fps\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:51.853105 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:51.853103 5605 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8c546e7-4bc7-459f-8fa8-65e18c1585b3-istio-token\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:51.853105 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:51.853113 5605 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8c546e7-4bc7-459f-8fa8-65e18c1585b3-istiod-ca-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:51.853401 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:51.853124 5605 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8c546e7-4bc7-459f-8fa8-65e18c1585b3-istio-podinfo\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:51.853401 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:51.853133 5605 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8c546e7-4bc7-459f-8fa8-65e18c1585b3-istio-data\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:51.853401 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:51.853142 5605 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c546e7-4bc7-459f-8fa8-65e18c1585b3-credential-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:51.853401 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:51.853149 5605 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8c546e7-4bc7-459f-8fa8-65e18c1585b3-istio-envoy\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:52.506883 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:52.506844 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9" event={"ID":"f8c546e7-4bc7-459f-8fa8-65e18c1585b3","Type":"ContainerDied","Data":"a3728bfcf6732b2dd20ba14f711856c329fd32bf6a13b1be61452766c957afec"} Nov 03 02:38:52.507381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:52.506887 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9" Nov 03 02:38:52.507381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:52.506934 5605 scope.go:122] "RemoveContainer" containerID="3295455d507f7dd60fb02ea4f8e29dbf7f192fed0c1f023c1da904a6fd24c8b7" Nov 03 02:38:52.707905 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:52.707867 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9"] Nov 03 02:38:52.754928 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:52.754896 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-sv2g9"] Nov 03 02:38:52.853181 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:52.853147 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8c546e7-4bc7-459f-8fa8-65e18c1585b3" path="/var/lib/kubelet/pods/f8c546e7-4bc7-459f-8fa8-65e18c1585b3/volumes" Nov 03 02:38:53.920248 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:38:53.920215 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:38:54.250993 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:54.250915 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-f735f4/auth-test-model-f735f4-kserve-c545848fc-glzwl"] Nov 03 02:38:54.251455 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:54.251441 5605 cpu_manager.go:414] "containerMap: removing container" podUID="a15ffc44-4fe7-41cf-9370-794bebfda952" containerName="istio-proxy" Nov 03 02:38:54.251520 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:54.251456 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a15ffc44-4fe7-41cf-9370-794bebfda952" containerName="istio-proxy" Nov 03 02:38:54.251520 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:54.251495 5605 cpu_manager.go:414] "containerMap: removing container" podUID="f8c546e7-4bc7-459f-8fa8-65e18c1585b3" containerName="istio-proxy" Nov 03 02:38:54.251520 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:54.251500 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8c546e7-4bc7-459f-8fa8-65e18c1585b3" containerName="istio-proxy" Nov 03 02:38:54.251633 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:54.251573 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8c546e7-4bc7-459f-8fa8-65e18c1585b3" containerName="istio-proxy" Nov 03 02:38:54.251633 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:54.251586 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="a15ffc44-4fe7-41cf-9370-794bebfda952" containerName="istio-proxy" Nov 03 02:38:54.256324 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:54.256300 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-f735f4/auth-test-model-f735f4-kserve-c545848fc-glzwl" Nov 03 02:38:54.351659 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:54.351623 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-f735f4\"/\"openshift-service-ca.crt\"" Nov 03 02:38:54.351659 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:54.351648 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-f735f4\"/\"auth-test-model-f735f4-kserve-self-signed-certs\"" Nov 03 02:38:54.351881 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:54.351629 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-f735f4\"/\"kube-root-ca.crt\"" Nov 03 02:38:54.351881 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:54.351712 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-f735f4\"/\"default-dockercfg-n7nmc\"" Nov 03 02:38:54.354453 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:54.354430 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-f735f4/auth-test-model-f735f4-kserve-c545848fc-glzwl"] Nov 03 02:38:54.384528 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:54.384502 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/757daf90-e81c-4b1d-86ff-c6a6afb9d4e4-dshm\") pod \"auth-test-model-f735f4-kserve-c545848fc-glzwl\" (UID: \"757daf90-e81c-4b1d-86ff-c6a6afb9d4e4\") " pod="ai-tenant-e2e-shared-a-w2-f735f4/auth-test-model-f735f4-kserve-c545848fc-glzwl" Nov 03 02:38:54.384642 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:54.384534 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/757daf90-e81c-4b1d-86ff-c6a6afb9d4e4-home\") pod \"auth-test-model-f735f4-kserve-c545848fc-glzwl\" (UID: \"757daf90-e81c-4b1d-86ff-c6a6afb9d4e4\") " pod="ai-tenant-e2e-shared-a-w2-f735f4/auth-test-model-f735f4-kserve-c545848fc-glzwl" Nov 03 02:38:54.384690 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:54.384653 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69rwr\" (UniqueName: \"kubernetes.io/projected/757daf90-e81c-4b1d-86ff-c6a6afb9d4e4-kube-api-access-69rwr\") pod \"auth-test-model-f735f4-kserve-c545848fc-glzwl\" (UID: \"757daf90-e81c-4b1d-86ff-c6a6afb9d4e4\") " pod="ai-tenant-e2e-shared-a-w2-f735f4/auth-test-model-f735f4-kserve-c545848fc-glzwl" Nov 03 02:38:54.384791 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:54.384772 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/757daf90-e81c-4b1d-86ff-c6a6afb9d4e4-tls-certs\") pod \"auth-test-model-f735f4-kserve-c545848fc-glzwl\" (UID: \"757daf90-e81c-4b1d-86ff-c6a6afb9d4e4\") " pod="ai-tenant-e2e-shared-a-w2-f735f4/auth-test-model-f735f4-kserve-c545848fc-glzwl" Nov 03 02:38:54.384850 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:54.384810 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/757daf90-e81c-4b1d-86ff-c6a6afb9d4e4-model-cache\") pod \"auth-test-model-f735f4-kserve-c545848fc-glzwl\" (UID: \"757daf90-e81c-4b1d-86ff-c6a6afb9d4e4\") " pod="ai-tenant-e2e-shared-a-w2-f735f4/auth-test-model-f735f4-kserve-c545848fc-glzwl" Nov 03 02:38:54.384926 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:54.384912 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/757daf90-e81c-4b1d-86ff-c6a6afb9d4e4-tmp-dir\") pod \"auth-test-model-f735f4-kserve-c545848fc-glzwl\" (UID: \"757daf90-e81c-4b1d-86ff-c6a6afb9d4e4\") " pod="ai-tenant-e2e-shared-a-w2-f735f4/auth-test-model-f735f4-kserve-c545848fc-glzwl" Nov 03 02:38:54.486859 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:54.486817 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/757daf90-e81c-4b1d-86ff-c6a6afb9d4e4-tmp-dir\") pod \"auth-test-model-f735f4-kserve-c545848fc-glzwl\" (UID: \"757daf90-e81c-4b1d-86ff-c6a6afb9d4e4\") " pod="ai-tenant-e2e-shared-a-w2-f735f4/auth-test-model-f735f4-kserve-c545848fc-glzwl" Nov 03 02:38:54.487066 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:54.486922 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/757daf90-e81c-4b1d-86ff-c6a6afb9d4e4-dshm\") pod \"auth-test-model-f735f4-kserve-c545848fc-glzwl\" (UID: \"757daf90-e81c-4b1d-86ff-c6a6afb9d4e4\") " pod="ai-tenant-e2e-shared-a-w2-f735f4/auth-test-model-f735f4-kserve-c545848fc-glzwl" Nov 03 02:38:54.487066 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:54.486956 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/757daf90-e81c-4b1d-86ff-c6a6afb9d4e4-home\") pod \"auth-test-model-f735f4-kserve-c545848fc-glzwl\" (UID: \"757daf90-e81c-4b1d-86ff-c6a6afb9d4e4\") " pod="ai-tenant-e2e-shared-a-w2-f735f4/auth-test-model-f735f4-kserve-c545848fc-glzwl" Nov 03 02:38:54.487066 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:54.487059 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69rwr\" (UniqueName: \"kubernetes.io/projected/757daf90-e81c-4b1d-86ff-c6a6afb9d4e4-kube-api-access-69rwr\") pod \"auth-test-model-f735f4-kserve-c545848fc-glzwl\" (UID: \"757daf90-e81c-4b1d-86ff-c6a6afb9d4e4\") " pod="ai-tenant-e2e-shared-a-w2-f735f4/auth-test-model-f735f4-kserve-c545848fc-glzwl" Nov 03 02:38:54.487259 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:54.487163 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/757daf90-e81c-4b1d-86ff-c6a6afb9d4e4-tls-certs\") pod \"auth-test-model-f735f4-kserve-c545848fc-glzwl\" (UID: \"757daf90-e81c-4b1d-86ff-c6a6afb9d4e4\") " pod="ai-tenant-e2e-shared-a-w2-f735f4/auth-test-model-f735f4-kserve-c545848fc-glzwl" Nov 03 02:38:54.487259 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:54.487194 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/757daf90-e81c-4b1d-86ff-c6a6afb9d4e4-model-cache\") pod \"auth-test-model-f735f4-kserve-c545848fc-glzwl\" (UID: \"757daf90-e81c-4b1d-86ff-c6a6afb9d4e4\") " pod="ai-tenant-e2e-shared-a-w2-f735f4/auth-test-model-f735f4-kserve-c545848fc-glzwl" Nov 03 02:38:54.487398 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:54.487295 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/757daf90-e81c-4b1d-86ff-c6a6afb9d4e4-tmp-dir\") pod \"auth-test-model-f735f4-kserve-c545848fc-glzwl\" (UID: \"757daf90-e81c-4b1d-86ff-c6a6afb9d4e4\") " pod="ai-tenant-e2e-shared-a-w2-f735f4/auth-test-model-f735f4-kserve-c545848fc-glzwl" Nov 03 02:38:54.487629 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:54.487607 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/757daf90-e81c-4b1d-86ff-c6a6afb9d4e4-model-cache\") pod \"auth-test-model-f735f4-kserve-c545848fc-glzwl\" (UID: \"757daf90-e81c-4b1d-86ff-c6a6afb9d4e4\") " pod="ai-tenant-e2e-shared-a-w2-f735f4/auth-test-model-f735f4-kserve-c545848fc-glzwl" Nov 03 02:38:54.487741 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:54.487672 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/757daf90-e81c-4b1d-86ff-c6a6afb9d4e4-home\") pod \"auth-test-model-f735f4-kserve-c545848fc-glzwl\" (UID: \"757daf90-e81c-4b1d-86ff-c6a6afb9d4e4\") " pod="ai-tenant-e2e-shared-a-w2-f735f4/auth-test-model-f735f4-kserve-c545848fc-glzwl" Nov 03 02:38:54.489931 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:54.489900 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/757daf90-e81c-4b1d-86ff-c6a6afb9d4e4-dshm\") pod \"auth-test-model-f735f4-kserve-c545848fc-glzwl\" (UID: \"757daf90-e81c-4b1d-86ff-c6a6afb9d4e4\") " pod="ai-tenant-e2e-shared-a-w2-f735f4/auth-test-model-f735f4-kserve-c545848fc-glzwl" Nov 03 02:38:54.490119 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:54.490100 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/757daf90-e81c-4b1d-86ff-c6a6afb9d4e4-tls-certs\") pod \"auth-test-model-f735f4-kserve-c545848fc-glzwl\" (UID: \"757daf90-e81c-4b1d-86ff-c6a6afb9d4e4\") " pod="ai-tenant-e2e-shared-a-w2-f735f4/auth-test-model-f735f4-kserve-c545848fc-glzwl" Nov 03 02:38:54.542283 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:54.542197 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69rwr\" (UniqueName: \"kubernetes.io/projected/757daf90-e81c-4b1d-86ff-c6a6afb9d4e4-kube-api-access-69rwr\") pod \"auth-test-model-f735f4-kserve-c545848fc-glzwl\" (UID: \"757daf90-e81c-4b1d-86ff-c6a6afb9d4e4\") " pod="ai-tenant-e2e-shared-a-w2-f735f4/auth-test-model-f735f4-kserve-c545848fc-glzwl" Nov 03 02:38:54.568841 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:54.568809 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-f735f4/auth-test-model-f735f4-kserve-c545848fc-glzwl" Nov 03 02:38:54.945695 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:54.945660 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-f735f4/auth-test-model-f735f4-kserve-c545848fc-glzwl"] Nov 03 02:38:54.951486 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:38:54.951456 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod757daf90_e81c_4b1d_86ff_c6a6afb9d4e4.slice/crio-c0b3ceb91705244058b117f6aba4f597f0552770ecd085034be521b90b0cf178 WatchSource:0}: Error finding container c0b3ceb91705244058b117f6aba4f597f0552770ecd085034be521b90b0cf178: Status 404 returned error can't find the container with id c0b3ceb91705244058b117f6aba4f597f0552770ecd085034be521b90b0cf178 Nov 03 02:38:55.530761 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:55.530721 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-f735f4/auth-test-model-f735f4-kserve-c545848fc-glzwl" event={"ID":"757daf90-e81c-4b1d-86ff-c6a6afb9d4e4","Type":"ContainerStarted","Data":"4d1c87c8aae1c676743b90e006d7fd96f24bf038c0da1c90cd2a3571191ecdda"} Nov 03 02:38:55.530921 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:55.530770 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-f735f4/auth-test-model-f735f4-kserve-c545848fc-glzwl" event={"ID":"757daf90-e81c-4b1d-86ff-c6a6afb9d4e4","Type":"ContainerStarted","Data":"c0b3ceb91705244058b117f6aba4f597f0552770ecd085034be521b90b0cf178"} Nov 03 02:38:55.569742 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:55.569712 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-f735f4/auth-test-model-f735f4-kserve-c545848fc-glzwl" Nov 03 02:38:55.640925 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:55.640855 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-f735f4/auth-test-model-f735f4-kserve-c545848fc-glzwl" podStartSLOduration=1.640841312 podStartE2EDuration="1.640841312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:38:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:38:55.638453456 +0000 UTC m=+2812.897651604" watchObservedRunningTime="2026-11-03 02:38:55.640841312 +0000 UTC m=+2812.900039465" Nov 03 02:38:59.444272 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:59.444233 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp"] Nov 03 02:38:59.444789 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:38:59.444509 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp" podUID="643d6829-f08f-4bb6-a2e4-d9a9c523224a" containerName="istio-proxy" containerID="cri-o://5e8616c2b2ea0e3e7c9941d908c4a4205ac9dd3901935a02c41365eeaec943f8" gracePeriod=30 Nov 03 02:39:04.010418 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:04.010349 5605 scope.go:122] "RemoveContainer" containerID="6d90d7638aa4ed0a78feed1f2d2d439a3198f94fe75e57dc98a3346627d71e2f" Nov 03 02:39:04.020531 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:04.020504 5605 scope.go:122] "RemoveContainer" containerID="ca8c80f70fca0ef91f9caec8b34bf8d948a754bf4eb5cccf0a2e1e26d142205f" Nov 03 02:39:04.029724 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:04.029701 5605 scope.go:122] "RemoveContainer" containerID="373a5861ef0c8c72652212fdcde83aef2272192d0ffa3e243dd9e1a69d3e9ee9" Nov 03 02:39:04.039263 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:04.039241 5605 scope.go:122] "RemoveContainer" containerID="8faa9ea80d5c1cc9f447dc3b1d7c8606ce366f05f66ff6514d3502c5f12e952a" Nov 03 02:39:04.048321 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:04.048299 5605 scope.go:122] "RemoveContainer" containerID="0f4a551f235a5c12b722108087fa287fee71658ee02836ba2aa609bb373a0f1c" Nov 03 02:39:04.059586 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:04.059562 5605 scope.go:122] "RemoveContainer" containerID="41fa1bc5a0243e2b57427ce8a166d13f1fbab51076f5289b772000f65bd17c98" Nov 03 02:39:04.068283 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:04.068263 5605 scope.go:122] "RemoveContainer" containerID="1d5ad459e171e22ec017f7af74297ebb0b0a751a5515ad6fd4886c41430bdc7b" Nov 03 02:39:04.077541 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:04.077517 5605 scope.go:122] "RemoveContainer" containerID="875ef69df9b7aadaa0f8960739bcfefa753e3c6df0ba4aa19f7a4767f9df253e" Nov 03 02:39:04.086578 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:04.086556 5605 scope.go:122] "RemoveContainer" containerID="a9c18f4cb75508b88063b8933c6801cdebddfeff84e141fc2f51356a70b3623f" Nov 03 02:39:04.569397 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:04.569346 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-f735f4/auth-test-model-f735f4-kserve-c545848fc-glzwl" Nov 03 02:39:04.583101 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:04.583069 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-f735f4/auth-test-model-f735f4-kserve-c545848fc-glzwl" Nov 03 02:39:04.588878 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:04.588811 5605 generic.go:356] "Generic (PLEG): container finished" podID="643d6829-f08f-4bb6-a2e4-d9a9c523224a" containerID="5e8616c2b2ea0e3e7c9941d908c4a4205ac9dd3901935a02c41365eeaec943f8" exitCode=0 Nov 03 02:39:04.589066 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:04.588890 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp" event={"ID":"643d6829-f08f-4bb6-a2e4-d9a9c523224a","Type":"ContainerDied","Data":"5e8616c2b2ea0e3e7c9941d908c4a4205ac9dd3901935a02c41365eeaec943f8"} Nov 03 02:39:04.601295 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:04.601259 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-f735f4/auth-test-model-f735f4-kserve-c545848fc-glzwl" Nov 03 02:39:04.723305 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:04.723243 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp" Nov 03 02:39:04.818639 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:04.818594 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/643d6829-f08f-4bb6-a2e4-d9a9c523224a-istio-token\" (UniqueName: \"kubernetes.io/projected/643d6829-f08f-4bb6-a2e4-d9a9c523224a-istio-token\") pod \"643d6829-f08f-4bb6-a2e4-d9a9c523224a\" (UID: \"643d6829-f08f-4bb6-a2e4-d9a9c523224a\") " Nov 03 02:39:04.818785 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:04.818682 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/643d6829-f08f-4bb6-a2e4-d9a9c523224a-kube-api-access-5j7xv\" (UniqueName: \"kubernetes.io/projected/643d6829-f08f-4bb6-a2e4-d9a9c523224a-kube-api-access-5j7xv\") pod \"643d6829-f08f-4bb6-a2e4-d9a9c523224a\" (UID: \"643d6829-f08f-4bb6-a2e4-d9a9c523224a\") " Nov 03 02:39:04.818785 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:04.818759 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/643d6829-f08f-4bb6-a2e4-d9a9c523224a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/643d6829-f08f-4bb6-a2e4-d9a9c523224a-workload-socket\") pod \"643d6829-f08f-4bb6-a2e4-d9a9c523224a\" (UID: \"643d6829-f08f-4bb6-a2e4-d9a9c523224a\") " Nov 03 02:39:04.818860 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:04.818830 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/643d6829-f08f-4bb6-a2e4-d9a9c523224a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/643d6829-f08f-4bb6-a2e4-d9a9c523224a-istiod-ca-cert\") pod \"643d6829-f08f-4bb6-a2e4-d9a9c523224a\" (UID: \"643d6829-f08f-4bb6-a2e4-d9a9c523224a\") " Nov 03 02:39:04.818930 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:04.818864 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/643d6829-f08f-4bb6-a2e4-d9a9c523224a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/643d6829-f08f-4bb6-a2e4-d9a9c523224a-workload-certs\") pod \"643d6829-f08f-4bb6-a2e4-d9a9c523224a\" (UID: \"643d6829-f08f-4bb6-a2e4-d9a9c523224a\") " Nov 03 02:39:04.819055 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:04.819036 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/643d6829-f08f-4bb6-a2e4-d9a9c523224a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/643d6829-f08f-4bb6-a2e4-d9a9c523224a-credential-socket\") pod \"643d6829-f08f-4bb6-a2e4-d9a9c523224a\" (UID: \"643d6829-f08f-4bb6-a2e4-d9a9c523224a\") " Nov 03 02:39:04.819093 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:04.819080 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/643d6829-f08f-4bb6-a2e4-d9a9c523224a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/643d6829-f08f-4bb6-a2e4-d9a9c523224a-istio-data\") pod \"643d6829-f08f-4bb6-a2e4-d9a9c523224a\" (UID: \"643d6829-f08f-4bb6-a2e4-d9a9c523224a\") " Nov 03 02:39:04.819159 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:04.819143 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/643d6829-f08f-4bb6-a2e4-d9a9c523224a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/643d6829-f08f-4bb6-a2e4-d9a9c523224a-istio-envoy\") pod \"643d6829-f08f-4bb6-a2e4-d9a9c523224a\" (UID: \"643d6829-f08f-4bb6-a2e4-d9a9c523224a\") " Nov 03 02:39:04.819203 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:04.819188 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/643d6829-f08f-4bb6-a2e4-d9a9c523224a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/643d6829-f08f-4bb6-a2e4-d9a9c523224a-istio-podinfo\") pod \"643d6829-f08f-4bb6-a2e4-d9a9c523224a\" (UID: \"643d6829-f08f-4bb6-a2e4-d9a9c523224a\") " Nov 03 02:39:04.820431 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:04.820396 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/643d6829-f08f-4bb6-a2e4-d9a9c523224a-istiod-ca-cert" pod "643d6829-f08f-4bb6-a2e4-d9a9c523224a" (UID: "643d6829-f08f-4bb6-a2e4-d9a9c523224a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:39:04.820625 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:04.820595 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/643d6829-f08f-4bb6-a2e4-d9a9c523224a-workload-certs" pod "643d6829-f08f-4bb6-a2e4-d9a9c523224a" (UID: "643d6829-f08f-4bb6-a2e4-d9a9c523224a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:39:04.820720 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:04.820680 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/643d6829-f08f-4bb6-a2e4-d9a9c523224a-workload-socket" pod "643d6829-f08f-4bb6-a2e4-d9a9c523224a" (UID: "643d6829-f08f-4bb6-a2e4-d9a9c523224a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:39:04.820783 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:04.820714 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/643d6829-f08f-4bb6-a2e4-d9a9c523224a-istio-data" pod "643d6829-f08f-4bb6-a2e4-d9a9c523224a" (UID: "643d6829-f08f-4bb6-a2e4-d9a9c523224a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:39:04.820904 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:04.820883 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/643d6829-f08f-4bb6-a2e4-d9a9c523224a-credential-socket" pod "643d6829-f08f-4bb6-a2e4-d9a9c523224a" (UID: "643d6829-f08f-4bb6-a2e4-d9a9c523224a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:39:04.824400 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:04.823042 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/643d6829-f08f-4bb6-a2e4-d9a9c523224a-istio-podinfo" pod "643d6829-f08f-4bb6-a2e4-d9a9c523224a" (UID: "643d6829-f08f-4bb6-a2e4-d9a9c523224a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:39:04.824400 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:04.823139 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/643d6829-f08f-4bb6-a2e4-d9a9c523224a-istio-envoy" pod "643d6829-f08f-4bb6-a2e4-d9a9c523224a" (UID: "643d6829-f08f-4bb6-a2e4-d9a9c523224a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:39:04.824400 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:04.823332 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/643d6829-f08f-4bb6-a2e4-d9a9c523224a-kube-api-access-5j7xv" pod "643d6829-f08f-4bb6-a2e4-d9a9c523224a" (UID: "643d6829-f08f-4bb6-a2e4-d9a9c523224a"). InnerVolumeSpecName "kube-api-access-5j7xv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:39:04.824400 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:04.823694 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/643d6829-f08f-4bb6-a2e4-d9a9c523224a-istio-token" pod "643d6829-f08f-4bb6-a2e4-d9a9c523224a" (UID: "643d6829-f08f-4bb6-a2e4-d9a9c523224a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:39:04.921474 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:04.921435 5605 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/643d6829-f08f-4bb6-a2e4-d9a9c523224a-istio-podinfo\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:04.921474 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:04.921474 5605 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/643d6829-f08f-4bb6-a2e4-d9a9c523224a-istio-token\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:04.921702 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:04.921489 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5j7xv\" (UniqueName: \"kubernetes.io/projected/643d6829-f08f-4bb6-a2e4-d9a9c523224a-kube-api-access-5j7xv\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:04.921702 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:04.921503 5605 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/643d6829-f08f-4bb6-a2e4-d9a9c523224a-workload-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:04.921702 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:04.921515 5605 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/643d6829-f08f-4bb6-a2e4-d9a9c523224a-istiod-ca-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:04.921702 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:04.921527 5605 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/643d6829-f08f-4bb6-a2e4-d9a9c523224a-workload-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:04.921702 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:04.921538 5605 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/643d6829-f08f-4bb6-a2e4-d9a9c523224a-credential-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:04.921702 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:04.921550 5605 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/643d6829-f08f-4bb6-a2e4-d9a9c523224a-istio-data\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:04.921702 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:04.921561 5605 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/643d6829-f08f-4bb6-a2e4-d9a9c523224a-istio-envoy\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:05.596716 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:05.596684 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp" Nov 03 02:39:05.596716 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:05.596705 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp" event={"ID":"643d6829-f08f-4bb6-a2e4-d9a9c523224a","Type":"ContainerDied","Data":"637cc68be77205bffa1e1f208af471b828f8d0b583ed4a86201ef433e3e17379"} Nov 03 02:39:05.597268 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:05.596769 5605 scope.go:122] "RemoveContainer" containerID="5e8616c2b2ea0e3e7c9941d908c4a4205ac9dd3901935a02c41365eeaec943f8" Nov 03 02:39:05.751110 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:05.751075 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp"] Nov 03 02:39:05.797771 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:05.797734 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-gl6xp"] Nov 03 02:39:06.853818 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:06.850989 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="643d6829-f08f-4bb6-a2e4-d9a9c523224a" path="/var/lib/kubelet/pods/643d6829-f08f-4bb6-a2e4-d9a9c523224a/volumes" Nov 03 02:39:11.554383 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:11.554338 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:39:13.125437 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:13.125387 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-7273af/auth-test-model-7273af-kserve-dfd448c77-g45rq"] Nov 03 02:39:13.126409 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:13.126375 5605 cpu_manager.go:414] "containerMap: removing container" podUID="643d6829-f08f-4bb6-a2e4-d9a9c523224a" containerName="istio-proxy" Nov 03 02:39:13.126409 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:13.126408 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="643d6829-f08f-4bb6-a2e4-d9a9c523224a" containerName="istio-proxy" Nov 03 02:39:13.126740 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:13.126717 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="643d6829-f08f-4bb6-a2e4-d9a9c523224a" containerName="istio-proxy" Nov 03 02:39:13.132706 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:13.132661 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7273af/auth-test-model-7273af-kserve-dfd448c77-g45rq" Nov 03 02:39:13.191584 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:13.191555 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7273af/auth-test-model-7273af-kserve-dfd448c77-g45rq"] Nov 03 02:39:13.213805 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:13.213765 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b495ce9-2ec7-4132-9769-8d1526622c48-dshm\") pod \"auth-test-model-7273af-kserve-dfd448c77-g45rq\" (UID: \"1b495ce9-2ec7-4132-9769-8d1526622c48\") " pod="ai-tenant-e2e-shared-b-w3-7273af/auth-test-model-7273af-kserve-dfd448c77-g45rq" Nov 03 02:39:13.214051 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:13.213870 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b495ce9-2ec7-4132-9769-8d1526622c48-tls-certs\") pod \"auth-test-model-7273af-kserve-dfd448c77-g45rq\" (UID: \"1b495ce9-2ec7-4132-9769-8d1526622c48\") " pod="ai-tenant-e2e-shared-b-w3-7273af/auth-test-model-7273af-kserve-dfd448c77-g45rq" Nov 03 02:39:13.214051 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:13.213898 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b495ce9-2ec7-4132-9769-8d1526622c48-model-cache\") pod \"auth-test-model-7273af-kserve-dfd448c77-g45rq\" (UID: \"1b495ce9-2ec7-4132-9769-8d1526622c48\") " pod="ai-tenant-e2e-shared-b-w3-7273af/auth-test-model-7273af-kserve-dfd448c77-g45rq" Nov 03 02:39:13.214051 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:13.213942 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b495ce9-2ec7-4132-9769-8d1526622c48-tmp-dir\") pod \"auth-test-model-7273af-kserve-dfd448c77-g45rq\" (UID: \"1b495ce9-2ec7-4132-9769-8d1526622c48\") " pod="ai-tenant-e2e-shared-b-w3-7273af/auth-test-model-7273af-kserve-dfd448c77-g45rq" Nov 03 02:39:13.214051 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:13.213989 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b495ce9-2ec7-4132-9769-8d1526622c48-home\") pod \"auth-test-model-7273af-kserve-dfd448c77-g45rq\" (UID: \"1b495ce9-2ec7-4132-9769-8d1526622c48\") " pod="ai-tenant-e2e-shared-b-w3-7273af/auth-test-model-7273af-kserve-dfd448c77-g45rq" Nov 03 02:39:13.214271 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:13.214091 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4566t\" (UniqueName: \"kubernetes.io/projected/1b495ce9-2ec7-4132-9769-8d1526622c48-kube-api-access-4566t\") pod \"auth-test-model-7273af-kserve-dfd448c77-g45rq\" (UID: \"1b495ce9-2ec7-4132-9769-8d1526622c48\") " pod="ai-tenant-e2e-shared-b-w3-7273af/auth-test-model-7273af-kserve-dfd448c77-g45rq" Nov 03 02:39:13.234656 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:13.234537 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7273af\"/\"openshift-service-ca.crt\"" Nov 03 02:39:13.234879 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:13.234616 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7273af\"/\"kube-root-ca.crt\"" Nov 03 02:39:13.234879 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:13.234625 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7273af\"/\"auth-test-model-7273af-kserve-self-signed-certs\"" Nov 03 02:39:13.234879 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:13.234647 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7273af\"/\"default-dockercfg-8x88d\"" Nov 03 02:39:13.315508 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:13.315468 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b495ce9-2ec7-4132-9769-8d1526622c48-dshm\") pod \"auth-test-model-7273af-kserve-dfd448c77-g45rq\" (UID: \"1b495ce9-2ec7-4132-9769-8d1526622c48\") " pod="ai-tenant-e2e-shared-b-w3-7273af/auth-test-model-7273af-kserve-dfd448c77-g45rq" Nov 03 02:39:13.315722 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:13.315599 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b495ce9-2ec7-4132-9769-8d1526622c48-tls-certs\") pod \"auth-test-model-7273af-kserve-dfd448c77-g45rq\" (UID: \"1b495ce9-2ec7-4132-9769-8d1526622c48\") " pod="ai-tenant-e2e-shared-b-w3-7273af/auth-test-model-7273af-kserve-dfd448c77-g45rq" Nov 03 02:39:13.315722 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:13.315628 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b495ce9-2ec7-4132-9769-8d1526622c48-model-cache\") pod \"auth-test-model-7273af-kserve-dfd448c77-g45rq\" (UID: \"1b495ce9-2ec7-4132-9769-8d1526622c48\") " pod="ai-tenant-e2e-shared-b-w3-7273af/auth-test-model-7273af-kserve-dfd448c77-g45rq" Nov 03 02:39:13.315722 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:13.315677 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b495ce9-2ec7-4132-9769-8d1526622c48-tmp-dir\") pod \"auth-test-model-7273af-kserve-dfd448c77-g45rq\" (UID: \"1b495ce9-2ec7-4132-9769-8d1526622c48\") " pod="ai-tenant-e2e-shared-b-w3-7273af/auth-test-model-7273af-kserve-dfd448c77-g45rq" Nov 03 02:39:13.315899 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:13.315731 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b495ce9-2ec7-4132-9769-8d1526622c48-home\") pod \"auth-test-model-7273af-kserve-dfd448c77-g45rq\" (UID: \"1b495ce9-2ec7-4132-9769-8d1526622c48\") " pod="ai-tenant-e2e-shared-b-w3-7273af/auth-test-model-7273af-kserve-dfd448c77-g45rq" Nov 03 02:39:13.315899 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:13.315855 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4566t\" (UniqueName: \"kubernetes.io/projected/1b495ce9-2ec7-4132-9769-8d1526622c48-kube-api-access-4566t\") pod \"auth-test-model-7273af-kserve-dfd448c77-g45rq\" (UID: \"1b495ce9-2ec7-4132-9769-8d1526622c48\") " pod="ai-tenant-e2e-shared-b-w3-7273af/auth-test-model-7273af-kserve-dfd448c77-g45rq" Nov 03 02:39:13.316244 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:13.316213 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b495ce9-2ec7-4132-9769-8d1526622c48-model-cache\") pod \"auth-test-model-7273af-kserve-dfd448c77-g45rq\" (UID: \"1b495ce9-2ec7-4132-9769-8d1526622c48\") " pod="ai-tenant-e2e-shared-b-w3-7273af/auth-test-model-7273af-kserve-dfd448c77-g45rq" Nov 03 02:39:13.316624 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:13.316600 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b495ce9-2ec7-4132-9769-8d1526622c48-tmp-dir\") pod \"auth-test-model-7273af-kserve-dfd448c77-g45rq\" (UID: \"1b495ce9-2ec7-4132-9769-8d1526622c48\") " pod="ai-tenant-e2e-shared-b-w3-7273af/auth-test-model-7273af-kserve-dfd448c77-g45rq" Nov 03 02:39:13.316884 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:13.316858 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b495ce9-2ec7-4132-9769-8d1526622c48-home\") pod \"auth-test-model-7273af-kserve-dfd448c77-g45rq\" (UID: \"1b495ce9-2ec7-4132-9769-8d1526622c48\") " pod="ai-tenant-e2e-shared-b-w3-7273af/auth-test-model-7273af-kserve-dfd448c77-g45rq" Nov 03 02:39:13.327440 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:13.327407 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b495ce9-2ec7-4132-9769-8d1526622c48-tls-certs\") pod \"auth-test-model-7273af-kserve-dfd448c77-g45rq\" (UID: \"1b495ce9-2ec7-4132-9769-8d1526622c48\") " pod="ai-tenant-e2e-shared-b-w3-7273af/auth-test-model-7273af-kserve-dfd448c77-g45rq" Nov 03 02:39:13.327948 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:13.327927 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b495ce9-2ec7-4132-9769-8d1526622c48-dshm\") pod \"auth-test-model-7273af-kserve-dfd448c77-g45rq\" (UID: \"1b495ce9-2ec7-4132-9769-8d1526622c48\") " pod="ai-tenant-e2e-shared-b-w3-7273af/auth-test-model-7273af-kserve-dfd448c77-g45rq" Nov 03 02:39:13.396829 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:13.396745 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4566t\" (UniqueName: \"kubernetes.io/projected/1b495ce9-2ec7-4132-9769-8d1526622c48-kube-api-access-4566t\") pod \"auth-test-model-7273af-kserve-dfd448c77-g45rq\" (UID: \"1b495ce9-2ec7-4132-9769-8d1526622c48\") " pod="ai-tenant-e2e-shared-b-w3-7273af/auth-test-model-7273af-kserve-dfd448c77-g45rq" Nov 03 02:39:13.455685 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:13.455641 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7273af/auth-test-model-7273af-kserve-dfd448c77-g45rq" Nov 03 02:39:14.145180 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:14.145091 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7273af/auth-test-model-7273af-kserve-dfd448c77-g45rq"] Nov 03 02:39:14.152390 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:39:14.152333 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b495ce9_2ec7_4132_9769_8d1526622c48.slice/crio-1b7295a215c240503578d460ab39cccac7c1aa3c3084ac3ebd62bcbcfb0a3433 WatchSource:0}: Error finding container 1b7295a215c240503578d460ab39cccac7c1aa3c3084ac3ebd62bcbcfb0a3433: Status 404 returned error can't find the container with id 1b7295a215c240503578d460ab39cccac7c1aa3c3084ac3ebd62bcbcfb0a3433 Nov 03 02:39:14.671845 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:14.671813 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7273af/auth-test-model-7273af-kserve-dfd448c77-g45rq" event={"ID":"1b495ce9-2ec7-4132-9769-8d1526622c48","Type":"ContainerStarted","Data":"ae5caa9da9ece80fae3cf1bc1995cc51f6ad9d0e5b68c4b2c700e99edc1fa3b0"} Nov 03 02:39:14.672023 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:14.671860 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7273af/auth-test-model-7273af-kserve-dfd448c77-g45rq" event={"ID":"1b495ce9-2ec7-4132-9769-8d1526622c48","Type":"ContainerStarted","Data":"1b7295a215c240503578d460ab39cccac7c1aa3c3084ac3ebd62bcbcfb0a3433"} Nov 03 02:39:14.779458 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:14.779402 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-7273af/auth-test-model-7273af-kserve-dfd448c77-g45rq" podStartSLOduration=1.779387061 podStartE2EDuration="1.779387061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:39:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:39:14.778061696 +0000 UTC m=+2832.037259849" watchObservedRunningTime="2026-11-03 02:39:14.779387061 +0000 UTC m=+2832.038585209" Nov 03 02:39:15.456157 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:15.456116 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-7273af/auth-test-model-7273af-kserve-dfd448c77-g45rq" Nov 03 02:39:23.456610 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:23.456564 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-7273af/auth-test-model-7273af-kserve-dfd448c77-g45rq" Nov 03 02:39:23.474035 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:23.474006 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-7273af/auth-test-model-7273af-kserve-dfd448c77-g45rq" Nov 03 02:39:23.733791 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:23.733706 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-7273af/auth-test-model-7273af-kserve-dfd448c77-g45rq" Nov 03 02:39:23.912891 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:39:23.912855 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:39:24.275324 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:24.275293 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw"] Nov 03 02:39:24.275523 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:24.275496 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw" podUID="b337ace7-57a1-4583-8591-9c2f3040dba6" containerName="istio-proxy" containerID="cri-o://0a547352e01d73c715ee3e182e060cd3f9a1a49618a56830f7f334d5d8e0ad18" gracePeriod=30 Nov 03 02:39:29.543383 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:29.543314 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw" Nov 03 02:39:29.641512 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:29.641474 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b337ace7-57a1-4583-8591-9c2f3040dba6-kube-api-access-75dg4\" (UniqueName: \"kubernetes.io/projected/b337ace7-57a1-4583-8591-9c2f3040dba6-kube-api-access-75dg4\") pod \"b337ace7-57a1-4583-8591-9c2f3040dba6\" (UID: \"b337ace7-57a1-4583-8591-9c2f3040dba6\") " Nov 03 02:39:29.641512 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:29.641515 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b337ace7-57a1-4583-8591-9c2f3040dba6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b337ace7-57a1-4583-8591-9c2f3040dba6-istiod-ca-cert\") pod \"b337ace7-57a1-4583-8591-9c2f3040dba6\" (UID: \"b337ace7-57a1-4583-8591-9c2f3040dba6\") " Nov 03 02:39:29.641740 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:29.641555 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b337ace7-57a1-4583-8591-9c2f3040dba6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b337ace7-57a1-4583-8591-9c2f3040dba6-workload-certs\") pod \"b337ace7-57a1-4583-8591-9c2f3040dba6\" (UID: \"b337ace7-57a1-4583-8591-9c2f3040dba6\") " Nov 03 02:39:29.641740 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:29.641605 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b337ace7-57a1-4583-8591-9c2f3040dba6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b337ace7-57a1-4583-8591-9c2f3040dba6-istio-podinfo\") pod \"b337ace7-57a1-4583-8591-9c2f3040dba6\" (UID: \"b337ace7-57a1-4583-8591-9c2f3040dba6\") " Nov 03 02:39:29.641740 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:29.641624 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b337ace7-57a1-4583-8591-9c2f3040dba6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b337ace7-57a1-4583-8591-9c2f3040dba6-istio-data\") pod \"b337ace7-57a1-4583-8591-9c2f3040dba6\" (UID: \"b337ace7-57a1-4583-8591-9c2f3040dba6\") " Nov 03 02:39:29.641740 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:29.641649 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b337ace7-57a1-4583-8591-9c2f3040dba6-istio-token\" (UniqueName: \"kubernetes.io/projected/b337ace7-57a1-4583-8591-9c2f3040dba6-istio-token\") pod \"b337ace7-57a1-4583-8591-9c2f3040dba6\" (UID: \"b337ace7-57a1-4583-8591-9c2f3040dba6\") " Nov 03 02:39:29.641740 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:29.641694 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b337ace7-57a1-4583-8591-9c2f3040dba6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b337ace7-57a1-4583-8591-9c2f3040dba6-istio-envoy\") pod \"b337ace7-57a1-4583-8591-9c2f3040dba6\" (UID: \"b337ace7-57a1-4583-8591-9c2f3040dba6\") " Nov 03 02:39:29.641740 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:29.641726 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b337ace7-57a1-4583-8591-9c2f3040dba6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b337ace7-57a1-4583-8591-9c2f3040dba6-credential-socket\") pod \"b337ace7-57a1-4583-8591-9c2f3040dba6\" (UID: \"b337ace7-57a1-4583-8591-9c2f3040dba6\") " Nov 03 02:39:29.642057 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:29.641794 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b337ace7-57a1-4583-8591-9c2f3040dba6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b337ace7-57a1-4583-8591-9c2f3040dba6-workload-socket\") pod \"b337ace7-57a1-4583-8591-9c2f3040dba6\" (UID: \"b337ace7-57a1-4583-8591-9c2f3040dba6\") " Nov 03 02:39:29.642057 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:29.641956 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b337ace7-57a1-4583-8591-9c2f3040dba6-istiod-ca-cert" pod "b337ace7-57a1-4583-8591-9c2f3040dba6" (UID: "b337ace7-57a1-4583-8591-9c2f3040dba6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:39:29.642057 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:29.641967 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b337ace7-57a1-4583-8591-9c2f3040dba6-workload-certs" pod "b337ace7-57a1-4583-8591-9c2f3040dba6" (UID: "b337ace7-57a1-4583-8591-9c2f3040dba6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:39:29.642221 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:29.642052 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b337ace7-57a1-4583-8591-9c2f3040dba6-istio-data" pod "b337ace7-57a1-4583-8591-9c2f3040dba6" (UID: "b337ace7-57a1-4583-8591-9c2f3040dba6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:39:29.642221 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:29.642196 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b337ace7-57a1-4583-8591-9c2f3040dba6-credential-socket" pod "b337ace7-57a1-4583-8591-9c2f3040dba6" (UID: "b337ace7-57a1-4583-8591-9c2f3040dba6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:39:29.642334 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:29.642271 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b337ace7-57a1-4583-8591-9c2f3040dba6-workload-socket" pod "b337ace7-57a1-4583-8591-9c2f3040dba6" (UID: "b337ace7-57a1-4583-8591-9c2f3040dba6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:39:29.642555 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:29.642532 5605 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b337ace7-57a1-4583-8591-9c2f3040dba6-workload-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:29.642656 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:29.642559 5605 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b337ace7-57a1-4583-8591-9c2f3040dba6-istiod-ca-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:29.642656 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:29.642574 5605 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b337ace7-57a1-4583-8591-9c2f3040dba6-workload-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:29.642656 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:29.642586 5605 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b337ace7-57a1-4583-8591-9c2f3040dba6-istio-data\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:29.642656 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:29.642599 5605 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b337ace7-57a1-4583-8591-9c2f3040dba6-credential-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:29.644026 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:29.643994 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b337ace7-57a1-4583-8591-9c2f3040dba6-istio-token" pod "b337ace7-57a1-4583-8591-9c2f3040dba6" (UID: "b337ace7-57a1-4583-8591-9c2f3040dba6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:39:29.644120 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:29.644035 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b337ace7-57a1-4583-8591-9c2f3040dba6-istio-envoy" pod "b337ace7-57a1-4583-8591-9c2f3040dba6" (UID: "b337ace7-57a1-4583-8591-9c2f3040dba6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:39:29.644167 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:29.644120 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b337ace7-57a1-4583-8591-9c2f3040dba6-istio-podinfo" pod "b337ace7-57a1-4583-8591-9c2f3040dba6" (UID: "b337ace7-57a1-4583-8591-9c2f3040dba6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:39:29.644216 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:29.644198 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b337ace7-57a1-4583-8591-9c2f3040dba6-kube-api-access-75dg4" pod "b337ace7-57a1-4583-8591-9c2f3040dba6" (UID: "b337ace7-57a1-4583-8591-9c2f3040dba6"). InnerVolumeSpecName "kube-api-access-75dg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:39:29.744181 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:29.744158 5605 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b337ace7-57a1-4583-8591-9c2f3040dba6-istio-envoy\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:29.744181 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:29.744181 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75dg4\" (UniqueName: \"kubernetes.io/projected/b337ace7-57a1-4583-8591-9c2f3040dba6-kube-api-access-75dg4\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:29.744347 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:29.744191 5605 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b337ace7-57a1-4583-8591-9c2f3040dba6-istio-podinfo\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:29.744347 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:29.744199 5605 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b337ace7-57a1-4583-8591-9c2f3040dba6-istio-token\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:29.759556 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:29.759526 5605 generic.go:356] "Generic (PLEG): container finished" podID="b337ace7-57a1-4583-8591-9c2f3040dba6" containerID="0a547352e01d73c715ee3e182e060cd3f9a1a49618a56830f7f334d5d8e0ad18" exitCode=0 Nov 03 02:39:29.759700 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:29.759588 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw" Nov 03 02:39:29.759700 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:29.759596 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw" event={"ID":"b337ace7-57a1-4583-8591-9c2f3040dba6","Type":"ContainerDied","Data":"0a547352e01d73c715ee3e182e060cd3f9a1a49618a56830f7f334d5d8e0ad18"} Nov 03 02:39:29.759700 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:29.759652 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw" event={"ID":"b337ace7-57a1-4583-8591-9c2f3040dba6","Type":"ContainerDied","Data":"b6aed4dc3b9967030e3077b2c8af616c9e7116662e739be7a2ce899f28a7ab41"} Nov 03 02:39:29.759700 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:29.759677 5605 scope.go:122] "RemoveContainer" containerID="0a547352e01d73c715ee3e182e060cd3f9a1a49618a56830f7f334d5d8e0ad18" Nov 03 02:39:29.770000 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:29.769984 5605 scope.go:122] "RemoveContainer" containerID="0a547352e01d73c715ee3e182e060cd3f9a1a49618a56830f7f334d5d8e0ad18" Nov 03 02:39:29.770272 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:39:29.770255 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a547352e01d73c715ee3e182e060cd3f9a1a49618a56830f7f334d5d8e0ad18\": container with ID starting with 0a547352e01d73c715ee3e182e060cd3f9a1a49618a56830f7f334d5d8e0ad18 not found: ID does not exist" containerID="0a547352e01d73c715ee3e182e060cd3f9a1a49618a56830f7f334d5d8e0ad18" Nov 03 02:39:29.770321 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:29.770280 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a547352e01d73c715ee3e182e060cd3f9a1a49618a56830f7f334d5d8e0ad18"} err="failed to get container status \"0a547352e01d73c715ee3e182e060cd3f9a1a49618a56830f7f334d5d8e0ad18\": rpc error: code = NotFound desc = could not find container \"0a547352e01d73c715ee3e182e060cd3f9a1a49618a56830f7f334d5d8e0ad18\": container with ID starting with 0a547352e01d73c715ee3e182e060cd3f9a1a49618a56830f7f334d5d8e0ad18 not found: ID does not exist" Nov 03 02:39:29.915527 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:29.915495 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw"] Nov 03 02:39:29.918161 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:29.918139 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pr5hw"] Nov 03 02:39:30.850009 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:30.849972 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b337ace7-57a1-4583-8591-9c2f3040dba6" path="/var/lib/kubelet/pods/b337ace7-57a1-4583-8591-9c2f3040dba6/volumes" Nov 03 02:39:42.575871 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:42.575837 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb"] Nov 03 02:39:42.576287 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:42.576041 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb" podUID="2e3a3bd1-724d-454c-9e5e-615b71be6ab4" containerName="istio-proxy" containerID="cri-o://a1825dc55be22a1bf31e31e56f7d8d3e130f679efd8f44aebc0c230faa5da06c" gracePeriod=30 Nov 03 02:39:42.624018 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:42.623982 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc"] Nov 03 02:39:42.624231 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:42.624209 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc" podUID="4a2dba2d-d002-4f4e-ab37-298b6dadc320" containerName="istio-proxy" containerID="cri-o://50aecd8f9fadd74fbe86eb1b7501f9d14698eceff2b1c005436503b39b1af3c6" gracePeriod=30 Nov 03 02:39:42.627444 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:42.627409 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm"] Nov 03 02:39:42.627689 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:42.627649 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm" podUID="dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4" containerName="istio-proxy" containerID="cri-o://b6c9856b6582019ea7bf17dc916bb9619be30a2c1c0269bdc75343d7e83f5516" gracePeriod=30 Nov 03 02:39:42.669499 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:42.669465 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45"] Nov 03 02:39:42.669706 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:42.669664 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45" podUID="ed134ef4-d99b-4f5d-abf4-3b04723c2d99" containerName="istio-proxy" containerID="cri-o://5341ae2a71e228242acc3c5f105558ae8cdcbc5216497959a44b153cdd416170" gracePeriod=30 Nov 03 02:39:47.782728 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:47.782693 5605 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Nov 03 02:39:47.783119 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:47.782763 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb" podUID="2e3a3bd1-724d-454c-9e5e-615b71be6ab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Nov 03 02:39:47.880712 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:47.880238 5605 generic.go:356] "Generic (PLEG): container finished" podID="2e3a3bd1-724d-454c-9e5e-615b71be6ab4" containerID="a1825dc55be22a1bf31e31e56f7d8d3e130f679efd8f44aebc0c230faa5da06c" exitCode=0 Nov 03 02:39:47.880712 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:47.880455 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb" event={"ID":"2e3a3bd1-724d-454c-9e5e-615b71be6ab4","Type":"ContainerDied","Data":"a1825dc55be22a1bf31e31e56f7d8d3e130f679efd8f44aebc0c230faa5da06c"} Nov 03 02:39:47.883502 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:47.883258 5605 generic.go:356] "Generic (PLEG): container finished" podID="4a2dba2d-d002-4f4e-ab37-298b6dadc320" containerID="50aecd8f9fadd74fbe86eb1b7501f9d14698eceff2b1c005436503b39b1af3c6" exitCode=0 Nov 03 02:39:47.883502 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:47.883331 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc" event={"ID":"4a2dba2d-d002-4f4e-ab37-298b6dadc320","Type":"ContainerDied","Data":"50aecd8f9fadd74fbe86eb1b7501f9d14698eceff2b1c005436503b39b1af3c6"} Nov 03 02:39:47.890019 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:47.889992 5605 generic.go:356] "Generic (PLEG): container finished" podID="ed134ef4-d99b-4f5d-abf4-3b04723c2d99" containerID="5341ae2a71e228242acc3c5f105558ae8cdcbc5216497959a44b153cdd416170" exitCode=0 Nov 03 02:39:47.890820 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:47.890159 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45" event={"ID":"ed134ef4-d99b-4f5d-abf4-3b04723c2d99","Type":"ContainerDied","Data":"5341ae2a71e228242acc3c5f105558ae8cdcbc5216497959a44b153cdd416170"} Nov 03 02:39:47.893066 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:47.892709 5605 generic.go:356] "Generic (PLEG): container finished" podID="dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4" containerID="b6c9856b6582019ea7bf17dc916bb9619be30a2c1c0269bdc75343d7e83f5516" exitCode=0 Nov 03 02:39:47.893066 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:47.892780 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm" event={"ID":"dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4","Type":"ContainerDied","Data":"b6c9856b6582019ea7bf17dc916bb9619be30a2c1c0269bdc75343d7e83f5516"} Nov 03 02:39:47.912991 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:47.912969 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb" Nov 03 02:39:47.955753 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:47.955726 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2e3a3bd1-724d-454c-9e5e-615b71be6ab4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e3a3bd1-724d-454c-9e5e-615b71be6ab4-istio-podinfo\") pod \"2e3a3bd1-724d-454c-9e5e-615b71be6ab4\" (UID: \"2e3a3bd1-724d-454c-9e5e-615b71be6ab4\") " Nov 03 02:39:47.955883 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:47.955781 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e3a3bd1-724d-454c-9e5e-615b71be6ab4-istio-token\" (UniqueName: \"kubernetes.io/projected/2e3a3bd1-724d-454c-9e5e-615b71be6ab4-istio-token\") pod \"2e3a3bd1-724d-454c-9e5e-615b71be6ab4\" (UID: \"2e3a3bd1-724d-454c-9e5e-615b71be6ab4\") " Nov 03 02:39:47.955883 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:47.955812 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e3a3bd1-724d-454c-9e5e-615b71be6ab4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e3a3bd1-724d-454c-9e5e-615b71be6ab4-istio-data\") pod \"2e3a3bd1-724d-454c-9e5e-615b71be6ab4\" (UID: \"2e3a3bd1-724d-454c-9e5e-615b71be6ab4\") " Nov 03 02:39:47.955883 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:47.955876 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e3a3bd1-724d-454c-9e5e-615b71be6ab4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e3a3bd1-724d-454c-9e5e-615b71be6ab4-istio-envoy\") pod \"2e3a3bd1-724d-454c-9e5e-615b71be6ab4\" (UID: \"2e3a3bd1-724d-454c-9e5e-615b71be6ab4\") " Nov 03 02:39:47.955987 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:47.955909 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e3a3bd1-724d-454c-9e5e-615b71be6ab4-kube-api-access-z5l6k\" (UniqueName: \"kubernetes.io/projected/2e3a3bd1-724d-454c-9e5e-615b71be6ab4-kube-api-access-z5l6k\") pod \"2e3a3bd1-724d-454c-9e5e-615b71be6ab4\" (UID: \"2e3a3bd1-724d-454c-9e5e-615b71be6ab4\") " Nov 03 02:39:47.955987 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:47.955952 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e3a3bd1-724d-454c-9e5e-615b71be6ab4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e3a3bd1-724d-454c-9e5e-615b71be6ab4-credential-socket\") pod \"2e3a3bd1-724d-454c-9e5e-615b71be6ab4\" (UID: \"2e3a3bd1-724d-454c-9e5e-615b71be6ab4\") " Nov 03 02:39:47.956055 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:47.956000 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e3a3bd1-724d-454c-9e5e-615b71be6ab4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e3a3bd1-724d-454c-9e5e-615b71be6ab4-workload-socket\") pod \"2e3a3bd1-724d-454c-9e5e-615b71be6ab4\" (UID: \"2e3a3bd1-724d-454c-9e5e-615b71be6ab4\") " Nov 03 02:39:47.956094 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:47.956082 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e3a3bd1-724d-454c-9e5e-615b71be6ab4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e3a3bd1-724d-454c-9e5e-615b71be6ab4-istiod-ca-cert\") pod \"2e3a3bd1-724d-454c-9e5e-615b71be6ab4\" (UID: \"2e3a3bd1-724d-454c-9e5e-615b71be6ab4\") " Nov 03 02:39:47.956130 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:47.956098 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e3a3bd1-724d-454c-9e5e-615b71be6ab4-istio-data" pod "2e3a3bd1-724d-454c-9e5e-615b71be6ab4" (UID: "2e3a3bd1-724d-454c-9e5e-615b71be6ab4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:39:47.956237 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:47.956221 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e3a3bd1-724d-454c-9e5e-615b71be6ab4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e3a3bd1-724d-454c-9e5e-615b71be6ab4-workload-certs\") pod \"2e3a3bd1-724d-454c-9e5e-615b71be6ab4\" (UID: \"2e3a3bd1-724d-454c-9e5e-615b71be6ab4\") " Nov 03 02:39:47.956639 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:47.956611 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e3a3bd1-724d-454c-9e5e-615b71be6ab4-workload-socket" pod "2e3a3bd1-724d-454c-9e5e-615b71be6ab4" (UID: "2e3a3bd1-724d-454c-9e5e-615b71be6ab4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:39:47.957011 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:47.956974 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e3a3bd1-724d-454c-9e5e-615b71be6ab4-credential-socket" pod "2e3a3bd1-724d-454c-9e5e-615b71be6ab4" (UID: "2e3a3bd1-724d-454c-9e5e-615b71be6ab4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:39:47.957122 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:47.957073 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e3a3bd1-724d-454c-9e5e-615b71be6ab4-istiod-ca-cert" pod "2e3a3bd1-724d-454c-9e5e-615b71be6ab4" (UID: "2e3a3bd1-724d-454c-9e5e-615b71be6ab4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:39:47.957198 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:47.957159 5605 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e3a3bd1-724d-454c-9e5e-615b71be6ab4-istio-data\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:47.957198 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:47.957182 5605 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e3a3bd1-724d-454c-9e5e-615b71be6ab4-credential-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:47.957327 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:47.957230 5605 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e3a3bd1-724d-454c-9e5e-615b71be6ab4-workload-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:47.957653 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:47.957320 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e3a3bd1-724d-454c-9e5e-615b71be6ab4-workload-certs" pod "2e3a3bd1-724d-454c-9e5e-615b71be6ab4" (UID: "2e3a3bd1-724d-454c-9e5e-615b71be6ab4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:39:47.960522 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:47.960491 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e3a3bd1-724d-454c-9e5e-615b71be6ab4-istio-token" pod "2e3a3bd1-724d-454c-9e5e-615b71be6ab4" (UID: "2e3a3bd1-724d-454c-9e5e-615b71be6ab4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:39:47.960698 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:47.960677 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e3a3bd1-724d-454c-9e5e-615b71be6ab4-istio-envoy" pod "2e3a3bd1-724d-454c-9e5e-615b71be6ab4" (UID: "2e3a3bd1-724d-454c-9e5e-615b71be6ab4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:39:47.960762 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:47.960727 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e3a3bd1-724d-454c-9e5e-615b71be6ab4-kube-api-access-z5l6k" pod "2e3a3bd1-724d-454c-9e5e-615b71be6ab4" (UID: "2e3a3bd1-724d-454c-9e5e-615b71be6ab4"). InnerVolumeSpecName "kube-api-access-z5l6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:39:47.962582 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:47.962557 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e3a3bd1-724d-454c-9e5e-615b71be6ab4-istio-podinfo" pod "2e3a3bd1-724d-454c-9e5e-615b71be6ab4" (UID: "2e3a3bd1-724d-454c-9e5e-615b71be6ab4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:39:47.993002 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:47.992980 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm" Nov 03 02:39:48.015855 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.015833 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc" Nov 03 02:39:48.018633 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.018615 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45" Nov 03 02:39:48.057760 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.057739 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed134ef4-d99b-4f5d-abf4-3b04723c2d99-kube-api-access-mjhrh\" (UniqueName: \"kubernetes.io/projected/ed134ef4-d99b-4f5d-abf4-3b04723c2d99-kube-api-access-mjhrh\") pod \"ed134ef4-d99b-4f5d-abf4-3b04723c2d99\" (UID: \"ed134ef4-d99b-4f5d-abf4-3b04723c2d99\") " Nov 03 02:39:48.057870 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.057775 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4-istio-data\") pod \"dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4\" (UID: \"dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4\") " Nov 03 02:39:48.057870 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.057809 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4-workload-socket\") pod \"dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4\" (UID: \"dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4\") " Nov 03 02:39:48.058000 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.057920 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4-kube-api-access-jgvxz\" (UniqueName: \"kubernetes.io/projected/dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4-kube-api-access-jgvxz\") pod \"dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4\" (UID: \"dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4\") " Nov 03 02:39:48.058000 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.057948 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed134ef4-d99b-4f5d-abf4-3b04723c2d99-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed134ef4-d99b-4f5d-abf4-3b04723c2d99-istio-envoy\") pod \"ed134ef4-d99b-4f5d-abf4-3b04723c2d99\" (UID: \"ed134ef4-d99b-4f5d-abf4-3b04723c2d99\") " Nov 03 02:39:48.058000 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.057985 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ed134ef4-d99b-4f5d-abf4-3b04723c2d99-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed134ef4-d99b-4f5d-abf4-3b04723c2d99-istio-podinfo\") pod \"ed134ef4-d99b-4f5d-abf4-3b04723c2d99\" (UID: \"ed134ef4-d99b-4f5d-abf4-3b04723c2d99\") " Nov 03 02:39:48.058251 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.058017 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed134ef4-d99b-4f5d-abf4-3b04723c2d99-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed134ef4-d99b-4f5d-abf4-3b04723c2d99-istiod-ca-cert\") pod \"ed134ef4-d99b-4f5d-abf4-3b04723c2d99\" (UID: \"ed134ef4-d99b-4f5d-abf4-3b04723c2d99\") " Nov 03 02:39:48.058251 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.058045 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a2dba2d-d002-4f4e-ab37-298b6dadc320-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a2dba2d-d002-4f4e-ab37-298b6dadc320-istiod-ca-cert\") pod \"4a2dba2d-d002-4f4e-ab37-298b6dadc320\" (UID: \"4a2dba2d-d002-4f4e-ab37-298b6dadc320\") " Nov 03 02:39:48.058251 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.058050 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4-istio-data" pod "dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4" (UID: "dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:39:48.058251 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.058032 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4-workload-socket" pod "dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4" (UID: "dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:39:48.058251 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.058128 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a2dba2d-d002-4f4e-ab37-298b6dadc320-istio-token\" (UniqueName: \"kubernetes.io/projected/4a2dba2d-d002-4f4e-ab37-298b6dadc320-istio-token\") pod \"4a2dba2d-d002-4f4e-ab37-298b6dadc320\" (UID: \"4a2dba2d-d002-4f4e-ab37-298b6dadc320\") " Nov 03 02:39:48.058251 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.058175 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4-istiod-ca-cert\") pod \"dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4\" (UID: \"dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4\") " Nov 03 02:39:48.058251 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.058218 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4-workload-certs\") pod \"dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4\" (UID: \"dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4\") " Nov 03 02:39:48.058251 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.058247 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4-istio-envoy\") pod \"dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4\" (UID: \"dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4\") " Nov 03 02:39:48.060501 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.058283 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4-istio-token\" (UniqueName: \"kubernetes.io/projected/dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4-istio-token\") pod \"dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4\" (UID: \"dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4\") " Nov 03 02:39:48.060501 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.058316 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed134ef4-d99b-4f5d-abf4-3b04723c2d99-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed134ef4-d99b-4f5d-abf4-3b04723c2d99-istio-data\") pod \"ed134ef4-d99b-4f5d-abf4-3b04723c2d99\" (UID: \"ed134ef4-d99b-4f5d-abf4-3b04723c2d99\") " Nov 03 02:39:48.060501 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.058352 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a2dba2d-d002-4f4e-ab37-298b6dadc320-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a2dba2d-d002-4f4e-ab37-298b6dadc320-workload-certs\") pod \"4a2dba2d-d002-4f4e-ab37-298b6dadc320\" (UID: \"4a2dba2d-d002-4f4e-ab37-298b6dadc320\") " Nov 03 02:39:48.060501 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.058413 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a2dba2d-d002-4f4e-ab37-298b6dadc320-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a2dba2d-d002-4f4e-ab37-298b6dadc320-workload-socket\") pod \"4a2dba2d-d002-4f4e-ab37-298b6dadc320\" (UID: \"4a2dba2d-d002-4f4e-ab37-298b6dadc320\") " Nov 03 02:39:48.060501 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.058492 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a2dba2d-d002-4f4e-ab37-298b6dadc320-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a2dba2d-d002-4f4e-ab37-298b6dadc320-istio-data\") pod \"4a2dba2d-d002-4f4e-ab37-298b6dadc320\" (UID: \"4a2dba2d-d002-4f4e-ab37-298b6dadc320\") " Nov 03 02:39:48.060501 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.058543 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed134ef4-d99b-4f5d-abf4-3b04723c2d99-istio-token\" (UniqueName: \"kubernetes.io/projected/ed134ef4-d99b-4f5d-abf4-3b04723c2d99-istio-token\") pod \"ed134ef4-d99b-4f5d-abf4-3b04723c2d99\" (UID: \"ed134ef4-d99b-4f5d-abf4-3b04723c2d99\") " Nov 03 02:39:48.060501 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.058581 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed134ef4-d99b-4f5d-abf4-3b04723c2d99-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed134ef4-d99b-4f5d-abf4-3b04723c2d99-workload-socket\") pod \"ed134ef4-d99b-4f5d-abf4-3b04723c2d99\" (UID: \"ed134ef4-d99b-4f5d-abf4-3b04723c2d99\") " Nov 03 02:39:48.060501 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.058582 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4-workload-certs" pod "dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4" (UID: "dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:39:48.060501 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.058615 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a2dba2d-d002-4f4e-ab37-298b6dadc320-istiod-ca-cert" pod "4a2dba2d-d002-4f4e-ab37-298b6dadc320" (UID: "4a2dba2d-d002-4f4e-ab37-298b6dadc320"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:39:48.060501 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.058638 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4a2dba2d-d002-4f4e-ab37-298b6dadc320-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a2dba2d-d002-4f4e-ab37-298b6dadc320-istio-podinfo\") pod \"4a2dba2d-d002-4f4e-ab37-298b6dadc320\" (UID: \"4a2dba2d-d002-4f4e-ab37-298b6dadc320\") " Nov 03 02:39:48.060501 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.058670 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4-credential-socket\") pod \"dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4\" (UID: \"dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4\") " Nov 03 02:39:48.060501 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.058699 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4-istio-podinfo\") pod \"dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4\" (UID: \"dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4\") " Nov 03 02:39:48.060501 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.058729 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed134ef4-d99b-4f5d-abf4-3b04723c2d99-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed134ef4-d99b-4f5d-abf4-3b04723c2d99-workload-certs\") pod \"ed134ef4-d99b-4f5d-abf4-3b04723c2d99\" (UID: \"ed134ef4-d99b-4f5d-abf4-3b04723c2d99\") " Nov 03 02:39:48.060501 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.058811 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a2dba2d-d002-4f4e-ab37-298b6dadc320-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a2dba2d-d002-4f4e-ab37-298b6dadc320-credential-socket\") pod \"4a2dba2d-d002-4f4e-ab37-298b6dadc320\" (UID: \"4a2dba2d-d002-4f4e-ab37-298b6dadc320\") " Nov 03 02:39:48.060501 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.058848 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a2dba2d-d002-4f4e-ab37-298b6dadc320-kube-api-access-sknjj\" (UniqueName: \"kubernetes.io/projected/4a2dba2d-d002-4f4e-ab37-298b6dadc320-kube-api-access-sknjj\") pod \"4a2dba2d-d002-4f4e-ab37-298b6dadc320\" (UID: \"4a2dba2d-d002-4f4e-ab37-298b6dadc320\") " Nov 03 02:39:48.060501 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.058891 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed134ef4-d99b-4f5d-abf4-3b04723c2d99-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed134ef4-d99b-4f5d-abf4-3b04723c2d99-credential-socket\") pod \"ed134ef4-d99b-4f5d-abf4-3b04723c2d99\" (UID: \"ed134ef4-d99b-4f5d-abf4-3b04723c2d99\") " Nov 03 02:39:48.060501 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.058943 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a2dba2d-d002-4f4e-ab37-298b6dadc320-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a2dba2d-d002-4f4e-ab37-298b6dadc320-istio-envoy\") pod \"4a2dba2d-d002-4f4e-ab37-298b6dadc320\" (UID: \"4a2dba2d-d002-4f4e-ab37-298b6dadc320\") " Nov 03 02:39:48.060501 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.059562 5605 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e3a3bd1-724d-454c-9e5e-615b71be6ab4-istio-podinfo\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:48.060501 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.059585 5605 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e3a3bd1-724d-454c-9e5e-615b71be6ab4-istio-token\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:48.060501 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.059603 5605 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e3a3bd1-724d-454c-9e5e-615b71be6ab4-istio-envoy\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:48.060501 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.059618 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5l6k\" (UniqueName: \"kubernetes.io/projected/2e3a3bd1-724d-454c-9e5e-615b71be6ab4-kube-api-access-z5l6k\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:48.060501 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.059635 5605 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4-istio-data\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:48.060501 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.059649 5605 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4-workload-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:48.060501 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.059663 5605 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a2dba2d-d002-4f4e-ab37-298b6dadc320-istiod-ca-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:48.060501 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.059677 5605 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e3a3bd1-724d-454c-9e5e-615b71be6ab4-istiod-ca-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:48.060501 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.059693 5605 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4-workload-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:48.060501 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.059707 5605 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e3a3bd1-724d-454c-9e5e-615b71be6ab4-workload-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:48.060501 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.058617 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4-istiod-ca-cert" pod "dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4" (UID: "dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:39:48.060501 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.058836 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a2dba2d-d002-4f4e-ab37-298b6dadc320-istio-data" pod "4a2dba2d-d002-4f4e-ab37-298b6dadc320" (UID: "4a2dba2d-d002-4f4e-ab37-298b6dadc320"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:39:48.060501 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.059001 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed134ef4-d99b-4f5d-abf4-3b04723c2d99-istio-data" pod "ed134ef4-d99b-4f5d-abf4-3b04723c2d99" (UID: "ed134ef4-d99b-4f5d-abf4-3b04723c2d99"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:39:48.060501 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.059089 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed134ef4-d99b-4f5d-abf4-3b04723c2d99-istiod-ca-cert" pod "ed134ef4-d99b-4f5d-abf4-3b04723c2d99" (UID: "ed134ef4-d99b-4f5d-abf4-3b04723c2d99"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:39:48.060501 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.059219 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a2dba2d-d002-4f4e-ab37-298b6dadc320-workload-certs" pod "4a2dba2d-d002-4f4e-ab37-298b6dadc320" (UID: "4a2dba2d-d002-4f4e-ab37-298b6dadc320"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:39:48.060501 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.059331 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4-credential-socket" pod "dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4" (UID: "dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:39:48.060501 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.059429 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a2dba2d-d002-4f4e-ab37-298b6dadc320-credential-socket" pod "4a2dba2d-d002-4f4e-ab37-298b6dadc320" (UID: "4a2dba2d-d002-4f4e-ab37-298b6dadc320"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:39:48.060501 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.059445 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a2dba2d-d002-4f4e-ab37-298b6dadc320-workload-socket" pod "4a2dba2d-d002-4f4e-ab37-298b6dadc320" (UID: "4a2dba2d-d002-4f4e-ab37-298b6dadc320"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:39:48.060501 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.059635 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed134ef4-d99b-4f5d-abf4-3b04723c2d99-workload-certs" pod "ed134ef4-d99b-4f5d-abf4-3b04723c2d99" (UID: "ed134ef4-d99b-4f5d-abf4-3b04723c2d99"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:39:48.060501 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.060276 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed134ef4-d99b-4f5d-abf4-3b04723c2d99-workload-socket" pod "ed134ef4-d99b-4f5d-abf4-3b04723c2d99" (UID: "ed134ef4-d99b-4f5d-abf4-3b04723c2d99"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:39:48.060501 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.060329 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed134ef4-d99b-4f5d-abf4-3b04723c2d99-credential-socket" pod "ed134ef4-d99b-4f5d-abf4-3b04723c2d99" (UID: "ed134ef4-d99b-4f5d-abf4-3b04723c2d99"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:39:48.060501 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.060507 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed134ef4-d99b-4f5d-abf4-3b04723c2d99-istio-envoy" pod "ed134ef4-d99b-4f5d-abf4-3b04723c2d99" (UID: "ed134ef4-d99b-4f5d-abf4-3b04723c2d99"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:39:48.062809 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.061184 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed134ef4-d99b-4f5d-abf4-3b04723c2d99-kube-api-access-mjhrh" pod "ed134ef4-d99b-4f5d-abf4-3b04723c2d99" (UID: "ed134ef4-d99b-4f5d-abf4-3b04723c2d99"). InnerVolumeSpecName "kube-api-access-mjhrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:39:48.062809 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.062102 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4-istio-envoy" pod "dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4" (UID: "dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:39:48.062809 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.062162 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4-istio-token" pod "dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4" (UID: "dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:39:48.062809 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.062161 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a2dba2d-d002-4f4e-ab37-298b6dadc320-istio-envoy" pod "4a2dba2d-d002-4f4e-ab37-298b6dadc320" (UID: "4a2dba2d-d002-4f4e-ab37-298b6dadc320"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:39:48.062809 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.062405 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4a2dba2d-d002-4f4e-ab37-298b6dadc320-istio-podinfo" pod "4a2dba2d-d002-4f4e-ab37-298b6dadc320" (UID: "4a2dba2d-d002-4f4e-ab37-298b6dadc320"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:39:48.062809 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.062641 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4-kube-api-access-jgvxz" pod "dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4" (UID: "dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4"). InnerVolumeSpecName "kube-api-access-jgvxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:39:48.063180 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.062901 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed134ef4-d99b-4f5d-abf4-3b04723c2d99-istio-podinfo" pod "ed134ef4-d99b-4f5d-abf4-3b04723c2d99" (UID: "ed134ef4-d99b-4f5d-abf4-3b04723c2d99"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:39:48.063180 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.062912 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed134ef4-d99b-4f5d-abf4-3b04723c2d99-istio-token" pod "ed134ef4-d99b-4f5d-abf4-3b04723c2d99" (UID: "ed134ef4-d99b-4f5d-abf4-3b04723c2d99"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:39:48.063180 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.062966 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a2dba2d-d002-4f4e-ab37-298b6dadc320-istio-token" pod "4a2dba2d-d002-4f4e-ab37-298b6dadc320" (UID: "4a2dba2d-d002-4f4e-ab37-298b6dadc320"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:39:48.063537 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.063513 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a2dba2d-d002-4f4e-ab37-298b6dadc320-kube-api-access-sknjj" pod "4a2dba2d-d002-4f4e-ab37-298b6dadc320" (UID: "4a2dba2d-d002-4f4e-ab37-298b6dadc320"). InnerVolumeSpecName "kube-api-access-sknjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:39:48.063607 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.063556 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4-istio-podinfo" pod "dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4" (UID: "dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:39:48.160849 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.160769 5605 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a2dba2d-d002-4f4e-ab37-298b6dadc320-istio-token\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:48.160849 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.160794 5605 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4-istiod-ca-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:48.160849 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.160803 5605 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4-istio-envoy\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:48.160849 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.160811 5605 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4-istio-token\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:48.160849 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.160821 5605 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed134ef4-d99b-4f5d-abf4-3b04723c2d99-istio-data\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:48.160849 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.160830 5605 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a2dba2d-d002-4f4e-ab37-298b6dadc320-workload-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:48.160849 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.160838 5605 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a2dba2d-d002-4f4e-ab37-298b6dadc320-workload-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:48.160849 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.160846 5605 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a2dba2d-d002-4f4e-ab37-298b6dadc320-istio-data\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:48.160849 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.160854 5605 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed134ef4-d99b-4f5d-abf4-3b04723c2d99-istio-token\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:48.161308 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.160863 5605 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed134ef4-d99b-4f5d-abf4-3b04723c2d99-workload-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:48.161308 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.160871 5605 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a2dba2d-d002-4f4e-ab37-298b6dadc320-istio-podinfo\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:48.161308 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.160881 5605 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4-credential-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:48.161308 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.160890 5605 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4-istio-podinfo\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:48.161308 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.160899 5605 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed134ef4-d99b-4f5d-abf4-3b04723c2d99-workload-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:48.161308 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.160908 5605 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a2dba2d-d002-4f4e-ab37-298b6dadc320-credential-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:48.161308 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.160917 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sknjj\" (UniqueName: \"kubernetes.io/projected/4a2dba2d-d002-4f4e-ab37-298b6dadc320-kube-api-access-sknjj\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:48.161308 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.160925 5605 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed134ef4-d99b-4f5d-abf4-3b04723c2d99-credential-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:48.161308 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.160934 5605 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a2dba2d-d002-4f4e-ab37-298b6dadc320-istio-envoy\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:48.161308 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.160944 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjhrh\" (UniqueName: \"kubernetes.io/projected/ed134ef4-d99b-4f5d-abf4-3b04723c2d99-kube-api-access-mjhrh\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:48.161308 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.160952 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgvxz\" (UniqueName: \"kubernetes.io/projected/dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4-kube-api-access-jgvxz\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:48.161308 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.160961 5605 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed134ef4-d99b-4f5d-abf4-3b04723c2d99-istio-envoy\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:48.161308 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.160970 5605 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed134ef4-d99b-4f5d-abf4-3b04723c2d99-istio-podinfo\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:48.161308 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.160978 5605 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed134ef4-d99b-4f5d-abf4-3b04723c2d99-istiod-ca-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:48.899429 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.899408 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc" Nov 03 02:39:48.899792 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.899403 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc" event={"ID":"4a2dba2d-d002-4f4e-ab37-298b6dadc320","Type":"ContainerDied","Data":"a586e3ec1d9e39cff747925eca283d7507b8b47727f0ce9f0aa16e7e6c2a08ed"} Nov 03 02:39:48.899792 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.899529 5605 scope.go:122] "RemoveContainer" containerID="50aecd8f9fadd74fbe86eb1b7501f9d14698eceff2b1c005436503b39b1af3c6" Nov 03 02:39:48.900945 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.900925 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45" Nov 03 02:39:48.901066 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.900961 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45" event={"ID":"ed134ef4-d99b-4f5d-abf4-3b04723c2d99","Type":"ContainerDied","Data":"c826ef085227cb2f5b4131b063692c7be34278d27e13b9374d6908ba615c4244"} Nov 03 02:39:48.910369 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.902883 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm" event={"ID":"dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4","Type":"ContainerDied","Data":"a8a6822d64209b19c2f4b3d90937469c30a73f29f7111a28c80d7ad5caa3e068"} Nov 03 02:39:48.910369 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.902896 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm" Nov 03 02:39:48.910369 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.904817 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb" event={"ID":"2e3a3bd1-724d-454c-9e5e-615b71be6ab4","Type":"ContainerDied","Data":"ae549b6b84c9129da7277df47b4495170f03ae5f6da64e2bfae555ec4b70462a"} Nov 03 02:39:48.910369 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.904883 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb" Nov 03 02:39:48.912778 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.912760 5605 scope.go:122] "RemoveContainer" containerID="5341ae2a71e228242acc3c5f105558ae8cdcbc5216497959a44b153cdd416170" Nov 03 02:39:48.921381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.921344 5605 scope.go:122] "RemoveContainer" containerID="b6c9856b6582019ea7bf17dc916bb9619be30a2c1c0269bdc75343d7e83f5516" Nov 03 02:39:48.929397 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:48.929379 5605 scope.go:122] "RemoveContainer" containerID="a1825dc55be22a1bf31e31e56f7d8d3e130f679efd8f44aebc0c230faa5da06c" Nov 03 02:39:49.052998 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:49.052973 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm"] Nov 03 02:39:49.099206 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:49.099186 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-4hvbm"] Nov 03 02:39:49.159164 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:49.159113 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45"] Nov 03 02:39:49.163877 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:49.163859 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-s9f45"] Nov 03 02:39:49.265409 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:49.265377 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc"] Nov 03 02:39:49.269455 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:49.269435 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-rtkhc"] Nov 03 02:39:49.371204 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:49.371178 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb"] Nov 03 02:39:49.374743 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:49.374722 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-nxnbb"] Nov 03 02:39:50.850867 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:50.850837 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e3a3bd1-724d-454c-9e5e-615b71be6ab4" path="/var/lib/kubelet/pods/2e3a3bd1-724d-454c-9e5e-615b71be6ab4/volumes" Nov 03 02:39:50.851266 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:50.851252 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a2dba2d-d002-4f4e-ab37-298b6dadc320" path="/var/lib/kubelet/pods/4a2dba2d-d002-4f4e-ab37-298b6dadc320/volumes" Nov 03 02:39:50.851686 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:50.851672 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4" path="/var/lib/kubelet/pods/dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4/volumes" Nov 03 02:39:50.852324 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:50.852307 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed134ef4-d99b-4f5d-abf4-3b04723c2d99" path="/var/lib/kubelet/pods/ed134ef4-d99b-4f5d-abf4-3b04723c2d99/volumes" Nov 03 02:39:53.251346 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:39:53.251310 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:39:53.915267 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:39:53.915233 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:40:00.340273 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:00.340236 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-78b5c5/auth-test-model-78b5c5-kserve-5bc8bf4cf7-gptc4"] Nov 03 02:40:00.340781 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:00.340750 5605 cpu_manager.go:414] "containerMap: removing container" podUID="dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4" containerName="istio-proxy" Nov 03 02:40:00.340781 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:00.340763 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4" containerName="istio-proxy" Nov 03 02:40:00.340781 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:00.340781 5605 cpu_manager.go:414] "containerMap: removing container" podUID="2e3a3bd1-724d-454c-9e5e-615b71be6ab4" containerName="istio-proxy" Nov 03 02:40:00.340883 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:00.340787 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e3a3bd1-724d-454c-9e5e-615b71be6ab4" containerName="istio-proxy" Nov 03 02:40:00.340883 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:00.340800 5605 cpu_manager.go:414] "containerMap: removing container" podUID="4a2dba2d-d002-4f4e-ab37-298b6dadc320" containerName="istio-proxy" Nov 03 02:40:00.340883 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:00.340806 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a2dba2d-d002-4f4e-ab37-298b6dadc320" containerName="istio-proxy" Nov 03 02:40:00.340883 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:00.340828 5605 cpu_manager.go:414] "containerMap: removing container" podUID="b337ace7-57a1-4583-8591-9c2f3040dba6" containerName="istio-proxy" Nov 03 02:40:00.340883 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:00.340832 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b337ace7-57a1-4583-8591-9c2f3040dba6" containerName="istio-proxy" Nov 03 02:40:00.340883 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:00.340848 5605 cpu_manager.go:414] "containerMap: removing container" podUID="ed134ef4-d99b-4f5d-abf4-3b04723c2d99" containerName="istio-proxy" Nov 03 02:40:00.340883 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:00.340852 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed134ef4-d99b-4f5d-abf4-3b04723c2d99" containerName="istio-proxy" Nov 03 02:40:00.341094 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:00.340917 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a2dba2d-d002-4f4e-ab37-298b6dadc320" containerName="istio-proxy" Nov 03 02:40:00.341094 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:00.340929 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcc23e04-d38a-4ca6-a3b2-67aa5495a9b4" containerName="istio-proxy" Nov 03 02:40:00.341094 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:00.340935 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed134ef4-d99b-4f5d-abf4-3b04723c2d99" containerName="istio-proxy" Nov 03 02:40:00.341094 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:00.340942 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e3a3bd1-724d-454c-9e5e-615b71be6ab4" containerName="istio-proxy" Nov 03 02:40:00.341094 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:00.340948 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="b337ace7-57a1-4583-8591-9c2f3040dba6" containerName="istio-proxy" Nov 03 02:40:00.344125 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:00.344104 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-78b5c5/auth-test-model-78b5c5-kserve-5bc8bf4cf7-gptc4" Nov 03 02:40:00.393250 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:00.393217 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-78b5c5\"/\"kube-root-ca.crt\"" Nov 03 02:40:00.439891 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:00.439857 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-78b5c5\"/\"openshift-service-ca.crt\"" Nov 03 02:40:00.439891 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:00.439864 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-78b5c5\"/\"default-dockercfg-cdhld\"" Nov 03 02:40:00.440097 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:00.439929 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-78b5c5\"/\"auth-test-model-78b5c5-kserve-self-signed-certs\"" Nov 03 02:40:00.443269 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:00.443245 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-78b5c5/auth-test-model-78b5c5-kserve-5bc8bf4cf7-gptc4"] Nov 03 02:40:00.486606 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:00.486572 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e3f152e-f3d1-4e14-85a9-6f456467cf6a-home\") pod \"auth-test-model-78b5c5-kserve-5bc8bf4cf7-gptc4\" (UID: \"7e3f152e-f3d1-4e14-85a9-6f456467cf6a\") " pod="ai-tenant-e2e-shared-b-w1-78b5c5/auth-test-model-78b5c5-kserve-5bc8bf4cf7-gptc4" Nov 03 02:40:00.486778 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:00.486648 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgb4w\" (UniqueName: \"kubernetes.io/projected/7e3f152e-f3d1-4e14-85a9-6f456467cf6a-kube-api-access-qgb4w\") pod \"auth-test-model-78b5c5-kserve-5bc8bf4cf7-gptc4\" (UID: \"7e3f152e-f3d1-4e14-85a9-6f456467cf6a\") " pod="ai-tenant-e2e-shared-b-w1-78b5c5/auth-test-model-78b5c5-kserve-5bc8bf4cf7-gptc4" Nov 03 02:40:00.486778 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:00.486716 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e3f152e-f3d1-4e14-85a9-6f456467cf6a-model-cache\") pod \"auth-test-model-78b5c5-kserve-5bc8bf4cf7-gptc4\" (UID: \"7e3f152e-f3d1-4e14-85a9-6f456467cf6a\") " pod="ai-tenant-e2e-shared-b-w1-78b5c5/auth-test-model-78b5c5-kserve-5bc8bf4cf7-gptc4" Nov 03 02:40:00.486778 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:00.486748 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e3f152e-f3d1-4e14-85a9-6f456467cf6a-tls-certs\") pod \"auth-test-model-78b5c5-kserve-5bc8bf4cf7-gptc4\" (UID: \"7e3f152e-f3d1-4e14-85a9-6f456467cf6a\") " pod="ai-tenant-e2e-shared-b-w1-78b5c5/auth-test-model-78b5c5-kserve-5bc8bf4cf7-gptc4" Nov 03 02:40:00.486951 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:00.486888 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e3f152e-f3d1-4e14-85a9-6f456467cf6a-dshm\") pod \"auth-test-model-78b5c5-kserve-5bc8bf4cf7-gptc4\" (UID: \"7e3f152e-f3d1-4e14-85a9-6f456467cf6a\") " pod="ai-tenant-e2e-shared-b-w1-78b5c5/auth-test-model-78b5c5-kserve-5bc8bf4cf7-gptc4" Nov 03 02:40:00.486951 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:00.486929 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e3f152e-f3d1-4e14-85a9-6f456467cf6a-tmp-dir\") pod \"auth-test-model-78b5c5-kserve-5bc8bf4cf7-gptc4\" (UID: \"7e3f152e-f3d1-4e14-85a9-6f456467cf6a\") " pod="ai-tenant-e2e-shared-b-w1-78b5c5/auth-test-model-78b5c5-kserve-5bc8bf4cf7-gptc4" Nov 03 02:40:00.588851 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:00.588802 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e3f152e-f3d1-4e14-85a9-6f456467cf6a-model-cache\") pod \"auth-test-model-78b5c5-kserve-5bc8bf4cf7-gptc4\" (UID: \"7e3f152e-f3d1-4e14-85a9-6f456467cf6a\") " pod="ai-tenant-e2e-shared-b-w1-78b5c5/auth-test-model-78b5c5-kserve-5bc8bf4cf7-gptc4" Nov 03 02:40:00.589057 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:00.588870 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e3f152e-f3d1-4e14-85a9-6f456467cf6a-tls-certs\") pod \"auth-test-model-78b5c5-kserve-5bc8bf4cf7-gptc4\" (UID: \"7e3f152e-f3d1-4e14-85a9-6f456467cf6a\") " pod="ai-tenant-e2e-shared-b-w1-78b5c5/auth-test-model-78b5c5-kserve-5bc8bf4cf7-gptc4" Nov 03 02:40:00.589057 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:00.588972 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e3f152e-f3d1-4e14-85a9-6f456467cf6a-dshm\") pod \"auth-test-model-78b5c5-kserve-5bc8bf4cf7-gptc4\" (UID: \"7e3f152e-f3d1-4e14-85a9-6f456467cf6a\") " pod="ai-tenant-e2e-shared-b-w1-78b5c5/auth-test-model-78b5c5-kserve-5bc8bf4cf7-gptc4" Nov 03 02:40:00.589057 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:00.588998 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e3f152e-f3d1-4e14-85a9-6f456467cf6a-tmp-dir\") pod \"auth-test-model-78b5c5-kserve-5bc8bf4cf7-gptc4\" (UID: \"7e3f152e-f3d1-4e14-85a9-6f456467cf6a\") " pod="ai-tenant-e2e-shared-b-w1-78b5c5/auth-test-model-78b5c5-kserve-5bc8bf4cf7-gptc4" Nov 03 02:40:00.589251 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:00.589086 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e3f152e-f3d1-4e14-85a9-6f456467cf6a-home\") pod \"auth-test-model-78b5c5-kserve-5bc8bf4cf7-gptc4\" (UID: \"7e3f152e-f3d1-4e14-85a9-6f456467cf6a\") " pod="ai-tenant-e2e-shared-b-w1-78b5c5/auth-test-model-78b5c5-kserve-5bc8bf4cf7-gptc4" Nov 03 02:40:00.589251 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:00.589188 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgb4w\" (UniqueName: \"kubernetes.io/projected/7e3f152e-f3d1-4e14-85a9-6f456467cf6a-kube-api-access-qgb4w\") pod \"auth-test-model-78b5c5-kserve-5bc8bf4cf7-gptc4\" (UID: \"7e3f152e-f3d1-4e14-85a9-6f456467cf6a\") " pod="ai-tenant-e2e-shared-b-w1-78b5c5/auth-test-model-78b5c5-kserve-5bc8bf4cf7-gptc4" Nov 03 02:40:00.589381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:00.589282 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e3f152e-f3d1-4e14-85a9-6f456467cf6a-model-cache\") pod \"auth-test-model-78b5c5-kserve-5bc8bf4cf7-gptc4\" (UID: \"7e3f152e-f3d1-4e14-85a9-6f456467cf6a\") " pod="ai-tenant-e2e-shared-b-w1-78b5c5/auth-test-model-78b5c5-kserve-5bc8bf4cf7-gptc4" Nov 03 02:40:00.589681 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:00.589655 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e3f152e-f3d1-4e14-85a9-6f456467cf6a-home\") pod \"auth-test-model-78b5c5-kserve-5bc8bf4cf7-gptc4\" (UID: \"7e3f152e-f3d1-4e14-85a9-6f456467cf6a\") " pod="ai-tenant-e2e-shared-b-w1-78b5c5/auth-test-model-78b5c5-kserve-5bc8bf4cf7-gptc4" Nov 03 02:40:00.589805 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:00.589722 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e3f152e-f3d1-4e14-85a9-6f456467cf6a-tmp-dir\") pod \"auth-test-model-78b5c5-kserve-5bc8bf4cf7-gptc4\" (UID: \"7e3f152e-f3d1-4e14-85a9-6f456467cf6a\") " pod="ai-tenant-e2e-shared-b-w1-78b5c5/auth-test-model-78b5c5-kserve-5bc8bf4cf7-gptc4" Nov 03 02:40:00.591892 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:00.591823 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e3f152e-f3d1-4e14-85a9-6f456467cf6a-dshm\") pod \"auth-test-model-78b5c5-kserve-5bc8bf4cf7-gptc4\" (UID: \"7e3f152e-f3d1-4e14-85a9-6f456467cf6a\") " pod="ai-tenant-e2e-shared-b-w1-78b5c5/auth-test-model-78b5c5-kserve-5bc8bf4cf7-gptc4" Nov 03 02:40:00.592158 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:00.592132 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e3f152e-f3d1-4e14-85a9-6f456467cf6a-tls-certs\") pod \"auth-test-model-78b5c5-kserve-5bc8bf4cf7-gptc4\" (UID: \"7e3f152e-f3d1-4e14-85a9-6f456467cf6a\") " pod="ai-tenant-e2e-shared-b-w1-78b5c5/auth-test-model-78b5c5-kserve-5bc8bf4cf7-gptc4" Nov 03 02:40:00.643491 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:00.643453 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgb4w\" (UniqueName: \"kubernetes.io/projected/7e3f152e-f3d1-4e14-85a9-6f456467cf6a-kube-api-access-qgb4w\") pod \"auth-test-model-78b5c5-kserve-5bc8bf4cf7-gptc4\" (UID: \"7e3f152e-f3d1-4e14-85a9-6f456467cf6a\") " pod="ai-tenant-e2e-shared-b-w1-78b5c5/auth-test-model-78b5c5-kserve-5bc8bf4cf7-gptc4" Nov 03 02:40:00.658146 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:00.658116 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-78b5c5/auth-test-model-78b5c5-kserve-5bc8bf4cf7-gptc4" Nov 03 02:40:01.030883 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:01.030803 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-78b5c5/auth-test-model-78b5c5-kserve-5bc8bf4cf7-gptc4"] Nov 03 02:40:01.037095 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:40:01.037064 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e3f152e_f3d1_4e14_85a9_6f456467cf6a.slice/crio-87e54581ae566e3c8ba6152b070686c426ef471d66c8c4b855141f54aee4e49f WatchSource:0}: Error finding container 87e54581ae566e3c8ba6152b070686c426ef471d66c8c4b855141f54aee4e49f: Status 404 returned error can't find the container with id 87e54581ae566e3c8ba6152b070686c426ef471d66c8c4b855141f54aee4e49f Nov 03 02:40:01.985755 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:01.985723 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-78b5c5/auth-test-model-78b5c5-kserve-5bc8bf4cf7-gptc4" event={"ID":"7e3f152e-f3d1-4e14-85a9-6f456467cf6a","Type":"ContainerStarted","Data":"2c805c7a180e859e26dd6526605c5357ade0f14dbac6b817f3f7f8a4e4fb44f0"} Nov 03 02:40:01.986175 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:01.985766 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-78b5c5/auth-test-model-78b5c5-kserve-5bc8bf4cf7-gptc4" event={"ID":"7e3f152e-f3d1-4e14-85a9-6f456467cf6a","Type":"ContainerStarted","Data":"87e54581ae566e3c8ba6152b070686c426ef471d66c8c4b855141f54aee4e49f"} Nov 03 02:40:02.094655 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:02.094608 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-78b5c5/auth-test-model-78b5c5-kserve-5bc8bf4cf7-gptc4" podStartSLOduration=2.094594353 podStartE2EDuration="2.094594353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:40:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:40:02.092979513 +0000 UTC m=+2879.352177668" watchObservedRunningTime="2026-11-03 02:40:02.094594353 +0000 UTC m=+2879.353792506" Nov 03 02:40:02.658694 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:02.658661 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-78b5c5/auth-test-model-78b5c5-kserve-5bc8bf4cf7-gptc4" Nov 03 02:40:04.159401 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:04.159343 5605 scope.go:122] "RemoveContainer" containerID="e4d32e09eea7895e4cc3bd8165d50e307123480badc04c606055be17268c5825" Nov 03 02:40:10.659052 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:10.659017 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-78b5c5/auth-test-model-78b5c5-kserve-5bc8bf4cf7-gptc4" Nov 03 02:40:10.671474 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:10.671452 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-78b5c5/auth-test-model-78b5c5-kserve-5bc8bf4cf7-gptc4" Nov 03 02:40:11.045653 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:11.045552 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-78b5c5/auth-test-model-78b5c5-kserve-5bc8bf4cf7-gptc4" Nov 03 02:40:23.914546 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:40:23.914506 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:40:45.300703 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:45.300669 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:40:48.469374 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:48.469319 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-ffba43/auth-test-model-ffba43-kserve-5bddd45877-ln2tc"] Nov 03 02:40:48.475661 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:48.475623 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-ffba43/auth-test-model-ffba43-kserve-5bddd45877-ln2tc" Nov 03 02:40:48.524886 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:48.524852 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-ffba43/auth-test-model-ffba43-kserve-5bddd45877-ln2tc"] Nov 03 02:40:48.567859 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:48.567826 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-ffba43\"/\"kube-root-ca.crt\"" Nov 03 02:40:48.567859 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:48.567837 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-ffba43\"/\"openshift-service-ca.crt\"" Nov 03 02:40:48.568075 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:48.567901 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-ffba43\"/\"default-dockercfg-pgrbk\"" Nov 03 02:40:48.570992 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:48.570963 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-ffba43\"/\"auth-test-model-ffba43-kserve-self-signed-certs\"" Nov 03 02:40:48.595791 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:48.595759 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0ff182d-490c-47e5-b098-e87bbf5f0e1c-model-cache\") pod \"auth-test-model-ffba43-kserve-5bddd45877-ln2tc\" (UID: \"f0ff182d-490c-47e5-b098-e87bbf5f0e1c\") " pod="ai-tenant-e2e-shared-b-w0-ffba43/auth-test-model-ffba43-kserve-5bddd45877-ln2tc" Nov 03 02:40:48.595945 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:48.595855 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0ff182d-490c-47e5-b098-e87bbf5f0e1c-tls-certs\") pod \"auth-test-model-ffba43-kserve-5bddd45877-ln2tc\" (UID: \"f0ff182d-490c-47e5-b098-e87bbf5f0e1c\") " pod="ai-tenant-e2e-shared-b-w0-ffba43/auth-test-model-ffba43-kserve-5bddd45877-ln2tc" Nov 03 02:40:48.595945 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:48.595885 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0ff182d-490c-47e5-b098-e87bbf5f0e1c-tmp-dir\") pod \"auth-test-model-ffba43-kserve-5bddd45877-ln2tc\" (UID: \"f0ff182d-490c-47e5-b098-e87bbf5f0e1c\") " pod="ai-tenant-e2e-shared-b-w0-ffba43/auth-test-model-ffba43-kserve-5bddd45877-ln2tc" Nov 03 02:40:48.596035 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:48.595980 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0ff182d-490c-47e5-b098-e87bbf5f0e1c-home\") pod \"auth-test-model-ffba43-kserve-5bddd45877-ln2tc\" (UID: \"f0ff182d-490c-47e5-b098-e87bbf5f0e1c\") " pod="ai-tenant-e2e-shared-b-w0-ffba43/auth-test-model-ffba43-kserve-5bddd45877-ln2tc" Nov 03 02:40:48.596073 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:48.596053 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wj5p\" (UniqueName: \"kubernetes.io/projected/f0ff182d-490c-47e5-b098-e87bbf5f0e1c-kube-api-access-8wj5p\") pod \"auth-test-model-ffba43-kserve-5bddd45877-ln2tc\" (UID: \"f0ff182d-490c-47e5-b098-e87bbf5f0e1c\") " pod="ai-tenant-e2e-shared-b-w0-ffba43/auth-test-model-ffba43-kserve-5bddd45877-ln2tc" Nov 03 02:40:48.596170 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:48.596147 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0ff182d-490c-47e5-b098-e87bbf5f0e1c-dshm\") pod \"auth-test-model-ffba43-kserve-5bddd45877-ln2tc\" (UID: \"f0ff182d-490c-47e5-b098-e87bbf5f0e1c\") " pod="ai-tenant-e2e-shared-b-w0-ffba43/auth-test-model-ffba43-kserve-5bddd45877-ln2tc" Nov 03 02:40:48.697646 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:48.697592 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0ff182d-490c-47e5-b098-e87bbf5f0e1c-home\") pod \"auth-test-model-ffba43-kserve-5bddd45877-ln2tc\" (UID: \"f0ff182d-490c-47e5-b098-e87bbf5f0e1c\") " pod="ai-tenant-e2e-shared-b-w0-ffba43/auth-test-model-ffba43-kserve-5bddd45877-ln2tc" Nov 03 02:40:48.697849 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:48.697729 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wj5p\" (UniqueName: \"kubernetes.io/projected/f0ff182d-490c-47e5-b098-e87bbf5f0e1c-kube-api-access-8wj5p\") pod \"auth-test-model-ffba43-kserve-5bddd45877-ln2tc\" (UID: \"f0ff182d-490c-47e5-b098-e87bbf5f0e1c\") " pod="ai-tenant-e2e-shared-b-w0-ffba43/auth-test-model-ffba43-kserve-5bddd45877-ln2tc" Nov 03 02:40:48.697849 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:48.697760 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0ff182d-490c-47e5-b098-e87bbf5f0e1c-dshm\") pod \"auth-test-model-ffba43-kserve-5bddd45877-ln2tc\" (UID: \"f0ff182d-490c-47e5-b098-e87bbf5f0e1c\") " pod="ai-tenant-e2e-shared-b-w0-ffba43/auth-test-model-ffba43-kserve-5bddd45877-ln2tc" Nov 03 02:40:48.697983 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:48.697855 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0ff182d-490c-47e5-b098-e87bbf5f0e1c-model-cache\") pod \"auth-test-model-ffba43-kserve-5bddd45877-ln2tc\" (UID: \"f0ff182d-490c-47e5-b098-e87bbf5f0e1c\") " pod="ai-tenant-e2e-shared-b-w0-ffba43/auth-test-model-ffba43-kserve-5bddd45877-ln2tc" Nov 03 02:40:48.697983 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:48.697903 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0ff182d-490c-47e5-b098-e87bbf5f0e1c-tls-certs\") pod \"auth-test-model-ffba43-kserve-5bddd45877-ln2tc\" (UID: \"f0ff182d-490c-47e5-b098-e87bbf5f0e1c\") " pod="ai-tenant-e2e-shared-b-w0-ffba43/auth-test-model-ffba43-kserve-5bddd45877-ln2tc" Nov 03 02:40:48.697983 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:48.697928 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0ff182d-490c-47e5-b098-e87bbf5f0e1c-tmp-dir\") pod \"auth-test-model-ffba43-kserve-5bddd45877-ln2tc\" (UID: \"f0ff182d-490c-47e5-b098-e87bbf5f0e1c\") " pod="ai-tenant-e2e-shared-b-w0-ffba43/auth-test-model-ffba43-kserve-5bddd45877-ln2tc" Nov 03 02:40:48.698149 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:48.698009 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0ff182d-490c-47e5-b098-e87bbf5f0e1c-home\") pod \"auth-test-model-ffba43-kserve-5bddd45877-ln2tc\" (UID: \"f0ff182d-490c-47e5-b098-e87bbf5f0e1c\") " pod="ai-tenant-e2e-shared-b-w0-ffba43/auth-test-model-ffba43-kserve-5bddd45877-ln2tc" Nov 03 02:40:48.698413 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:48.698382 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0ff182d-490c-47e5-b098-e87bbf5f0e1c-model-cache\") pod \"auth-test-model-ffba43-kserve-5bddd45877-ln2tc\" (UID: \"f0ff182d-490c-47e5-b098-e87bbf5f0e1c\") " pod="ai-tenant-e2e-shared-b-w0-ffba43/auth-test-model-ffba43-kserve-5bddd45877-ln2tc" Nov 03 02:40:48.698413 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:48.698394 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0ff182d-490c-47e5-b098-e87bbf5f0e1c-tmp-dir\") pod \"auth-test-model-ffba43-kserve-5bddd45877-ln2tc\" (UID: \"f0ff182d-490c-47e5-b098-e87bbf5f0e1c\") " pod="ai-tenant-e2e-shared-b-w0-ffba43/auth-test-model-ffba43-kserve-5bddd45877-ln2tc" Nov 03 02:40:48.700195 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:48.700171 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0ff182d-490c-47e5-b098-e87bbf5f0e1c-dshm\") pod \"auth-test-model-ffba43-kserve-5bddd45877-ln2tc\" (UID: \"f0ff182d-490c-47e5-b098-e87bbf5f0e1c\") " pod="ai-tenant-e2e-shared-b-w0-ffba43/auth-test-model-ffba43-kserve-5bddd45877-ln2tc" Nov 03 02:40:48.700777 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:48.700758 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0ff182d-490c-47e5-b098-e87bbf5f0e1c-tls-certs\") pod \"auth-test-model-ffba43-kserve-5bddd45877-ln2tc\" (UID: \"f0ff182d-490c-47e5-b098-e87bbf5f0e1c\") " pod="ai-tenant-e2e-shared-b-w0-ffba43/auth-test-model-ffba43-kserve-5bddd45877-ln2tc" Nov 03 02:40:48.752115 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:48.752020 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wj5p\" (UniqueName: \"kubernetes.io/projected/f0ff182d-490c-47e5-b098-e87bbf5f0e1c-kube-api-access-8wj5p\") pod \"auth-test-model-ffba43-kserve-5bddd45877-ln2tc\" (UID: \"f0ff182d-490c-47e5-b098-e87bbf5f0e1c\") " pod="ai-tenant-e2e-shared-b-w0-ffba43/auth-test-model-ffba43-kserve-5bddd45877-ln2tc" Nov 03 02:40:48.787886 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:48.787853 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-ffba43/auth-test-model-ffba43-kserve-5bddd45877-ln2tc" Nov 03 02:40:49.170035 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:49.170002 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-ffba43/auth-test-model-ffba43-kserve-5bddd45877-ln2tc"] Nov 03 02:40:49.175469 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:40:49.175444 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf0ff182d_490c_47e5_b098_e87bbf5f0e1c.slice/crio-716017d907a27efd9f4832ab3ff2bd3792237a8e8829eaa912597c847e2e771d WatchSource:0}: Error finding container 716017d907a27efd9f4832ab3ff2bd3792237a8e8829eaa912597c847e2e771d: Status 404 returned error can't find the container with id 716017d907a27efd9f4832ab3ff2bd3792237a8e8829eaa912597c847e2e771d Nov 03 02:40:49.242870 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:49.242841 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-ffba43/auth-test-model-ffba43-kserve-5bddd45877-ln2tc" event={"ID":"f0ff182d-490c-47e5-b098-e87bbf5f0e1c","Type":"ContainerStarted","Data":"1fb2107621324bc4809f1eec256547c30696316013b5bfd40edab3286cbdbd03"} Nov 03 02:40:49.242993 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:49.242889 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-ffba43/auth-test-model-ffba43-kserve-5bddd45877-ln2tc" event={"ID":"f0ff182d-490c-47e5-b098-e87bbf5f0e1c","Type":"ContainerStarted","Data":"716017d907a27efd9f4832ab3ff2bd3792237a8e8829eaa912597c847e2e771d"} Nov 03 02:40:49.350957 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:49.350894 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-ffba43/auth-test-model-ffba43-kserve-5bddd45877-ln2tc" podStartSLOduration=1.350875788 podStartE2EDuration="1.350875788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:40:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:40:49.350069088 +0000 UTC m=+2926.609267255" watchObservedRunningTime="2026-11-03 02:40:49.350875788 +0000 UTC m=+2926.610073943" Nov 03 02:40:49.788006 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:49.787973 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-ffba43/auth-test-model-ffba43-kserve-5bddd45877-ln2tc" Nov 03 02:40:49.822442 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:49.822410 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s9dp7"] Nov 03 02:40:49.822649 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:49.822625 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s9dp7" podUID="c3ef2131-720c-4db4-8c4f-e618edf4b5e9" containerName="istio-proxy" containerID="cri-o://3fdf31d4ddd19f1cda9ab61a21b75e60d82c89698c3f569c1ba5ccc47daacc3c" gracePeriod=30 Nov 03 02:40:53.745415 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:53.745380 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kcdqm"] Nov 03 02:40:53.745859 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:53.745569 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kcdqm" podUID="2ad21c46-2e06-4b88-a787-0ca89fde517d" containerName="discovery" containerID="cri-o://776be96a343db2ef38ca0cbec040649bcd4ae3422810ff3a5ce8fd71aa4bdd1d" gracePeriod=30 Nov 03 02:40:53.917627 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:40:53.917588 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:40:54.023051 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:54.023021 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kcdqm" Nov 03 02:40:54.159270 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:54.159228 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ad21c46-2e06-4b88-a787-0ca89fde517d-kube-api-access-mr4qg\" (UniqueName: \"kubernetes.io/projected/2ad21c46-2e06-4b88-a787-0ca89fde517d-kube-api-access-mr4qg\") pod \"2ad21c46-2e06-4b88-a787-0ca89fde517d\" (UID: \"2ad21c46-2e06-4b88-a787-0ca89fde517d\") " Nov 03 02:40:54.159470 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:54.159338 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ad21c46-2e06-4b88-a787-0ca89fde517d-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2ad21c46-2e06-4b88-a787-0ca89fde517d-istio-csr-dns-cert\") pod \"2ad21c46-2e06-4b88-a787-0ca89fde517d\" (UID: \"2ad21c46-2e06-4b88-a787-0ca89fde517d\") " Nov 03 02:40:54.159470 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:54.159448 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ad21c46-2e06-4b88-a787-0ca89fde517d-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ad21c46-2e06-4b88-a787-0ca89fde517d-local-certs\") pod \"2ad21c46-2e06-4b88-a787-0ca89fde517d\" (UID: \"2ad21c46-2e06-4b88-a787-0ca89fde517d\") " Nov 03 02:40:54.159602 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:54.159521 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ad21c46-2e06-4b88-a787-0ca89fde517d-istio-token\" (UniqueName: \"kubernetes.io/projected/2ad21c46-2e06-4b88-a787-0ca89fde517d-istio-token\") pod \"2ad21c46-2e06-4b88-a787-0ca89fde517d\" (UID: \"2ad21c46-2e06-4b88-a787-0ca89fde517d\") " Nov 03 02:40:54.159602 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:54.159565 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ad21c46-2e06-4b88-a787-0ca89fde517d-cacerts\" (UniqueName: \"kubernetes.io/secret/2ad21c46-2e06-4b88-a787-0ca89fde517d-cacerts\") pod \"2ad21c46-2e06-4b88-a787-0ca89fde517d\" (UID: \"2ad21c46-2e06-4b88-a787-0ca89fde517d\") " Nov 03 02:40:54.159718 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:54.159625 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ad21c46-2e06-4b88-a787-0ca89fde517d-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2ad21c46-2e06-4b88-a787-0ca89fde517d-istio-kubeconfig\") pod \"2ad21c46-2e06-4b88-a787-0ca89fde517d\" (UID: \"2ad21c46-2e06-4b88-a787-0ca89fde517d\") " Nov 03 02:40:54.159718 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:54.159661 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ad21c46-2e06-4b88-a787-0ca89fde517d-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2ad21c46-2e06-4b88-a787-0ca89fde517d-istio-csr-ca-configmap\") pod \"2ad21c46-2e06-4b88-a787-0ca89fde517d\" (UID: \"2ad21c46-2e06-4b88-a787-0ca89fde517d\") " Nov 03 02:40:54.160552 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:54.160521 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ad21c46-2e06-4b88-a787-0ca89fde517d-istio-csr-ca-configmap" pod "2ad21c46-2e06-4b88-a787-0ca89fde517d" (UID: "2ad21c46-2e06-4b88-a787-0ca89fde517d"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:40:54.162251 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:54.162211 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ad21c46-2e06-4b88-a787-0ca89fde517d-istio-token" pod "2ad21c46-2e06-4b88-a787-0ca89fde517d" (UID: "2ad21c46-2e06-4b88-a787-0ca89fde517d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:40:54.162484 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:54.162446 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ad21c46-2e06-4b88-a787-0ca89fde517d-kube-api-access-mr4qg" pod "2ad21c46-2e06-4b88-a787-0ca89fde517d" (UID: "2ad21c46-2e06-4b88-a787-0ca89fde517d"). InnerVolumeSpecName "kube-api-access-mr4qg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:40:54.162484 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:54.162464 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ad21c46-2e06-4b88-a787-0ca89fde517d-local-certs" pod "2ad21c46-2e06-4b88-a787-0ca89fde517d" (UID: "2ad21c46-2e06-4b88-a787-0ca89fde517d"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:40:54.163080 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:54.163059 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ad21c46-2e06-4b88-a787-0ca89fde517d-istio-csr-dns-cert" pod "2ad21c46-2e06-4b88-a787-0ca89fde517d" (UID: "2ad21c46-2e06-4b88-a787-0ca89fde517d"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:40:54.163409 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:54.163351 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ad21c46-2e06-4b88-a787-0ca89fde517d-istio-kubeconfig" pod "2ad21c46-2e06-4b88-a787-0ca89fde517d" (UID: "2ad21c46-2e06-4b88-a787-0ca89fde517d"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:40:54.163409 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:54.163396 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ad21c46-2e06-4b88-a787-0ca89fde517d-cacerts" pod "2ad21c46-2e06-4b88-a787-0ca89fde517d" (UID: "2ad21c46-2e06-4b88-a787-0ca89fde517d"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:40:54.261183 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:54.261098 5605 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2ad21c46-2e06-4b88-a787-0ca89fde517d-istio-csr-dns-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:54.261183 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:54.261130 5605 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ad21c46-2e06-4b88-a787-0ca89fde517d-local-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:54.261183 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:54.261153 5605 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ad21c46-2e06-4b88-a787-0ca89fde517d-istio-token\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:54.261183 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:54.261164 5605 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2ad21c46-2e06-4b88-a787-0ca89fde517d-cacerts\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:54.261183 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:54.261174 5605 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2ad21c46-2e06-4b88-a787-0ca89fde517d-istio-kubeconfig\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:54.261183 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:54.261185 5605 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2ad21c46-2e06-4b88-a787-0ca89fde517d-istio-csr-ca-configmap\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:54.261555 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:54.261195 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mr4qg\" (UniqueName: \"kubernetes.io/projected/2ad21c46-2e06-4b88-a787-0ca89fde517d-kube-api-access-mr4qg\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:54.271992 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:54.271956 5605 generic.go:356] "Generic (PLEG): container finished" podID="2ad21c46-2e06-4b88-a787-0ca89fde517d" containerID="776be96a343db2ef38ca0cbec040649bcd4ae3422810ff3a5ce8fd71aa4bdd1d" exitCode=0 Nov 03 02:40:54.272143 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:54.272009 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kcdqm" Nov 03 02:40:54.272143 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:54.272035 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kcdqm" event={"ID":"2ad21c46-2e06-4b88-a787-0ca89fde517d","Type":"ContainerDied","Data":"776be96a343db2ef38ca0cbec040649bcd4ae3422810ff3a5ce8fd71aa4bdd1d"} Nov 03 02:40:54.272143 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:54.272088 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kcdqm" event={"ID":"2ad21c46-2e06-4b88-a787-0ca89fde517d","Type":"ContainerDied","Data":"fa088bb9d9021e5798cc520085f45384d300b5add3a4db59d68387c2f5648749"} Nov 03 02:40:54.272143 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:54.272114 5605 scope.go:122] "RemoveContainer" containerID="776be96a343db2ef38ca0cbec040649bcd4ae3422810ff3a5ce8fd71aa4bdd1d" Nov 03 02:40:54.283152 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:54.283133 5605 scope.go:122] "RemoveContainer" containerID="776be96a343db2ef38ca0cbec040649bcd4ae3422810ff3a5ce8fd71aa4bdd1d" Nov 03 02:40:54.283473 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:40:54.283452 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"776be96a343db2ef38ca0cbec040649bcd4ae3422810ff3a5ce8fd71aa4bdd1d\": container with ID starting with 776be96a343db2ef38ca0cbec040649bcd4ae3422810ff3a5ce8fd71aa4bdd1d not found: ID does not exist" containerID="776be96a343db2ef38ca0cbec040649bcd4ae3422810ff3a5ce8fd71aa4bdd1d" Nov 03 02:40:54.283559 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:54.283483 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"776be96a343db2ef38ca0cbec040649bcd4ae3422810ff3a5ce8fd71aa4bdd1d"} err="failed to get container status \"776be96a343db2ef38ca0cbec040649bcd4ae3422810ff3a5ce8fd71aa4bdd1d\": rpc error: code = NotFound desc = could not find container \"776be96a343db2ef38ca0cbec040649bcd4ae3422810ff3a5ce8fd71aa4bdd1d\": container with ID starting with 776be96a343db2ef38ca0cbec040649bcd4ae3422810ff3a5ce8fd71aa4bdd1d not found: ID does not exist" Nov 03 02:40:54.427186 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:54.427153 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kcdqm"] Nov 03 02:40:54.432036 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:54.432005 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kcdqm"] Nov 03 02:40:54.850883 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:54.850856 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ad21c46-2e06-4b88-a787-0ca89fde517d" path="/var/lib/kubelet/pods/2ad21c46-2e06-4b88-a787-0ca89fde517d/volumes" Nov 03 02:40:55.087400 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:55.087306 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s9dp7" Nov 03 02:40:55.171457 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:55.171424 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3ef2131-720c-4db4-8c4f-e618edf4b5e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3ef2131-720c-4db4-8c4f-e618edf4b5e9-istiod-ca-cert\") pod \"c3ef2131-720c-4db4-8c4f-e618edf4b5e9\" (UID: \"c3ef2131-720c-4db4-8c4f-e618edf4b5e9\") " Nov 03 02:40:55.171644 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:55.171469 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c3ef2131-720c-4db4-8c4f-e618edf4b5e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3ef2131-720c-4db4-8c4f-e618edf4b5e9-istio-podinfo\") pod \"c3ef2131-720c-4db4-8c4f-e618edf4b5e9\" (UID: \"c3ef2131-720c-4db4-8c4f-e618edf4b5e9\") " Nov 03 02:40:55.171644 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:55.171562 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3ef2131-720c-4db4-8c4f-e618edf4b5e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ef2131-720c-4db4-8c4f-e618edf4b5e9-workload-socket\") pod \"c3ef2131-720c-4db4-8c4f-e618edf4b5e9\" (UID: \"c3ef2131-720c-4db4-8c4f-e618edf4b5e9\") " Nov 03 02:40:55.171644 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:55.171584 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3ef2131-720c-4db4-8c4f-e618edf4b5e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ef2131-720c-4db4-8c4f-e618edf4b5e9-credential-socket\") pod \"c3ef2131-720c-4db4-8c4f-e618edf4b5e9\" (UID: \"c3ef2131-720c-4db4-8c4f-e618edf4b5e9\") " Nov 03 02:40:55.171644 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:55.171614 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3ef2131-720c-4db4-8c4f-e618edf4b5e9-istio-token\" (UniqueName: \"kubernetes.io/projected/c3ef2131-720c-4db4-8c4f-e618edf4b5e9-istio-token\") pod \"c3ef2131-720c-4db4-8c4f-e618edf4b5e9\" (UID: \"c3ef2131-720c-4db4-8c4f-e618edf4b5e9\") " Nov 03 02:40:55.171837 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:55.171683 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3ef2131-720c-4db4-8c4f-e618edf4b5e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3ef2131-720c-4db4-8c4f-e618edf4b5e9-istio-envoy\") pod \"c3ef2131-720c-4db4-8c4f-e618edf4b5e9\" (UID: \"c3ef2131-720c-4db4-8c4f-e618edf4b5e9\") " Nov 03 02:40:55.171837 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:55.171718 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3ef2131-720c-4db4-8c4f-e618edf4b5e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3ef2131-720c-4db4-8c4f-e618edf4b5e9-workload-certs\") pod \"c3ef2131-720c-4db4-8c4f-e618edf4b5e9\" (UID: \"c3ef2131-720c-4db4-8c4f-e618edf4b5e9\") " Nov 03 02:40:55.171958 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:55.171878 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3ef2131-720c-4db4-8c4f-e618edf4b5e9-workload-socket" pod "c3ef2131-720c-4db4-8c4f-e618edf4b5e9" (UID: "c3ef2131-720c-4db4-8c4f-e618edf4b5e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:40:55.171958 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:55.171928 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3ef2131-720c-4db4-8c4f-e618edf4b5e9-credential-socket" pod "c3ef2131-720c-4db4-8c4f-e618edf4b5e9" (UID: "c3ef2131-720c-4db4-8c4f-e618edf4b5e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:40:55.171958 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:55.171937 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3ef2131-720c-4db4-8c4f-e618edf4b5e9-istiod-ca-cert" pod "c3ef2131-720c-4db4-8c4f-e618edf4b5e9" (UID: "c3ef2131-720c-4db4-8c4f-e618edf4b5e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:40:55.172129 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:55.172022 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3ef2131-720c-4db4-8c4f-e618edf4b5e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3ef2131-720c-4db4-8c4f-e618edf4b5e9-istio-data\") pod \"c3ef2131-720c-4db4-8c4f-e618edf4b5e9\" (UID: \"c3ef2131-720c-4db4-8c4f-e618edf4b5e9\") " Nov 03 02:40:55.172129 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:55.172080 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3ef2131-720c-4db4-8c4f-e618edf4b5e9-kube-api-access-79pd6\" (UniqueName: \"kubernetes.io/projected/c3ef2131-720c-4db4-8c4f-e618edf4b5e9-kube-api-access-79pd6\") pod \"c3ef2131-720c-4db4-8c4f-e618edf4b5e9\" (UID: \"c3ef2131-720c-4db4-8c4f-e618edf4b5e9\") " Nov 03 02:40:55.172240 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:55.172172 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3ef2131-720c-4db4-8c4f-e618edf4b5e9-workload-certs" pod "c3ef2131-720c-4db4-8c4f-e618edf4b5e9" (UID: "c3ef2131-720c-4db4-8c4f-e618edf4b5e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:40:55.172393 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:55.172339 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3ef2131-720c-4db4-8c4f-e618edf4b5e9-istio-data" pod "c3ef2131-720c-4db4-8c4f-e618edf4b5e9" (UID: "c3ef2131-720c-4db4-8c4f-e618edf4b5e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:40:55.172696 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:55.172674 5605 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ef2131-720c-4db4-8c4f-e618edf4b5e9-workload-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:55.172790 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:55.172705 5605 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ef2131-720c-4db4-8c4f-e618edf4b5e9-credential-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:55.172790 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:55.172724 5605 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3ef2131-720c-4db4-8c4f-e618edf4b5e9-workload-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:55.172790 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:55.172738 5605 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3ef2131-720c-4db4-8c4f-e618edf4b5e9-istio-data\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:55.172790 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:55.172751 5605 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3ef2131-720c-4db4-8c4f-e618edf4b5e9-istiod-ca-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:55.173990 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:55.173965 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3ef2131-720c-4db4-8c4f-e618edf4b5e9-istio-envoy" pod "c3ef2131-720c-4db4-8c4f-e618edf4b5e9" (UID: "c3ef2131-720c-4db4-8c4f-e618edf4b5e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:40:55.174095 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:55.174031 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3ef2131-720c-4db4-8c4f-e618edf4b5e9-istio-podinfo" pod "c3ef2131-720c-4db4-8c4f-e618edf4b5e9" (UID: "c3ef2131-720c-4db4-8c4f-e618edf4b5e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:40:55.174188 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:55.174167 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3ef2131-720c-4db4-8c4f-e618edf4b5e9-kube-api-access-79pd6" pod "c3ef2131-720c-4db4-8c4f-e618edf4b5e9" (UID: "c3ef2131-720c-4db4-8c4f-e618edf4b5e9"). InnerVolumeSpecName "kube-api-access-79pd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:40:55.174621 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:55.174595 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3ef2131-720c-4db4-8c4f-e618edf4b5e9-istio-token" pod "c3ef2131-720c-4db4-8c4f-e618edf4b5e9" (UID: "c3ef2131-720c-4db4-8c4f-e618edf4b5e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:40:55.273579 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:55.273545 5605 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3ef2131-720c-4db4-8c4f-e618edf4b5e9-istio-podinfo\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:55.273579 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:55.273576 5605 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3ef2131-720c-4db4-8c4f-e618edf4b5e9-istio-token\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:55.273579 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:55.273589 5605 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3ef2131-720c-4db4-8c4f-e618edf4b5e9-istio-envoy\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:55.273909 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:55.273602 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79pd6\" (UniqueName: \"kubernetes.io/projected/c3ef2131-720c-4db4-8c4f-e618edf4b5e9-kube-api-access-79pd6\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:55.279480 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:55.279449 5605 generic.go:356] "Generic (PLEG): container finished" podID="c3ef2131-720c-4db4-8c4f-e618edf4b5e9" containerID="3fdf31d4ddd19f1cda9ab61a21b75e60d82c89698c3f569c1ba5ccc47daacc3c" exitCode=0 Nov 03 02:40:55.279659 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:55.279499 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s9dp7" Nov 03 02:40:55.279659 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:55.279539 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s9dp7" event={"ID":"c3ef2131-720c-4db4-8c4f-e618edf4b5e9","Type":"ContainerDied","Data":"3fdf31d4ddd19f1cda9ab61a21b75e60d82c89698c3f569c1ba5ccc47daacc3c"} Nov 03 02:40:55.279659 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:55.279607 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s9dp7" event={"ID":"c3ef2131-720c-4db4-8c4f-e618edf4b5e9","Type":"ContainerDied","Data":"6227d73ecd88a25dd8ecc6867d767b3923ef0c304922f111c280b3c3057bd0de"} Nov 03 02:40:55.279659 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:55.279636 5605 scope.go:122] "RemoveContainer" containerID="3fdf31d4ddd19f1cda9ab61a21b75e60d82c89698c3f569c1ba5ccc47daacc3c" Nov 03 02:40:55.290158 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:55.290140 5605 scope.go:122] "RemoveContainer" containerID="3fdf31d4ddd19f1cda9ab61a21b75e60d82c89698c3f569c1ba5ccc47daacc3c" Nov 03 02:40:55.290453 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:40:55.290432 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fdf31d4ddd19f1cda9ab61a21b75e60d82c89698c3f569c1ba5ccc47daacc3c\": container with ID starting with 3fdf31d4ddd19f1cda9ab61a21b75e60d82c89698c3f569c1ba5ccc47daacc3c not found: ID does not exist" containerID="3fdf31d4ddd19f1cda9ab61a21b75e60d82c89698c3f569c1ba5ccc47daacc3c" Nov 03 02:40:55.290502 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:55.290460 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fdf31d4ddd19f1cda9ab61a21b75e60d82c89698c3f569c1ba5ccc47daacc3c"} err="failed to get container status \"3fdf31d4ddd19f1cda9ab61a21b75e60d82c89698c3f569c1ba5ccc47daacc3c\": rpc error: code = NotFound desc = could not find container \"3fdf31d4ddd19f1cda9ab61a21b75e60d82c89698c3f569c1ba5ccc47daacc3c\": container with ID starting with 3fdf31d4ddd19f1cda9ab61a21b75e60d82c89698c3f569c1ba5ccc47daacc3c not found: ID does not exist" Nov 03 02:40:55.434649 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:55.434567 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s9dp7"] Nov 03 02:40:55.439367 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:55.439324 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s9dp7"] Nov 03 02:40:56.850048 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:56.850011 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3ef2131-720c-4db4-8c4f-e618edf4b5e9" path="/var/lib/kubelet/pods/c3ef2131-720c-4db4-8c4f-e618edf4b5e9/volumes" Nov 03 02:40:58.788161 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:58.788120 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-ffba43/auth-test-model-ffba43-kserve-5bddd45877-ln2tc" Nov 03 02:40:58.800684 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:58.800655 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-ffba43/auth-test-model-ffba43-kserve-5bddd45877-ln2tc" Nov 03 02:40:59.315023 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:40:59.314993 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-ffba43/auth-test-model-ffba43-kserve-5bddd45877-ln2tc" Nov 03 02:41:08.895670 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:41:08.895639 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-957b33/auth-test-model-957b33-kserve-c8778cdbf-dlqkf"] Nov 03 02:41:08.896075 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:41:08.896063 5605 cpu_manager.go:414] "containerMap: removing container" podUID="c3ef2131-720c-4db4-8c4f-e618edf4b5e9" containerName="istio-proxy" Nov 03 02:41:08.896114 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:41:08.896076 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3ef2131-720c-4db4-8c4f-e618edf4b5e9" containerName="istio-proxy" Nov 03 02:41:08.896161 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:41:08.896128 5605 cpu_manager.go:414] "containerMap: removing container" podUID="2ad21c46-2e06-4b88-a787-0ca89fde517d" containerName="discovery" Nov 03 02:41:08.896161 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:41:08.896133 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ad21c46-2e06-4b88-a787-0ca89fde517d" containerName="discovery" Nov 03 02:41:08.896273 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:41:08.896207 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ad21c46-2e06-4b88-a787-0ca89fde517d" containerName="discovery" Nov 03 02:41:08.896273 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:41:08.896214 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3ef2131-720c-4db4-8c4f-e618edf4b5e9" containerName="istio-proxy" Nov 03 02:41:08.900733 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:41:08.900711 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-957b33/auth-test-model-957b33-kserve-c8778cdbf-dlqkf" Nov 03 02:41:08.998242 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:41:08.998194 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-957b33\"/\"default-dockercfg-gpnj4\"" Nov 03 02:41:08.998242 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:41:08.998211 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-957b33\"/\"auth-test-model-957b33-kserve-self-signed-certs\"" Nov 03 02:41:08.998500 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:41:08.998250 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-957b33\"/\"kube-root-ca.crt\"" Nov 03 02:41:08.998500 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:41:08.998270 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-957b33\"/\"openshift-service-ca.crt\"" Nov 03 02:41:09.000329 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:41:09.000310 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-957b33/auth-test-model-957b33-kserve-c8778cdbf-dlqkf"] Nov 03 02:41:09.025819 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:41:09.025793 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4hr9\" (UniqueName: \"kubernetes.io/projected/eddf0af3-8acf-4e49-bd31-a8235ab2e03b-kube-api-access-n4hr9\") pod \"auth-test-model-957b33-kserve-c8778cdbf-dlqkf\" (UID: \"eddf0af3-8acf-4e49-bd31-a8235ab2e03b\") " pod="ai-tenant-e2e-shared-b-w2-957b33/auth-test-model-957b33-kserve-c8778cdbf-dlqkf" Nov 03 02:41:09.025950 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:41:09.025893 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eddf0af3-8acf-4e49-bd31-a8235ab2e03b-home\") pod \"auth-test-model-957b33-kserve-c8778cdbf-dlqkf\" (UID: \"eddf0af3-8acf-4e49-bd31-a8235ab2e03b\") " pod="ai-tenant-e2e-shared-b-w2-957b33/auth-test-model-957b33-kserve-c8778cdbf-dlqkf" Nov 03 02:41:09.026005 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:41:09.025977 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eddf0af3-8acf-4e49-bd31-a8235ab2e03b-model-cache\") pod \"auth-test-model-957b33-kserve-c8778cdbf-dlqkf\" (UID: \"eddf0af3-8acf-4e49-bd31-a8235ab2e03b\") " pod="ai-tenant-e2e-shared-b-w2-957b33/auth-test-model-957b33-kserve-c8778cdbf-dlqkf" Nov 03 02:41:09.026053 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:41:09.026021 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eddf0af3-8acf-4e49-bd31-a8235ab2e03b-tls-certs\") pod \"auth-test-model-957b33-kserve-c8778cdbf-dlqkf\" (UID: \"eddf0af3-8acf-4e49-bd31-a8235ab2e03b\") " pod="ai-tenant-e2e-shared-b-w2-957b33/auth-test-model-957b33-kserve-c8778cdbf-dlqkf" Nov 03 02:41:09.026149 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:41:09.026131 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eddf0af3-8acf-4e49-bd31-a8235ab2e03b-dshm\") pod \"auth-test-model-957b33-kserve-c8778cdbf-dlqkf\" (UID: \"eddf0af3-8acf-4e49-bd31-a8235ab2e03b\") " pod="ai-tenant-e2e-shared-b-w2-957b33/auth-test-model-957b33-kserve-c8778cdbf-dlqkf" Nov 03 02:41:09.026196 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:41:09.026183 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eddf0af3-8acf-4e49-bd31-a8235ab2e03b-tmp-dir\") pod \"auth-test-model-957b33-kserve-c8778cdbf-dlqkf\" (UID: \"eddf0af3-8acf-4e49-bd31-a8235ab2e03b\") " pod="ai-tenant-e2e-shared-b-w2-957b33/auth-test-model-957b33-kserve-c8778cdbf-dlqkf" Nov 03 02:41:09.127988 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:41:09.127951 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4hr9\" (UniqueName: \"kubernetes.io/projected/eddf0af3-8acf-4e49-bd31-a8235ab2e03b-kube-api-access-n4hr9\") pod \"auth-test-model-957b33-kserve-c8778cdbf-dlqkf\" (UID: \"eddf0af3-8acf-4e49-bd31-a8235ab2e03b\") " pod="ai-tenant-e2e-shared-b-w2-957b33/auth-test-model-957b33-kserve-c8778cdbf-dlqkf" Nov 03 02:41:09.128179 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:41:09.128052 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eddf0af3-8acf-4e49-bd31-a8235ab2e03b-home\") pod \"auth-test-model-957b33-kserve-c8778cdbf-dlqkf\" (UID: \"eddf0af3-8acf-4e49-bd31-a8235ab2e03b\") " pod="ai-tenant-e2e-shared-b-w2-957b33/auth-test-model-957b33-kserve-c8778cdbf-dlqkf" Nov 03 02:41:09.128179 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:41:09.128090 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eddf0af3-8acf-4e49-bd31-a8235ab2e03b-model-cache\") pod \"auth-test-model-957b33-kserve-c8778cdbf-dlqkf\" (UID: \"eddf0af3-8acf-4e49-bd31-a8235ab2e03b\") " pod="ai-tenant-e2e-shared-b-w2-957b33/auth-test-model-957b33-kserve-c8778cdbf-dlqkf" Nov 03 02:41:09.128179 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:41:09.128128 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eddf0af3-8acf-4e49-bd31-a8235ab2e03b-tls-certs\") pod \"auth-test-model-957b33-kserve-c8778cdbf-dlqkf\" (UID: \"eddf0af3-8acf-4e49-bd31-a8235ab2e03b\") " pod="ai-tenant-e2e-shared-b-w2-957b33/auth-test-model-957b33-kserve-c8778cdbf-dlqkf" Nov 03 02:41:09.128179 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:41:09.128170 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eddf0af3-8acf-4e49-bd31-a8235ab2e03b-dshm\") pod \"auth-test-model-957b33-kserve-c8778cdbf-dlqkf\" (UID: \"eddf0af3-8acf-4e49-bd31-a8235ab2e03b\") " pod="ai-tenant-e2e-shared-b-w2-957b33/auth-test-model-957b33-kserve-c8778cdbf-dlqkf" Nov 03 02:41:09.128425 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:41:09.128220 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eddf0af3-8acf-4e49-bd31-a8235ab2e03b-tmp-dir\") pod \"auth-test-model-957b33-kserve-c8778cdbf-dlqkf\" (UID: \"eddf0af3-8acf-4e49-bd31-a8235ab2e03b\") " pod="ai-tenant-e2e-shared-b-w2-957b33/auth-test-model-957b33-kserve-c8778cdbf-dlqkf" Nov 03 02:41:09.128579 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:41:09.128553 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eddf0af3-8acf-4e49-bd31-a8235ab2e03b-home\") pod \"auth-test-model-957b33-kserve-c8778cdbf-dlqkf\" (UID: \"eddf0af3-8acf-4e49-bd31-a8235ab2e03b\") " pod="ai-tenant-e2e-shared-b-w2-957b33/auth-test-model-957b33-kserve-c8778cdbf-dlqkf" Nov 03 02:41:09.128639 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:41:09.128576 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eddf0af3-8acf-4e49-bd31-a8235ab2e03b-model-cache\") pod \"auth-test-model-957b33-kserve-c8778cdbf-dlqkf\" (UID: \"eddf0af3-8acf-4e49-bd31-a8235ab2e03b\") " pod="ai-tenant-e2e-shared-b-w2-957b33/auth-test-model-957b33-kserve-c8778cdbf-dlqkf" Nov 03 02:41:09.128679 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:41:09.128643 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eddf0af3-8acf-4e49-bd31-a8235ab2e03b-tmp-dir\") pod \"auth-test-model-957b33-kserve-c8778cdbf-dlqkf\" (UID: \"eddf0af3-8acf-4e49-bd31-a8235ab2e03b\") " pod="ai-tenant-e2e-shared-b-w2-957b33/auth-test-model-957b33-kserve-c8778cdbf-dlqkf" Nov 03 02:41:09.130485 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:41:09.130462 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eddf0af3-8acf-4e49-bd31-a8235ab2e03b-dshm\") pod \"auth-test-model-957b33-kserve-c8778cdbf-dlqkf\" (UID: \"eddf0af3-8acf-4e49-bd31-a8235ab2e03b\") " pod="ai-tenant-e2e-shared-b-w2-957b33/auth-test-model-957b33-kserve-c8778cdbf-dlqkf" Nov 03 02:41:09.130740 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:41:09.130721 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eddf0af3-8acf-4e49-bd31-a8235ab2e03b-tls-certs\") pod \"auth-test-model-957b33-kserve-c8778cdbf-dlqkf\" (UID: \"eddf0af3-8acf-4e49-bd31-a8235ab2e03b\") " pod="ai-tenant-e2e-shared-b-w2-957b33/auth-test-model-957b33-kserve-c8778cdbf-dlqkf" Nov 03 02:41:09.182207 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:41:09.182129 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4hr9\" (UniqueName: \"kubernetes.io/projected/eddf0af3-8acf-4e49-bd31-a8235ab2e03b-kube-api-access-n4hr9\") pod \"auth-test-model-957b33-kserve-c8778cdbf-dlqkf\" (UID: \"eddf0af3-8acf-4e49-bd31-a8235ab2e03b\") " pod="ai-tenant-e2e-shared-b-w2-957b33/auth-test-model-957b33-kserve-c8778cdbf-dlqkf" Nov 03 02:41:09.213171 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:41:09.213141 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-957b33/auth-test-model-957b33-kserve-c8778cdbf-dlqkf" Nov 03 02:41:09.603021 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:41:09.602990 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-957b33/auth-test-model-957b33-kserve-c8778cdbf-dlqkf"] Nov 03 02:41:09.608561 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:41:09.608529 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeddf0af3_8acf_4e49_bd31_a8235ab2e03b.slice/crio-4d02ffc5f80f36fdb513a3a2095913086e3b753390b33dbdf6214bd4923d9177 WatchSource:0}: Error finding container 4d02ffc5f80f36fdb513a3a2095913086e3b753390b33dbdf6214bd4923d9177: Status 404 returned error can't find the container with id 4d02ffc5f80f36fdb513a3a2095913086e3b753390b33dbdf6214bd4923d9177 Nov 03 02:41:10.379774 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:41:10.379737 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-957b33/auth-test-model-957b33-kserve-c8778cdbf-dlqkf" event={"ID":"eddf0af3-8acf-4e49-bd31-a8235ab2e03b","Type":"ContainerStarted","Data":"5b950fc72dcce6d2ae8980468caeadcc47e16957c80abbe1d2adc296455ff298"} Nov 03 02:41:10.380177 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:41:10.379788 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-957b33/auth-test-model-957b33-kserve-c8778cdbf-dlqkf" event={"ID":"eddf0af3-8acf-4e49-bd31-a8235ab2e03b","Type":"ContainerStarted","Data":"4d02ffc5f80f36fdb513a3a2095913086e3b753390b33dbdf6214bd4923d9177"} Nov 03 02:41:10.489408 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:41:10.489347 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-957b33/auth-test-model-957b33-kserve-c8778cdbf-dlqkf" podStartSLOduration=2.489333248 podStartE2EDuration="2.489333248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:41:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:41:10.488584303 +0000 UTC m=+2947.747782457" watchObservedRunningTime="2026-11-03 02:41:10.489333248 +0000 UTC m=+2947.748531401" Nov 03 02:41:11.214086 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:41:11.214046 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-957b33/auth-test-model-957b33-kserve-c8778cdbf-dlqkf" Nov 03 02:41:19.213739 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:41:19.213696 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-957b33/auth-test-model-957b33-kserve-c8778cdbf-dlqkf" Nov 03 02:41:19.226815 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:41:19.226788 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-957b33/auth-test-model-957b33-kserve-c8778cdbf-dlqkf" Nov 03 02:41:19.440221 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:41:19.440189 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-957b33/auth-test-model-957b33-kserve-c8778cdbf-dlqkf" Nov 03 02:41:23.914304 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:41:23.914266 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:41:43.416170 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:41:43.416091 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:41:53.913761 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:41:53.913719 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:42:23.912256 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:42:23.912215 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:42:27.508333 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:42:27.508295 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:42:53.913172 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:42:53.913131 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:42:55.795936 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:42:55.795893 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29"] Nov 03 02:42:55.796493 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:42:55.796156 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29" podUID="4846c4da-bf51-483f-8860-dc49d45f5d22" containerName="istio-proxy" containerID="cri-o://64773bd1dc784b2154f54a548b45a25f7be3f6c96fb58f616d6c989a52d966e1" gracePeriod=30 Nov 03 02:43:00.326735 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:00.326698 5605 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 02:43:00.327201 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:00.326758 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29" podUID="4846c4da-bf51-483f-8860-dc49d45f5d22" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 02:43:00.986601 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:00.986490 5605 generic.go:356] "Generic (PLEG): container finished" podID="4846c4da-bf51-483f-8860-dc49d45f5d22" containerID="64773bd1dc784b2154f54a548b45a25f7be3f6c96fb58f616d6c989a52d966e1" exitCode=0 Nov 03 02:43:00.986601 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:00.986552 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29" event={"ID":"4846c4da-bf51-483f-8860-dc49d45f5d22","Type":"ContainerDied","Data":"64773bd1dc784b2154f54a548b45a25f7be3f6c96fb58f616d6c989a52d966e1"} Nov 03 02:43:01.057945 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:01.057919 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29" Nov 03 02:43:01.203324 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:01.203251 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4846c4da-bf51-483f-8860-dc49d45f5d22-kube-api-access-pnfv6\" (UniqueName: \"kubernetes.io/projected/4846c4da-bf51-483f-8860-dc49d45f5d22-kube-api-access-pnfv6\") pod \"4846c4da-bf51-483f-8860-dc49d45f5d22\" (UID: \"4846c4da-bf51-483f-8860-dc49d45f5d22\") " Nov 03 02:43:01.203324 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:01.203288 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4846c4da-bf51-483f-8860-dc49d45f5d22-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4846c4da-bf51-483f-8860-dc49d45f5d22-workload-certs\") pod \"4846c4da-bf51-483f-8860-dc49d45f5d22\" (UID: \"4846c4da-bf51-483f-8860-dc49d45f5d22\") " Nov 03 02:43:01.203324 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:01.203323 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4846c4da-bf51-483f-8860-dc49d45f5d22-istio-token\" (UniqueName: \"kubernetes.io/projected/4846c4da-bf51-483f-8860-dc49d45f5d22-istio-token\") pod \"4846c4da-bf51-483f-8860-dc49d45f5d22\" (UID: \"4846c4da-bf51-483f-8860-dc49d45f5d22\") " Nov 03 02:43:01.203638 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:01.203439 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4846c4da-bf51-483f-8860-dc49d45f5d22-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4846c4da-bf51-483f-8860-dc49d45f5d22-credential-socket\") pod \"4846c4da-bf51-483f-8860-dc49d45f5d22\" (UID: \"4846c4da-bf51-483f-8860-dc49d45f5d22\") " Nov 03 02:43:01.203638 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:01.203493 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4846c4da-bf51-483f-8860-dc49d45f5d22-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4846c4da-bf51-483f-8860-dc49d45f5d22-istio-podinfo\") pod \"4846c4da-bf51-483f-8860-dc49d45f5d22\" (UID: \"4846c4da-bf51-483f-8860-dc49d45f5d22\") " Nov 03 02:43:01.203638 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:01.203540 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4846c4da-bf51-483f-8860-dc49d45f5d22-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4846c4da-bf51-483f-8860-dc49d45f5d22-istio-data\") pod \"4846c4da-bf51-483f-8860-dc49d45f5d22\" (UID: \"4846c4da-bf51-483f-8860-dc49d45f5d22\") " Nov 03 02:43:01.203638 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:01.203567 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4846c4da-bf51-483f-8860-dc49d45f5d22-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4846c4da-bf51-483f-8860-dc49d45f5d22-istiod-ca-cert\") pod \"4846c4da-bf51-483f-8860-dc49d45f5d22\" (UID: \"4846c4da-bf51-483f-8860-dc49d45f5d22\") " Nov 03 02:43:01.203864 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:01.203627 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4846c4da-bf51-483f-8860-dc49d45f5d22-workload-certs" pod "4846c4da-bf51-483f-8860-dc49d45f5d22" (UID: "4846c4da-bf51-483f-8860-dc49d45f5d22"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:43:01.203864 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:01.203669 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4846c4da-bf51-483f-8860-dc49d45f5d22-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4846c4da-bf51-483f-8860-dc49d45f5d22-workload-socket\") pod \"4846c4da-bf51-483f-8860-dc49d45f5d22\" (UID: \"4846c4da-bf51-483f-8860-dc49d45f5d22\") " Nov 03 02:43:01.203864 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:01.203714 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4846c4da-bf51-483f-8860-dc49d45f5d22-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4846c4da-bf51-483f-8860-dc49d45f5d22-istio-envoy\") pod \"4846c4da-bf51-483f-8860-dc49d45f5d22\" (UID: \"4846c4da-bf51-483f-8860-dc49d45f5d22\") " Nov 03 02:43:01.203864 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:01.203821 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4846c4da-bf51-483f-8860-dc49d45f5d22-istio-data" pod "4846c4da-bf51-483f-8860-dc49d45f5d22" (UID: "4846c4da-bf51-483f-8860-dc49d45f5d22"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:43:01.204070 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:01.203945 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4846c4da-bf51-483f-8860-dc49d45f5d22-workload-socket" pod "4846c4da-bf51-483f-8860-dc49d45f5d22" (UID: "4846c4da-bf51-483f-8860-dc49d45f5d22"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:43:01.204070 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:01.203963 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4846c4da-bf51-483f-8860-dc49d45f5d22-istiod-ca-cert" pod "4846c4da-bf51-483f-8860-dc49d45f5d22" (UID: "4846c4da-bf51-483f-8860-dc49d45f5d22"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:43:01.204242 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:01.204217 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4846c4da-bf51-483f-8860-dc49d45f5d22-credential-socket" pod "4846c4da-bf51-483f-8860-dc49d45f5d22" (UID: "4846c4da-bf51-483f-8860-dc49d45f5d22"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:43:01.204242 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:01.204235 5605 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4846c4da-bf51-483f-8860-dc49d45f5d22-workload-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:43:01.204408 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:01.204256 5605 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4846c4da-bf51-483f-8860-dc49d45f5d22-istio-data\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:43:01.204408 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:01.204272 5605 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4846c4da-bf51-483f-8860-dc49d45f5d22-istiod-ca-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:43:01.204408 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:01.204287 5605 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4846c4da-bf51-483f-8860-dc49d45f5d22-workload-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:43:01.205834 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:01.205809 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4846c4da-bf51-483f-8860-dc49d45f5d22-istio-envoy" pod "4846c4da-bf51-483f-8860-dc49d45f5d22" (UID: "4846c4da-bf51-483f-8860-dc49d45f5d22"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:43:01.205834 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:01.205821 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4846c4da-bf51-483f-8860-dc49d45f5d22-kube-api-access-pnfv6" pod "4846c4da-bf51-483f-8860-dc49d45f5d22" (UID: "4846c4da-bf51-483f-8860-dc49d45f5d22"). InnerVolumeSpecName "kube-api-access-pnfv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:43:01.205988 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:01.205931 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4846c4da-bf51-483f-8860-dc49d45f5d22-istio-podinfo" pod "4846c4da-bf51-483f-8860-dc49d45f5d22" (UID: "4846c4da-bf51-483f-8860-dc49d45f5d22"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:43:01.205988 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:01.205963 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4846c4da-bf51-483f-8860-dc49d45f5d22-istio-token" pod "4846c4da-bf51-483f-8860-dc49d45f5d22" (UID: "4846c4da-bf51-483f-8860-dc49d45f5d22"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:43:01.304978 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:01.304949 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnfv6\" (UniqueName: \"kubernetes.io/projected/4846c4da-bf51-483f-8860-dc49d45f5d22-kube-api-access-pnfv6\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:43:01.304978 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:01.304979 5605 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4846c4da-bf51-483f-8860-dc49d45f5d22-istio-token\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:43:01.305163 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:01.304991 5605 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4846c4da-bf51-483f-8860-dc49d45f5d22-credential-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:43:01.305163 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:01.304999 5605 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4846c4da-bf51-483f-8860-dc49d45f5d22-istio-podinfo\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:43:01.305163 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:01.305008 5605 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4846c4da-bf51-483f-8860-dc49d45f5d22-istio-envoy\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:43:01.993290 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:01.993260 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29" Nov 03 02:43:01.993290 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:01.993270 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29" event={"ID":"4846c4da-bf51-483f-8860-dc49d45f5d22","Type":"ContainerDied","Data":"d66620b9ee044d122dbdfe46f2502fe7034f460d6e1475ac41bc77bb3938c035"} Nov 03 02:43:01.993804 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:01.993332 5605 scope.go:122] "RemoveContainer" containerID="64773bd1dc784b2154f54a548b45a25f7be3f6c96fb58f616d6c989a52d966e1" Nov 03 02:43:02.194481 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:02.194453 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29"] Nov 03 02:43:02.241812 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:02.241785 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-98b29"] Nov 03 02:43:02.849763 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:02.849729 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4846c4da-bf51-483f-8860-dc49d45f5d22" path="/var/lib/kubelet/pods/4846c4da-bf51-483f-8860-dc49d45f5d22/volumes" Nov 03 02:43:14.695738 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:14.695654 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1249ddb0-tb64z"] Nov 03 02:43:14.696350 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:14.696329 5605 cpu_manager.go:414] "containerMap: removing container" podUID="4846c4da-bf51-483f-8860-dc49d45f5d22" containerName="istio-proxy" Nov 03 02:43:14.696478 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:14.696352 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4846c4da-bf51-483f-8860-dc49d45f5d22" containerName="istio-proxy" Nov 03 02:43:14.696565 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:14.696551 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="4846c4da-bf51-483f-8860-dc49d45f5d22" containerName="istio-proxy" Nov 03 02:43:14.701380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:14.701342 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1249ddb0-tb64z" Nov 03 02:43:14.790152 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:14.790117 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:43:14.790152 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:14.790136 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:43:14.797008 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:14.796983 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:43:14.797118 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:14.797038 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:43:14.799194 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:14.799175 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1249ddb0-tb64z"] Nov 03 02:43:14.847688 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:14.847656 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/806470e7-bfd4-48c0-aecd-be72ce7f78f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1249ddb0-tb64z\" (UID: \"806470e7-bfd4-48c0-aecd-be72ce7f78f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1249ddb0-tb64z" Nov 03 02:43:14.847842 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:14.847811 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55w8s\" (UniqueName: \"kubernetes.io/projected/806470e7-bfd4-48c0-aecd-be72ce7f78f9-kube-api-access-55w8s\") pod \"maas-api-revoke-sub-1249ddb0-tb64z\" (UID: \"806470e7-bfd4-48c0-aecd-be72ce7f78f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1249ddb0-tb64z" Nov 03 02:43:14.847915 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:14.847840 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/806470e7-bfd4-48c0-aecd-be72ce7f78f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1249ddb0-tb64z\" (UID: \"806470e7-bfd4-48c0-aecd-be72ce7f78f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1249ddb0-tb64z" Nov 03 02:43:14.949764 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:14.949679 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/806470e7-bfd4-48c0-aecd-be72ce7f78f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1249ddb0-tb64z\" (UID: \"806470e7-bfd4-48c0-aecd-be72ce7f78f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1249ddb0-tb64z" Nov 03 02:43:14.950072 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:14.950054 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55w8s\" (UniqueName: \"kubernetes.io/projected/806470e7-bfd4-48c0-aecd-be72ce7f78f9-kube-api-access-55w8s\") pod \"maas-api-revoke-sub-1249ddb0-tb64z\" (UID: \"806470e7-bfd4-48c0-aecd-be72ce7f78f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1249ddb0-tb64z" Nov 03 02:43:14.950193 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:14.950178 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/806470e7-bfd4-48c0-aecd-be72ce7f78f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1249ddb0-tb64z\" (UID: \"806470e7-bfd4-48c0-aecd-be72ce7f78f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1249ddb0-tb64z" Nov 03 02:43:14.951167 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:14.951137 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/806470e7-bfd4-48c0-aecd-be72ce7f78f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1249ddb0-tb64z\" (UID: \"806470e7-bfd4-48c0-aecd-be72ce7f78f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1249ddb0-tb64z" Nov 03 02:43:14.954124 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:14.954098 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/806470e7-bfd4-48c0-aecd-be72ce7f78f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1249ddb0-tb64z\" (UID: \"806470e7-bfd4-48c0-aecd-be72ce7f78f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1249ddb0-tb64z" Nov 03 02:43:15.003351 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:15.003317 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55w8s\" (UniqueName: \"kubernetes.io/projected/806470e7-bfd4-48c0-aecd-be72ce7f78f9-kube-api-access-55w8s\") pod \"maas-api-revoke-sub-1249ddb0-tb64z\" (UID: \"806470e7-bfd4-48c0-aecd-be72ce7f78f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1249ddb0-tb64z" Nov 03 02:43:15.025784 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:15.025760 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1249ddb0-tb64z" Nov 03 02:43:15.405694 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:15.405656 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1249ddb0-tb64z"] Nov 03 02:43:15.412169 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:43:15.412137 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod806470e7_bfd4_48c0_aecd_be72ce7f78f9.slice/crio-84e7d5ae226e1cd7194252f49ff7eb236dd27f0bc44d8a647caa4a0d65d16f93 WatchSource:0}: Error finding container 84e7d5ae226e1cd7194252f49ff7eb236dd27f0bc44d8a647caa4a0d65d16f93: Status 404 returned error can't find the container with id 84e7d5ae226e1cd7194252f49ff7eb236dd27f0bc44d8a647caa4a0d65d16f93 Nov 03 02:43:16.076119 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:16.076080 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1249ddb0-tb64z" event={"ID":"806470e7-bfd4-48c0-aecd-be72ce7f78f9","Type":"ContainerStarted","Data":"8eee4ac8500805be420fec4a7351f3e6b0d33aaa6e0bf20971f5af9b44d5d5c1"} Nov 03 02:43:16.076632 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:16.076147 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1249ddb0-tb64z" event={"ID":"806470e7-bfd4-48c0-aecd-be72ce7f78f9","Type":"ContainerStarted","Data":"84e7d5ae226e1cd7194252f49ff7eb236dd27f0bc44d8a647caa4a0d65d16f93"} Nov 03 02:43:16.182863 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:16.182807 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1249ddb0-tb64z" podStartSLOduration=2.182789857 podStartE2EDuration="2.182789857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:43:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:43:16.182267312 +0000 UTC m=+3073.441465468" watchObservedRunningTime="2026-11-03 02:43:16.182789857 +0000 UTC m=+3073.441988011" Nov 03 02:43:17.083854 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:17.083810 5605 generic.go:356] "Generic (PLEG): container finished" podID="806470e7-bfd4-48c0-aecd-be72ce7f78f9" containerID="8eee4ac8500805be420fec4a7351f3e6b0d33aaa6e0bf20971f5af9b44d5d5c1" exitCode=0 Nov 03 02:43:17.083854 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:17.083850 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1249ddb0-tb64z" event={"ID":"806470e7-bfd4-48c0-aecd-be72ce7f78f9","Type":"ContainerDied","Data":"8eee4ac8500805be420fec4a7351f3e6b0d33aaa6e0bf20971f5af9b44d5d5c1"} Nov 03 02:43:18.232829 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:18.232804 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1249ddb0-tb64z" Nov 03 02:43:18.290152 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:18.290130 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/806470e7-bfd4-48c0-aecd-be72ce7f78f9-kube-api-access-55w8s\" (UniqueName: \"kubernetes.io/projected/806470e7-bfd4-48c0-aecd-be72ce7f78f9-kube-api-access-55w8s\") pod \"806470e7-bfd4-48c0-aecd-be72ce7f78f9\" (UID: \"806470e7-bfd4-48c0-aecd-be72ce7f78f9\") " Nov 03 02:43:18.290273 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:18.290248 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/806470e7-bfd4-48c0-aecd-be72ce7f78f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/806470e7-bfd4-48c0-aecd-be72ce7f78f9-maas-api-openshift-ca\") pod \"806470e7-bfd4-48c0-aecd-be72ce7f78f9\" (UID: \"806470e7-bfd4-48c0-aecd-be72ce7f78f9\") " Nov 03 02:43:18.290316 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:18.290270 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/806470e7-bfd4-48c0-aecd-be72ce7f78f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/806470e7-bfd4-48c0-aecd-be72ce7f78f9-maas-api-xks-ca\") pod \"806470e7-bfd4-48c0-aecd-be72ce7f78f9\" (UID: \"806470e7-bfd4-48c0-aecd-be72ce7f78f9\") " Nov 03 02:43:18.290613 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:18.290583 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/806470e7-bfd4-48c0-aecd-be72ce7f78f9-maas-api-openshift-ca" pod "806470e7-bfd4-48c0-aecd-be72ce7f78f9" (UID: "806470e7-bfd4-48c0-aecd-be72ce7f78f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:43:18.290712 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:18.290698 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/806470e7-bfd4-48c0-aecd-be72ce7f78f9-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:43:18.292107 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:18.292080 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/806470e7-bfd4-48c0-aecd-be72ce7f78f9-kube-api-access-55w8s" pod "806470e7-bfd4-48c0-aecd-be72ce7f78f9" (UID: "806470e7-bfd4-48c0-aecd-be72ce7f78f9"). InnerVolumeSpecName "kube-api-access-55w8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:43:18.292196 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:18.292159 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/806470e7-bfd4-48c0-aecd-be72ce7f78f9-maas-api-xks-ca" pod "806470e7-bfd4-48c0-aecd-be72ce7f78f9" (UID: "806470e7-bfd4-48c0-aecd-be72ce7f78f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:43:18.391670 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:18.391610 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/806470e7-bfd4-48c0-aecd-be72ce7f78f9-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:43:18.391670 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:18.391635 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55w8s\" (UniqueName: \"kubernetes.io/projected/806470e7-bfd4-48c0-aecd-be72ce7f78f9-kube-api-access-55w8s\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:43:19.095745 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:19.095716 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1249ddb0-tb64z" Nov 03 02:43:19.095920 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:19.095749 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1249ddb0-tb64z" event={"ID":"806470e7-bfd4-48c0-aecd-be72ce7f78f9","Type":"ContainerDied","Data":"84e7d5ae226e1cd7194252f49ff7eb236dd27f0bc44d8a647caa4a0d65d16f93"} Nov 03 02:43:19.095920 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:19.095798 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84e7d5ae226e1cd7194252f49ff7eb236dd27f0bc44d8a647caa4a0d65d16f93" Nov 03 02:43:23.913318 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:43:23.913286 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:43:30.512838 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:30.512795 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:43:39.655216 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:39.655185 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94dc3580-5hlmr"] Nov 03 02:43:39.655684 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:39.655668 5605 cpu_manager.go:414] "containerMap: removing container" podUID="806470e7-bfd4-48c0-aecd-be72ce7f78f9" containerName="revoke-keys" Nov 03 02:43:39.655730 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:39.655686 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="806470e7-bfd4-48c0-aecd-be72ce7f78f9" containerName="revoke-keys" Nov 03 02:43:39.655815 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:39.655805 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="806470e7-bfd4-48c0-aecd-be72ce7f78f9" containerName="revoke-keys" Nov 03 02:43:39.659156 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:39.659141 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94dc3580-5hlmr" Nov 03 02:43:39.710263 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:39.710240 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:43:39.756964 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:39.756940 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:43:39.756964 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:39.756955 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:43:39.757136 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:39.756941 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:43:39.759043 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:39.759025 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94dc3580-5hlmr"] Nov 03 02:43:39.794769 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:39.794739 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/439ed92d-93fa-4b98-83ea-3eac12315fd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94dc3580-5hlmr\" (UID: \"439ed92d-93fa-4b98-83ea-3eac12315fd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94dc3580-5hlmr" Nov 03 02:43:39.794886 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:39.794849 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/439ed92d-93fa-4b98-83ea-3eac12315fd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94dc3580-5hlmr\" (UID: \"439ed92d-93fa-4b98-83ea-3eac12315fd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94dc3580-5hlmr" Nov 03 02:43:39.794931 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:39.794884 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5zgn\" (UniqueName: \"kubernetes.io/projected/439ed92d-93fa-4b98-83ea-3eac12315fd1-kube-api-access-f5zgn\") pod \"maas-api-revoke-sub-94dc3580-5hlmr\" (UID: \"439ed92d-93fa-4b98-83ea-3eac12315fd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94dc3580-5hlmr" Nov 03 02:43:39.895813 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:39.895786 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/439ed92d-93fa-4b98-83ea-3eac12315fd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94dc3580-5hlmr\" (UID: \"439ed92d-93fa-4b98-83ea-3eac12315fd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94dc3580-5hlmr" Nov 03 02:43:39.895930 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:39.895849 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/439ed92d-93fa-4b98-83ea-3eac12315fd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94dc3580-5hlmr\" (UID: \"439ed92d-93fa-4b98-83ea-3eac12315fd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94dc3580-5hlmr" Nov 03 02:43:39.895972 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:39.895933 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5zgn\" (UniqueName: \"kubernetes.io/projected/439ed92d-93fa-4b98-83ea-3eac12315fd1-kube-api-access-f5zgn\") pod \"maas-api-revoke-sub-94dc3580-5hlmr\" (UID: \"439ed92d-93fa-4b98-83ea-3eac12315fd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94dc3580-5hlmr" Nov 03 02:43:39.896521 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:39.896499 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/439ed92d-93fa-4b98-83ea-3eac12315fd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94dc3580-5hlmr\" (UID: \"439ed92d-93fa-4b98-83ea-3eac12315fd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94dc3580-5hlmr" Nov 03 02:43:39.898275 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:39.898255 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/439ed92d-93fa-4b98-83ea-3eac12315fd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94dc3580-5hlmr\" (UID: \"439ed92d-93fa-4b98-83ea-3eac12315fd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94dc3580-5hlmr" Nov 03 02:43:39.948993 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:39.948938 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5zgn\" (UniqueName: \"kubernetes.io/projected/439ed92d-93fa-4b98-83ea-3eac12315fd1-kube-api-access-f5zgn\") pod \"maas-api-revoke-sub-94dc3580-5hlmr\" (UID: \"439ed92d-93fa-4b98-83ea-3eac12315fd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94dc3580-5hlmr" Nov 03 02:43:39.971762 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:39.971726 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94dc3580-5hlmr" Nov 03 02:43:40.341589 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:40.341559 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94dc3580-5hlmr"] Nov 03 02:43:40.347411 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:43:40.347385 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod439ed92d_93fa_4b98_83ea_3eac12315fd1.slice/crio-ae55dbd33cd6b4eee855aa8ff84d715312b7ec24f52d349a515aa5cbcc6f7c08 WatchSource:0}: Error finding container ae55dbd33cd6b4eee855aa8ff84d715312b7ec24f52d349a515aa5cbcc6f7c08: Status 404 returned error can't find the container with id ae55dbd33cd6b4eee855aa8ff84d715312b7ec24f52d349a515aa5cbcc6f7c08 Nov 03 02:43:41.220368 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:41.220326 5605 generic.go:356] "Generic (PLEG): container finished" podID="439ed92d-93fa-4b98-83ea-3eac12315fd1" containerID="0caa287f58f24ee33f5b9f9911f42b3010a3cfe015cd14d685a5c96548d5f67e" exitCode=0 Nov 03 02:43:41.220705 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:41.220413 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94dc3580-5hlmr" event={"ID":"439ed92d-93fa-4b98-83ea-3eac12315fd1","Type":"ContainerDied","Data":"0caa287f58f24ee33f5b9f9911f42b3010a3cfe015cd14d685a5c96548d5f67e"} Nov 03 02:43:41.220705 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:41.220463 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94dc3580-5hlmr" event={"ID":"439ed92d-93fa-4b98-83ea-3eac12315fd1","Type":"ContainerStarted","Data":"ae55dbd33cd6b4eee855aa8ff84d715312b7ec24f52d349a515aa5cbcc6f7c08"} Nov 03 02:43:42.377340 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:42.377314 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94dc3580-5hlmr" Nov 03 02:43:42.522095 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:42.522031 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/439ed92d-93fa-4b98-83ea-3eac12315fd1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/439ed92d-93fa-4b98-83ea-3eac12315fd1-maas-api-xks-ca\") pod \"439ed92d-93fa-4b98-83ea-3eac12315fd1\" (UID: \"439ed92d-93fa-4b98-83ea-3eac12315fd1\") " Nov 03 02:43:42.522095 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:42.522083 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/439ed92d-93fa-4b98-83ea-3eac12315fd1-kube-api-access-f5zgn\" (UniqueName: \"kubernetes.io/projected/439ed92d-93fa-4b98-83ea-3eac12315fd1-kube-api-access-f5zgn\") pod \"439ed92d-93fa-4b98-83ea-3eac12315fd1\" (UID: \"439ed92d-93fa-4b98-83ea-3eac12315fd1\") " Nov 03 02:43:42.522254 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:42.522165 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/439ed92d-93fa-4b98-83ea-3eac12315fd1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/439ed92d-93fa-4b98-83ea-3eac12315fd1-maas-api-openshift-ca\") pod \"439ed92d-93fa-4b98-83ea-3eac12315fd1\" (UID: \"439ed92d-93fa-4b98-83ea-3eac12315fd1\") " Nov 03 02:43:42.522577 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:42.522548 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/439ed92d-93fa-4b98-83ea-3eac12315fd1-maas-api-openshift-ca" pod "439ed92d-93fa-4b98-83ea-3eac12315fd1" (UID: "439ed92d-93fa-4b98-83ea-3eac12315fd1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:43:42.524141 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:42.524118 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/439ed92d-93fa-4b98-83ea-3eac12315fd1-maas-api-xks-ca" pod "439ed92d-93fa-4b98-83ea-3eac12315fd1" (UID: "439ed92d-93fa-4b98-83ea-3eac12315fd1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:43:42.524225 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:42.524179 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/439ed92d-93fa-4b98-83ea-3eac12315fd1-kube-api-access-f5zgn" pod "439ed92d-93fa-4b98-83ea-3eac12315fd1" (UID: "439ed92d-93fa-4b98-83ea-3eac12315fd1"). InnerVolumeSpecName "kube-api-access-f5zgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:43:42.623936 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:42.623909 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/439ed92d-93fa-4b98-83ea-3eac12315fd1-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:43:42.623936 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:42.623934 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/439ed92d-93fa-4b98-83ea-3eac12315fd1-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:43:42.624073 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:42.623944 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5zgn\" (UniqueName: \"kubernetes.io/projected/439ed92d-93fa-4b98-83ea-3eac12315fd1-kube-api-access-f5zgn\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:43:43.234661 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:43.234590 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94dc3580-5hlmr" Nov 03 02:43:43.234799 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:43.234591 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94dc3580-5hlmr" event={"ID":"439ed92d-93fa-4b98-83ea-3eac12315fd1","Type":"ContainerDied","Data":"ae55dbd33cd6b4eee855aa8ff84d715312b7ec24f52d349a515aa5cbcc6f7c08"} Nov 03 02:43:43.234799 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:43.234716 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae55dbd33cd6b4eee855aa8ff84d715312b7ec24f52d349a515aa5cbcc6f7c08" Nov 03 02:43:50.425449 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:50.425415 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa6a259d-ql28g"] Nov 03 02:43:50.425927 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:50.425856 5605 cpu_manager.go:414] "containerMap: removing container" podUID="439ed92d-93fa-4b98-83ea-3eac12315fd1" containerName="revoke-keys" Nov 03 02:43:50.425927 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:50.425866 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="439ed92d-93fa-4b98-83ea-3eac12315fd1" containerName="revoke-keys" Nov 03 02:43:50.426005 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:50.425962 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="439ed92d-93fa-4b98-83ea-3eac12315fd1" containerName="revoke-keys" Nov 03 02:43:50.429232 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:50.429209 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa6a259d-ql28g" Nov 03 02:43:50.483203 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:50.483176 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa6a259d-ql28g"] Nov 03 02:43:50.497397 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:50.497374 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:43:50.497503 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:50.497413 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:43:50.527221 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:50.527193 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:43:50.527221 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:50.527203 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:43:50.602432 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:50.602406 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cf7e386-7d8e-487c-90fb-b367eb765d2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa6a259d-ql28g\" (UID: \"1cf7e386-7d8e-487c-90fb-b367eb765d2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa6a259d-ql28g" Nov 03 02:43:50.602541 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:50.602446 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99m2w\" (UniqueName: \"kubernetes.io/projected/1cf7e386-7d8e-487c-90fb-b367eb765d2a-kube-api-access-99m2w\") pod \"maas-api-revoke-sub-aa6a259d-ql28g\" (UID: \"1cf7e386-7d8e-487c-90fb-b367eb765d2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa6a259d-ql28g" Nov 03 02:43:50.602660 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:50.602643 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cf7e386-7d8e-487c-90fb-b367eb765d2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa6a259d-ql28g\" (UID: \"1cf7e386-7d8e-487c-90fb-b367eb765d2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa6a259d-ql28g" Nov 03 02:43:50.703547 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:50.703489 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cf7e386-7d8e-487c-90fb-b367eb765d2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa6a259d-ql28g\" (UID: \"1cf7e386-7d8e-487c-90fb-b367eb765d2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa6a259d-ql28g" Nov 03 02:43:50.703654 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:50.703548 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cf7e386-7d8e-487c-90fb-b367eb765d2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa6a259d-ql28g\" (UID: \"1cf7e386-7d8e-487c-90fb-b367eb765d2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa6a259d-ql28g" Nov 03 02:43:50.703654 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:50.703583 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99m2w\" (UniqueName: \"kubernetes.io/projected/1cf7e386-7d8e-487c-90fb-b367eb765d2a-kube-api-access-99m2w\") pod \"maas-api-revoke-sub-aa6a259d-ql28g\" (UID: \"1cf7e386-7d8e-487c-90fb-b367eb765d2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa6a259d-ql28g" Nov 03 02:43:50.704285 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:50.704266 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cf7e386-7d8e-487c-90fb-b367eb765d2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa6a259d-ql28g\" (UID: \"1cf7e386-7d8e-487c-90fb-b367eb765d2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa6a259d-ql28g" Nov 03 02:43:50.706001 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:50.705980 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cf7e386-7d8e-487c-90fb-b367eb765d2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa6a259d-ql28g\" (UID: \"1cf7e386-7d8e-487c-90fb-b367eb765d2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa6a259d-ql28g" Nov 03 02:43:50.756808 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:50.756781 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99m2w\" (UniqueName: \"kubernetes.io/projected/1cf7e386-7d8e-487c-90fb-b367eb765d2a-kube-api-access-99m2w\") pod \"maas-api-revoke-sub-aa6a259d-ql28g\" (UID: \"1cf7e386-7d8e-487c-90fb-b367eb765d2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa6a259d-ql28g" Nov 03 02:43:51.044797 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:51.044734 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa6a259d-ql28g" Nov 03 02:43:51.414154 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:51.414119 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa6a259d-ql28g"] Nov 03 02:43:51.419753 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:43:51.419727 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cf7e386_7d8e_487c_90fb_b367eb765d2a.slice/crio-533fb1c15cca368e8ada19e2280a0ed3c3546be20d78ffcdcbceffadbc53a8be WatchSource:0}: Error finding container 533fb1c15cca368e8ada19e2280a0ed3c3546be20d78ffcdcbceffadbc53a8be: Status 404 returned error can't find the container with id 533fb1c15cca368e8ada19e2280a0ed3c3546be20d78ffcdcbceffadbc53a8be Nov 03 02:43:52.282590 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:52.282552 5605 generic.go:356] "Generic (PLEG): container finished" podID="1cf7e386-7d8e-487c-90fb-b367eb765d2a" containerID="de7fbd833b45d502e8b8fc19b1496e4b30c97e6c20c181575b55ba77470ed375" exitCode=0 Nov 03 02:43:52.282962 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:52.282589 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa6a259d-ql28g" event={"ID":"1cf7e386-7d8e-487c-90fb-b367eb765d2a","Type":"ContainerDied","Data":"de7fbd833b45d502e8b8fc19b1496e4b30c97e6c20c181575b55ba77470ed375"} Nov 03 02:43:52.282962 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:52.282651 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa6a259d-ql28g" event={"ID":"1cf7e386-7d8e-487c-90fb-b367eb765d2a","Type":"ContainerStarted","Data":"533fb1c15cca368e8ada19e2280a0ed3c3546be20d78ffcdcbceffadbc53a8be"} Nov 03 02:43:53.426741 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:53.426715 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa6a259d-ql28g" Nov 03 02:43:53.533682 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:53.533652 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cf7e386-7d8e-487c-90fb-b367eb765d2a-kube-api-access-99m2w\" (UniqueName: \"kubernetes.io/projected/1cf7e386-7d8e-487c-90fb-b367eb765d2a-kube-api-access-99m2w\") pod \"1cf7e386-7d8e-487c-90fb-b367eb765d2a\" (UID: \"1cf7e386-7d8e-487c-90fb-b367eb765d2a\") " Nov 03 02:43:53.533838 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:53.533759 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1cf7e386-7d8e-487c-90fb-b367eb765d2a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cf7e386-7d8e-487c-90fb-b367eb765d2a-maas-api-xks-ca\") pod \"1cf7e386-7d8e-487c-90fb-b367eb765d2a\" (UID: \"1cf7e386-7d8e-487c-90fb-b367eb765d2a\") " Nov 03 02:43:53.533838 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:53.533811 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cf7e386-7d8e-487c-90fb-b367eb765d2a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cf7e386-7d8e-487c-90fb-b367eb765d2a-maas-api-openshift-ca\") pod \"1cf7e386-7d8e-487c-90fb-b367eb765d2a\" (UID: \"1cf7e386-7d8e-487c-90fb-b367eb765d2a\") " Nov 03 02:43:53.534294 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:53.534187 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cf7e386-7d8e-487c-90fb-b367eb765d2a-maas-api-openshift-ca" pod "1cf7e386-7d8e-487c-90fb-b367eb765d2a" (UID: "1cf7e386-7d8e-487c-90fb-b367eb765d2a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:43:53.534431 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:53.534299 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cf7e386-7d8e-487c-90fb-b367eb765d2a-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:43:53.536405 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:53.536336 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cf7e386-7d8e-487c-90fb-b367eb765d2a-maas-api-xks-ca" pod "1cf7e386-7d8e-487c-90fb-b367eb765d2a" (UID: "1cf7e386-7d8e-487c-90fb-b367eb765d2a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:43:53.536914 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:53.536889 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cf7e386-7d8e-487c-90fb-b367eb765d2a-kube-api-access-99m2w" pod "1cf7e386-7d8e-487c-90fb-b367eb765d2a" (UID: "1cf7e386-7d8e-487c-90fb-b367eb765d2a"). InnerVolumeSpecName "kube-api-access-99m2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:43:53.635769 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:53.635746 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99m2w\" (UniqueName: \"kubernetes.io/projected/1cf7e386-7d8e-487c-90fb-b367eb765d2a-kube-api-access-99m2w\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:43:53.635769 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:53.635771 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cf7e386-7d8e-487c-90fb-b367eb765d2a-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:43:53.912782 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:43:53.912737 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:43:54.295062 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:54.294985 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa6a259d-ql28g" Nov 03 02:43:54.295062 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:54.295019 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa6a259d-ql28g" event={"ID":"1cf7e386-7d8e-487c-90fb-b367eb765d2a","Type":"ContainerDied","Data":"533fb1c15cca368e8ada19e2280a0ed3c3546be20d78ffcdcbceffadbc53a8be"} Nov 03 02:43:54.295244 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:43:54.295072 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="533fb1c15cca368e8ada19e2280a0ed3c3546be20d78ffcdcbceffadbc53a8be" Nov 03 02:44:03.893034 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:03.893000 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f954db-6cssj"] Nov 03 02:44:03.893579 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:03.893441 5605 cpu_manager.go:414] "containerMap: removing container" podUID="1cf7e386-7d8e-487c-90fb-b367eb765d2a" containerName="revoke-keys" Nov 03 02:44:03.893579 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:03.893453 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cf7e386-7d8e-487c-90fb-b367eb765d2a" containerName="revoke-keys" Nov 03 02:44:03.893579 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:03.893551 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cf7e386-7d8e-487c-90fb-b367eb765d2a" containerName="revoke-keys" Nov 03 02:44:03.896816 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:03.896795 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f954db-6cssj" Nov 03 02:44:03.932680 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:03.932657 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21802adf-b888-473e-8994-16b453c2e0b9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f954db-6cssj\" (UID: \"21802adf-b888-473e-8994-16b453c2e0b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f954db-6cssj" Nov 03 02:44:03.932786 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:03.932723 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21802adf-b888-473e-8994-16b453c2e0b9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f954db-6cssj\" (UID: \"21802adf-b888-473e-8994-16b453c2e0b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f954db-6cssj" Nov 03 02:44:03.932869 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:03.932851 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccdnl\" (UniqueName: \"kubernetes.io/projected/21802adf-b888-473e-8994-16b453c2e0b9-kube-api-access-ccdnl\") pod \"maas-api-revoke-keys-e2e-praxis-a-f954db-6cssj\" (UID: \"21802adf-b888-473e-8994-16b453c2e0b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f954db-6cssj" Nov 03 02:44:03.994544 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:03.994512 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:44:03.994544 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:03.994523 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:44:03.994544 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:03.994537 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:44:03.994756 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:03.994624 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:44:03.996791 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:03.996772 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f954db-6cssj"] Nov 03 02:44:04.034351 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:04.034327 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21802adf-b888-473e-8994-16b453c2e0b9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f954db-6cssj\" (UID: \"21802adf-b888-473e-8994-16b453c2e0b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f954db-6cssj" Nov 03 02:44:04.034463 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:04.034410 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21802adf-b888-473e-8994-16b453c2e0b9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f954db-6cssj\" (UID: \"21802adf-b888-473e-8994-16b453c2e0b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f954db-6cssj" Nov 03 02:44:04.034463 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:04.034448 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccdnl\" (UniqueName: \"kubernetes.io/projected/21802adf-b888-473e-8994-16b453c2e0b9-kube-api-access-ccdnl\") pod \"maas-api-revoke-keys-e2e-praxis-a-f954db-6cssj\" (UID: \"21802adf-b888-473e-8994-16b453c2e0b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f954db-6cssj" Nov 03 02:44:04.035029 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:04.035002 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21802adf-b888-473e-8994-16b453c2e0b9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f954db-6cssj\" (UID: \"21802adf-b888-473e-8994-16b453c2e0b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f954db-6cssj" Nov 03 02:44:04.036771 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:04.036752 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21802adf-b888-473e-8994-16b453c2e0b9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f954db-6cssj\" (UID: \"21802adf-b888-473e-8994-16b453c2e0b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f954db-6cssj" Nov 03 02:44:04.088836 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:04.088809 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccdnl\" (UniqueName: \"kubernetes.io/projected/21802adf-b888-473e-8994-16b453c2e0b9-kube-api-access-ccdnl\") pod \"maas-api-revoke-keys-e2e-praxis-a-f954db-6cssj\" (UID: \"21802adf-b888-473e-8994-16b453c2e0b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f954db-6cssj" Nov 03 02:44:04.211163 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:04.211100 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f954db-6cssj" Nov 03 02:44:04.579487 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:04.579455 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f954db-6cssj"] Nov 03 02:44:04.584848 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:44:04.584823 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21802adf_b888_473e_8994_16b453c2e0b9.slice/crio-1925d30418e4f46f662bf2c156d1825cc39acc3d150bf8e52ff4bea78f920dac WatchSource:0}: Error finding container 1925d30418e4f46f662bf2c156d1825cc39acc3d150bf8e52ff4bea78f920dac: Status 404 returned error can't find the container with id 1925d30418e4f46f662bf2c156d1825cc39acc3d150bf8e52ff4bea78f920dac Nov 03 02:44:05.354240 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:05.354193 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f954db-6cssj" event={"ID":"21802adf-b888-473e-8994-16b453c2e0b9","Type":"ContainerStarted","Data":"3f7315d548fdfdca574c997ab7ba3bb07f61a07b7cc55237c2e46af124b552ac"} Nov 03 02:44:05.354753 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:05.354259 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f954db-6cssj" event={"ID":"21802adf-b888-473e-8994-16b453c2e0b9","Type":"ContainerStarted","Data":"1925d30418e4f46f662bf2c156d1825cc39acc3d150bf8e52ff4bea78f920dac"} Nov 03 02:44:05.467727 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:05.467678 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f954db-6cssj" podStartSLOduration=2.467664632 podStartE2EDuration="2.467664632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:44:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:44:05.465116883 +0000 UTC m=+3122.724315037" watchObservedRunningTime="2026-11-03 02:44:05.467664632 +0000 UTC m=+3122.726862785" Nov 03 02:44:06.361908 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:06.361866 5605 generic.go:356] "Generic (PLEG): container finished" podID="21802adf-b888-473e-8994-16b453c2e0b9" containerID="3f7315d548fdfdca574c997ab7ba3bb07f61a07b7cc55237c2e46af124b552ac" exitCode=0 Nov 03 02:44:06.362369 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:06.361912 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f954db-6cssj" event={"ID":"21802adf-b888-473e-8994-16b453c2e0b9","Type":"ContainerDied","Data":"3f7315d548fdfdca574c997ab7ba3bb07f61a07b7cc55237c2e46af124b552ac"} Nov 03 02:44:07.508662 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:07.508635 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f954db-6cssj" Nov 03 02:44:07.569781 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:07.569756 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21802adf-b888-473e-8994-16b453c2e0b9-kube-api-access-ccdnl\" (UniqueName: \"kubernetes.io/projected/21802adf-b888-473e-8994-16b453c2e0b9-kube-api-access-ccdnl\") pod \"21802adf-b888-473e-8994-16b453c2e0b9\" (UID: \"21802adf-b888-473e-8994-16b453c2e0b9\") " Nov 03 02:44:07.569922 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:07.569890 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21802adf-b888-473e-8994-16b453c2e0b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21802adf-b888-473e-8994-16b453c2e0b9-maas-api-openshift-ca\") pod \"21802adf-b888-473e-8994-16b453c2e0b9\" (UID: \"21802adf-b888-473e-8994-16b453c2e0b9\") " Nov 03 02:44:07.570005 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:07.569984 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21802adf-b888-473e-8994-16b453c2e0b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21802adf-b888-473e-8994-16b453c2e0b9-maas-api-xks-ca\") pod \"21802adf-b888-473e-8994-16b453c2e0b9\" (UID: \"21802adf-b888-473e-8994-16b453c2e0b9\") " Nov 03 02:44:07.570213 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:07.570189 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21802adf-b888-473e-8994-16b453c2e0b9-maas-api-openshift-ca" pod "21802adf-b888-473e-8994-16b453c2e0b9" (UID: "21802adf-b888-473e-8994-16b453c2e0b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:44:07.571950 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:07.571921 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21802adf-b888-473e-8994-16b453c2e0b9-maas-api-xks-ca" pod "21802adf-b888-473e-8994-16b453c2e0b9" (UID: "21802adf-b888-473e-8994-16b453c2e0b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:44:07.572056 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:07.571948 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21802adf-b888-473e-8994-16b453c2e0b9-kube-api-access-ccdnl" pod "21802adf-b888-473e-8994-16b453c2e0b9" (UID: "21802adf-b888-473e-8994-16b453c2e0b9"). InnerVolumeSpecName "kube-api-access-ccdnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:44:07.671111 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:07.671040 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccdnl\" (UniqueName: \"kubernetes.io/projected/21802adf-b888-473e-8994-16b453c2e0b9-kube-api-access-ccdnl\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:44:07.671111 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:07.671068 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21802adf-b888-473e-8994-16b453c2e0b9-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:44:07.671111 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:07.671081 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21802adf-b888-473e-8994-16b453c2e0b9-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:44:08.374382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:08.374335 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f954db-6cssj" Nov 03 02:44:08.374553 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:08.374381 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f954db-6cssj" event={"ID":"21802adf-b888-473e-8994-16b453c2e0b9","Type":"ContainerDied","Data":"1925d30418e4f46f662bf2c156d1825cc39acc3d150bf8e52ff4bea78f920dac"} Nov 03 02:44:08.374553 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:08.374425 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1925d30418e4f46f662bf2c156d1825cc39acc3d150bf8e52ff4bea78f920dac" Nov 03 02:44:14.641465 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:14.641431 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-965a07/auth-test-model-965a07-kserve-86544f48df-b4jcp"] Nov 03 02:44:23.915402 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:44:23.915353 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:44:27.186102 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:27.186069 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7273af/auth-test-model-7273af-kserve-dfd448c77-g45rq"] Nov 03 02:44:27.893717 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:27.893677 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f954db-6cssj"] Nov 03 02:44:27.896242 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:27.896212 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f954db-6cssj"] Nov 03 02:44:28.377207 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:28.377174 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4"] Nov 03 02:44:28.377735 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:28.377720 5605 cpu_manager.go:414] "containerMap: removing container" podUID="21802adf-b888-473e-8994-16b453c2e0b9" containerName="revoke-keys" Nov 03 02:44:28.377783 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:28.377737 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21802adf-b888-473e-8994-16b453c2e0b9" containerName="revoke-keys" Nov 03 02:44:28.377843 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:28.377834 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="21802adf-b888-473e-8994-16b453c2e0b9" containerName="revoke-keys" Nov 03 02:44:28.380593 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:28.380571 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4" Nov 03 02:44:28.457140 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:28.457099 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4"] Nov 03 02:44:28.478968 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:28.478933 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk8tn\" (UniqueName: \"kubernetes.io/projected/33cae564-16a9-42bb-a1a5-dc5c3257f56b-kube-api-access-bk8tn\") pod \"e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4\" (UID: \"33cae564-16a9-42bb-a1a5-dc5c3257f56b\") " pod="openshift-ingress/e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4" Nov 03 02:44:28.479268 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:28.479247 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33cae564-16a9-42bb-a1a5-dc5c3257f56b-istio-podinfo\") pod \"e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4\" (UID: \"33cae564-16a9-42bb-a1a5-dc5c3257f56b\") " pod="openshift-ingress/e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4" Nov 03 02:44:28.479472 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:28.479456 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33cae564-16a9-42bb-a1a5-dc5c3257f56b-workload-socket\") pod \"e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4\" (UID: \"33cae564-16a9-42bb-a1a5-dc5c3257f56b\") " pod="openshift-ingress/e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4" Nov 03 02:44:28.479639 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:28.479623 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33cae564-16a9-42bb-a1a5-dc5c3257f56b-workload-certs\") pod \"e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4\" (UID: \"33cae564-16a9-42bb-a1a5-dc5c3257f56b\") " pod="openshift-ingress/e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4" Nov 03 02:44:28.479811 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:28.479797 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33cae564-16a9-42bb-a1a5-dc5c3257f56b-istio-envoy\") pod \"e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4\" (UID: \"33cae564-16a9-42bb-a1a5-dc5c3257f56b\") " pod="openshift-ingress/e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4" Nov 03 02:44:28.479951 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:28.479936 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33cae564-16a9-42bb-a1a5-dc5c3257f56b-istio-data\") pod \"e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4\" (UID: \"33cae564-16a9-42bb-a1a5-dc5c3257f56b\") " pod="openshift-ingress/e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4" Nov 03 02:44:28.480123 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:28.480106 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33cae564-16a9-42bb-a1a5-dc5c3257f56b-credential-socket\") pod \"e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4\" (UID: \"33cae564-16a9-42bb-a1a5-dc5c3257f56b\") " pod="openshift-ingress/e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4" Nov 03 02:44:28.480295 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:28.480273 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33cae564-16a9-42bb-a1a5-dc5c3257f56b-istiod-ca-cert\") pod \"e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4\" (UID: \"33cae564-16a9-42bb-a1a5-dc5c3257f56b\") " pod="openshift-ingress/e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4" Nov 03 02:44:28.480484 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:28.480468 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33cae564-16a9-42bb-a1a5-dc5c3257f56b-istio-token\") pod \"e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4\" (UID: \"33cae564-16a9-42bb-a1a5-dc5c3257f56b\") " pod="openshift-ingress/e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4" Nov 03 02:44:28.496530 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:28.496498 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-63a61e-openshift-default-dockercfg-fxnj9\"" Nov 03 02:44:28.582245 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:28.582202 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33cae564-16a9-42bb-a1a5-dc5c3257f56b-credential-socket\") pod \"e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4\" (UID: \"33cae564-16a9-42bb-a1a5-dc5c3257f56b\") " pod="openshift-ingress/e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4" Nov 03 02:44:28.582458 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:28.582252 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33cae564-16a9-42bb-a1a5-dc5c3257f56b-istiod-ca-cert\") pod \"e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4\" (UID: \"33cae564-16a9-42bb-a1a5-dc5c3257f56b\") " pod="openshift-ingress/e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4" Nov 03 02:44:28.582458 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:28.582301 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33cae564-16a9-42bb-a1a5-dc5c3257f56b-istio-token\") pod \"e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4\" (UID: \"33cae564-16a9-42bb-a1a5-dc5c3257f56b\") " pod="openshift-ingress/e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4" Nov 03 02:44:28.582600 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:28.582462 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk8tn\" (UniqueName: \"kubernetes.io/projected/33cae564-16a9-42bb-a1a5-dc5c3257f56b-kube-api-access-bk8tn\") pod \"e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4\" (UID: \"33cae564-16a9-42bb-a1a5-dc5c3257f56b\") " pod="openshift-ingress/e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4" Nov 03 02:44:28.582600 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:28.582535 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33cae564-16a9-42bb-a1a5-dc5c3257f56b-istio-podinfo\") pod \"e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4\" (UID: \"33cae564-16a9-42bb-a1a5-dc5c3257f56b\") " pod="openshift-ingress/e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4" Nov 03 02:44:28.582600 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:28.582568 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33cae564-16a9-42bb-a1a5-dc5c3257f56b-workload-socket\") pod \"e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4\" (UID: \"33cae564-16a9-42bb-a1a5-dc5c3257f56b\") " pod="openshift-ingress/e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4" Nov 03 02:44:28.582754 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:28.582599 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33cae564-16a9-42bb-a1a5-dc5c3257f56b-workload-certs\") pod \"e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4\" (UID: \"33cae564-16a9-42bb-a1a5-dc5c3257f56b\") " pod="openshift-ingress/e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4" Nov 03 02:44:28.582754 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:28.582645 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33cae564-16a9-42bb-a1a5-dc5c3257f56b-istio-envoy\") pod \"e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4\" (UID: \"33cae564-16a9-42bb-a1a5-dc5c3257f56b\") " pod="openshift-ingress/e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4" Nov 03 02:44:28.582754 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:28.582668 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33cae564-16a9-42bb-a1a5-dc5c3257f56b-credential-socket\") pod \"e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4\" (UID: \"33cae564-16a9-42bb-a1a5-dc5c3257f56b\") " pod="openshift-ingress/e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4" Nov 03 02:44:28.582754 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:28.582680 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33cae564-16a9-42bb-a1a5-dc5c3257f56b-istio-data\") pod \"e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4\" (UID: \"33cae564-16a9-42bb-a1a5-dc5c3257f56b\") " pod="openshift-ingress/e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4" Nov 03 02:44:28.582945 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:28.582918 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33cae564-16a9-42bb-a1a5-dc5c3257f56b-workload-socket\") pod \"e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4\" (UID: \"33cae564-16a9-42bb-a1a5-dc5c3257f56b\") " pod="openshift-ingress/e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4" Nov 03 02:44:28.583107 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:28.583081 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33cae564-16a9-42bb-a1a5-dc5c3257f56b-workload-certs\") pod \"e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4\" (UID: \"33cae564-16a9-42bb-a1a5-dc5c3257f56b\") " pod="openshift-ingress/e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4" Nov 03 02:44:28.583174 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:28.583111 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33cae564-16a9-42bb-a1a5-dc5c3257f56b-istio-data\") pod \"e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4\" (UID: \"33cae564-16a9-42bb-a1a5-dc5c3257f56b\") " pod="openshift-ingress/e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4" Nov 03 02:44:28.583174 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:28.583112 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33cae564-16a9-42bb-a1a5-dc5c3257f56b-istiod-ca-cert\") pod \"e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4\" (UID: \"33cae564-16a9-42bb-a1a5-dc5c3257f56b\") " pod="openshift-ingress/e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4" Nov 03 02:44:28.584971 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:28.584950 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33cae564-16a9-42bb-a1a5-dc5c3257f56b-istio-envoy\") pod \"e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4\" (UID: \"33cae564-16a9-42bb-a1a5-dc5c3257f56b\") " pod="openshift-ingress/e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4" Nov 03 02:44:28.585273 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:28.585250 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33cae564-16a9-42bb-a1a5-dc5c3257f56b-istio-podinfo\") pod \"e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4\" (UID: \"33cae564-16a9-42bb-a1a5-dc5c3257f56b\") " pod="openshift-ingress/e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4" Nov 03 02:44:28.636104 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:28.636004 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33cae564-16a9-42bb-a1a5-dc5c3257f56b-istio-token\") pod \"e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4\" (UID: \"33cae564-16a9-42bb-a1a5-dc5c3257f56b\") " pod="openshift-ingress/e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4" Nov 03 02:44:28.636104 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:28.636088 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk8tn\" (UniqueName: \"kubernetes.io/projected/33cae564-16a9-42bb-a1a5-dc5c3257f56b-kube-api-access-bk8tn\") pod \"e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4\" (UID: \"33cae564-16a9-42bb-a1a5-dc5c3257f56b\") " pod="openshift-ingress/e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4" Nov 03 02:44:28.693726 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:28.693701 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4" Nov 03 02:44:28.852201 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:28.852164 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21802adf-b888-473e-8994-16b453c2e0b9" path="/var/lib/kubelet/pods/21802adf-b888-473e-8994-16b453c2e0b9/volumes" Nov 03 02:44:29.126911 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:29.126877 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4"] Nov 03 02:44:29.171194 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:44:29.171160 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33cae564_16a9_42bb_a1a5_dc5c3257f56b.slice/crio-568a4212a803d029fece1e38c59ebfaa780aadf6752bc8c3f77fc3edd88f4b3d WatchSource:0}: Error finding container 568a4212a803d029fece1e38c59ebfaa780aadf6752bc8c3f77fc3edd88f4b3d: Status 404 returned error can't find the container with id 568a4212a803d029fece1e38c59ebfaa780aadf6752bc8c3f77fc3edd88f4b3d Nov 03 02:44:29.173728 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:29.173694 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:44:29.173837 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:29.173766 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:44:29.173837 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:29.173795 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:44:29.493931 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:29.493845 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4" event={"ID":"33cae564-16a9-42bb-a1a5-dc5c3257f56b","Type":"ContainerStarted","Data":"eea83ba4c269a0fb4f25276b7d3c175be50b5f2d8f90be9f333bcf1dc73f65e3"} Nov 03 02:44:29.493931 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:29.493909 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4" event={"ID":"33cae564-16a9-42bb-a1a5-dc5c3257f56b","Type":"ContainerStarted","Data":"568a4212a803d029fece1e38c59ebfaa780aadf6752bc8c3f77fc3edd88f4b3d"} Nov 03 02:44:29.606143 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:29.606092 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4" podStartSLOduration=1.606079805 podStartE2EDuration="1.606079805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:44:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:44:29.605323324 +0000 UTC m=+3146.864521480" watchObservedRunningTime="2026-11-03 02:44:29.606079805 +0000 UTC m=+3146.865277953" Nov 03 02:44:29.661510 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:29.661473 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-965a07/auth-test-model-965a07-kserve-86544f48df-b4jcp" podUID="233b378d-0c02-4452-854f-563f18c1a66e" containerName="main" containerID="cri-o://51d5b4295c7ee99d5179b9cbc26e620e326e7409082b13abe8b761c8554e5dc1" gracePeriod=45 Nov 03 02:44:29.693918 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:29.693891 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4" Nov 03 02:44:29.828467 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:29.828425 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-965a07/auth-test-model-965a07-kserve-86544f48df-b4jcp" podUID="233b378d-0c02-4452-854f-563f18c1a66e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.152:8000/ready\": dial tcp 10.134.0.152:8000: connect: connection refused" Nov 03 02:44:29.922805 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:29.922781 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-965a07/auth-test-model-965a07-kserve-86544f48df-b4jcp" Nov 03 02:44:29.998458 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:29.998427 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/233b378d-0c02-4452-854f-563f18c1a66e-kube-api-access-942zn\" (UniqueName: \"kubernetes.io/projected/233b378d-0c02-4452-854f-563f18c1a66e-kube-api-access-942zn\") pod \"233b378d-0c02-4452-854f-563f18c1a66e\" (UID: \"233b378d-0c02-4452-854f-563f18c1a66e\") " Nov 03 02:44:29.998629 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:29.998528 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/233b378d-0c02-4452-854f-563f18c1a66e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/233b378d-0c02-4452-854f-563f18c1a66e-tmp-dir\") pod \"233b378d-0c02-4452-854f-563f18c1a66e\" (UID: \"233b378d-0c02-4452-854f-563f18c1a66e\") " Nov 03 02:44:29.998629 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:29.998554 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/233b378d-0c02-4452-854f-563f18c1a66e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/233b378d-0c02-4452-854f-563f18c1a66e-dshm\") pod \"233b378d-0c02-4452-854f-563f18c1a66e\" (UID: \"233b378d-0c02-4452-854f-563f18c1a66e\") " Nov 03 02:44:29.998629 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:29.998574 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/233b378d-0c02-4452-854f-563f18c1a66e-tls-certs\" (UniqueName: \"kubernetes.io/secret/233b378d-0c02-4452-854f-563f18c1a66e-tls-certs\") pod \"233b378d-0c02-4452-854f-563f18c1a66e\" (UID: \"233b378d-0c02-4452-854f-563f18c1a66e\") " Nov 03 02:44:29.998629 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:29.998607 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/233b378d-0c02-4452-854f-563f18c1a66e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/233b378d-0c02-4452-854f-563f18c1a66e-model-cache\") pod \"233b378d-0c02-4452-854f-563f18c1a66e\" (UID: \"233b378d-0c02-4452-854f-563f18c1a66e\") " Nov 03 02:44:29.998844 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:29.998640 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/233b378d-0c02-4452-854f-563f18c1a66e-home\" (UniqueName: \"kubernetes.io/empty-dir/233b378d-0c02-4452-854f-563f18c1a66e-home\") pod \"233b378d-0c02-4452-854f-563f18c1a66e\" (UID: \"233b378d-0c02-4452-854f-563f18c1a66e\") " Nov 03 02:44:29.998844 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:29.998786 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/233b378d-0c02-4452-854f-563f18c1a66e-tmp-dir" pod "233b378d-0c02-4452-854f-563f18c1a66e" (UID: "233b378d-0c02-4452-854f-563f18c1a66e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:44:29.998965 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:29.998912 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/233b378d-0c02-4452-854f-563f18c1a66e-home" pod "233b378d-0c02-4452-854f-563f18c1a66e" (UID: "233b378d-0c02-4452-854f-563f18c1a66e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:44:29.998965 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:29.998944 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/233b378d-0c02-4452-854f-563f18c1a66e-model-cache" pod "233b378d-0c02-4452-854f-563f18c1a66e" (UID: "233b378d-0c02-4452-854f-563f18c1a66e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:44:29.999218 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:29.999197 5605 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/233b378d-0c02-4452-854f-563f18c1a66e-model-cache\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:44:29.999266 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:29.999224 5605 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/233b378d-0c02-4452-854f-563f18c1a66e-home\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:44:29.999266 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:29.999240 5605 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/233b378d-0c02-4452-854f-563f18c1a66e-tmp-dir\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:44:30.000747 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:30.000727 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/233b378d-0c02-4452-854f-563f18c1a66e-dshm" pod "233b378d-0c02-4452-854f-563f18c1a66e" (UID: "233b378d-0c02-4452-854f-563f18c1a66e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:44:30.000827 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:30.000782 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/233b378d-0c02-4452-854f-563f18c1a66e-tls-certs" pod "233b378d-0c02-4452-854f-563f18c1a66e" (UID: "233b378d-0c02-4452-854f-563f18c1a66e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:44:30.000875 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:30.000838 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/233b378d-0c02-4452-854f-563f18c1a66e-kube-api-access-942zn" pod "233b378d-0c02-4452-854f-563f18c1a66e" (UID: "233b378d-0c02-4452-854f-563f18c1a66e"). InnerVolumeSpecName "kube-api-access-942zn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:44:30.100117 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:30.100081 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-942zn\" (UniqueName: \"kubernetes.io/projected/233b378d-0c02-4452-854f-563f18c1a66e-kube-api-access-942zn\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:44:30.100117 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:30.100112 5605 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/233b378d-0c02-4452-854f-563f18c1a66e-dshm\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:44:30.100398 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:30.100130 5605 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/233b378d-0c02-4452-854f-563f18c1a66e-tls-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:44:30.501057 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:30.500970 5605 generic.go:356] "Generic (PLEG): container finished" podID="233b378d-0c02-4452-854f-563f18c1a66e" containerID="51d5b4295c7ee99d5179b9cbc26e620e326e7409082b13abe8b761c8554e5dc1" exitCode=0 Nov 03 02:44:30.501057 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:30.501046 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-965a07/auth-test-model-965a07-kserve-86544f48df-b4jcp" Nov 03 02:44:30.501514 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:30.501062 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-965a07/auth-test-model-965a07-kserve-86544f48df-b4jcp" event={"ID":"233b378d-0c02-4452-854f-563f18c1a66e","Type":"ContainerDied","Data":"51d5b4295c7ee99d5179b9cbc26e620e326e7409082b13abe8b761c8554e5dc1"} Nov 03 02:44:30.501514 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:30.501107 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-965a07/auth-test-model-965a07-kserve-86544f48df-b4jcp" event={"ID":"233b378d-0c02-4452-854f-563f18c1a66e","Type":"ContainerDied","Data":"d3418640b162ee95be6a2f9ba431ada80412b1764cd93f6e38ef33ee3a0ff44e"} Nov 03 02:44:30.501514 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:30.501127 5605 scope.go:122] "RemoveContainer" containerID="51d5b4295c7ee99d5179b9cbc26e620e326e7409082b13abe8b761c8554e5dc1" Nov 03 02:44:30.511435 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:30.511419 5605 scope.go:122] "RemoveContainer" containerID="51d5b4295c7ee99d5179b9cbc26e620e326e7409082b13abe8b761c8554e5dc1" Nov 03 02:44:30.511685 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:44:30.511668 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51d5b4295c7ee99d5179b9cbc26e620e326e7409082b13abe8b761c8554e5dc1\": container with ID starting with 51d5b4295c7ee99d5179b9cbc26e620e326e7409082b13abe8b761c8554e5dc1 not found: ID does not exist" containerID="51d5b4295c7ee99d5179b9cbc26e620e326e7409082b13abe8b761c8554e5dc1" Nov 03 02:44:30.511752 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:30.511695 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51d5b4295c7ee99d5179b9cbc26e620e326e7409082b13abe8b761c8554e5dc1"} err="failed to get container status \"51d5b4295c7ee99d5179b9cbc26e620e326e7409082b13abe8b761c8554e5dc1\": rpc error: code = NotFound desc = could not find container \"51d5b4295c7ee99d5179b9cbc26e620e326e7409082b13abe8b761c8554e5dc1\": container with ID starting with 51d5b4295c7ee99d5179b9cbc26e620e326e7409082b13abe8b761c8554e5dc1 not found: ID does not exist" Nov 03 02:44:30.662273 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:30.662247 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-965a07/auth-test-model-965a07-kserve-86544f48df-b4jcp"] Nov 03 02:44:30.700049 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:30.700029 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4" Nov 03 02:44:30.708750 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:30.708727 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-965a07/auth-test-model-965a07-kserve-86544f48df-b4jcp"] Nov 03 02:44:30.850635 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:30.850609 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="233b378d-0c02-4452-854f-563f18c1a66e" path="/var/lib/kubelet/pods/233b378d-0c02-4452-854f-563f18c1a66e/volumes" Nov 03 02:44:31.509231 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:31.509202 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4" Nov 03 02:44:31.509985 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:31.509966 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4" Nov 03 02:44:32.698014 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:32.697980 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:44:36.242380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:36.242318 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-4616ec-84b7c6449f-xf5bh"] Nov 03 02:44:36.359807 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:36.359769 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4616ec-944fbf57f-n5n9d"] Nov 03 02:44:39.056531 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:39.056497 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbe69693-fb5t7"] Nov 03 02:44:39.056956 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:39.056943 5605 cpu_manager.go:414] "containerMap: removing container" podUID="233b378d-0c02-4452-854f-563f18c1a66e" containerName="main" Nov 03 02:44:39.057003 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:39.056956 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="233b378d-0c02-4452-854f-563f18c1a66e" containerName="main" Nov 03 02:44:39.057066 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:39.057056 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="233b378d-0c02-4452-854f-563f18c1a66e" containerName="main" Nov 03 02:44:39.060433 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:39.060414 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe69693-fb5t7" Nov 03 02:44:39.098795 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:39.098766 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbpp2\" (UniqueName: \"kubernetes.io/projected/4777275b-ba13-499d-81ef-1706e481f3f2-kube-api-access-hbpp2\") pod \"maas-api-revoke-sub-dbe69693-fb5t7\" (UID: \"4777275b-ba13-499d-81ef-1706e481f3f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe69693-fb5t7" Nov 03 02:44:39.098951 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:39.098934 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4777275b-ba13-499d-81ef-1706e481f3f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbe69693-fb5t7\" (UID: \"4777275b-ba13-499d-81ef-1706e481f3f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe69693-fb5t7" Nov 03 02:44:39.099001 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:39.098970 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4777275b-ba13-499d-81ef-1706e481f3f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbe69693-fb5t7\" (UID: \"4777275b-ba13-499d-81ef-1706e481f3f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe69693-fb5t7" Nov 03 02:44:39.157195 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:39.157166 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:44:39.157331 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:39.157213 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:44:39.157331 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:39.157233 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:44:39.157437 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:39.157419 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:44:39.159712 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:39.159695 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbe69693-fb5t7"] Nov 03 02:44:39.199695 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:39.199667 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4777275b-ba13-499d-81ef-1706e481f3f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbe69693-fb5t7\" (UID: \"4777275b-ba13-499d-81ef-1706e481f3f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe69693-fb5t7" Nov 03 02:44:39.199830 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:39.199707 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4777275b-ba13-499d-81ef-1706e481f3f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbe69693-fb5t7\" (UID: \"4777275b-ba13-499d-81ef-1706e481f3f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe69693-fb5t7" Nov 03 02:44:39.199830 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:39.199767 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbpp2\" (UniqueName: \"kubernetes.io/projected/4777275b-ba13-499d-81ef-1706e481f3f2-kube-api-access-hbpp2\") pod \"maas-api-revoke-sub-dbe69693-fb5t7\" (UID: \"4777275b-ba13-499d-81ef-1706e481f3f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe69693-fb5t7" Nov 03 02:44:39.200441 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:39.200415 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4777275b-ba13-499d-81ef-1706e481f3f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbe69693-fb5t7\" (UID: \"4777275b-ba13-499d-81ef-1706e481f3f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe69693-fb5t7" Nov 03 02:44:39.202226 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:39.202205 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4777275b-ba13-499d-81ef-1706e481f3f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbe69693-fb5t7\" (UID: \"4777275b-ba13-499d-81ef-1706e481f3f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe69693-fb5t7" Nov 03 02:44:39.254053 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:39.254023 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbpp2\" (UniqueName: \"kubernetes.io/projected/4777275b-ba13-499d-81ef-1706e481f3f2-kube-api-access-hbpp2\") pod \"maas-api-revoke-sub-dbe69693-fb5t7\" (UID: \"4777275b-ba13-499d-81ef-1706e481f3f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe69693-fb5t7" Nov 03 02:44:39.374243 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:39.374170 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe69693-fb5t7" Nov 03 02:44:39.747021 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:39.746948 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbe69693-fb5t7"] Nov 03 02:44:39.752427 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:44:39.752400 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4777275b_ba13_499d_81ef_1706e481f3f2.slice/crio-383564b0b132bdacaa53bcf9e65990108038d3f2d59c69af3d19b967834df7a1 WatchSource:0}: Error finding container 383564b0b132bdacaa53bcf9e65990108038d3f2d59c69af3d19b967834df7a1: Status 404 returned error can't find the container with id 383564b0b132bdacaa53bcf9e65990108038d3f2d59c69af3d19b967834df7a1 Nov 03 02:44:39.801909 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:39.801878 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9324d1e-8rvcj"] Nov 03 02:44:39.809040 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:39.809016 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9324d1e-8rvcj" Nov 03 02:44:39.857741 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:39.857713 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9324d1e-8rvcj"] Nov 03 02:44:39.909080 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:39.909054 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c10e9e9-acb8-4f2b-ae14-a8fde1a01aa4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9324d1e-8rvcj\" (UID: \"9c10e9e9-acb8-4f2b-ae14-a8fde1a01aa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9324d1e-8rvcj" Nov 03 02:44:39.909207 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:39.909098 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frgfq\" (UniqueName: \"kubernetes.io/projected/9c10e9e9-acb8-4f2b-ae14-a8fde1a01aa4-kube-api-access-frgfq\") pod \"maas-api-revoke-sub-d9324d1e-8rvcj\" (UID: \"9c10e9e9-acb8-4f2b-ae14-a8fde1a01aa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9324d1e-8rvcj" Nov 03 02:44:39.909377 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:39.909336 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c10e9e9-acb8-4f2b-ae14-a8fde1a01aa4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9324d1e-8rvcj\" (UID: \"9c10e9e9-acb8-4f2b-ae14-a8fde1a01aa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9324d1e-8rvcj" Nov 03 02:44:40.010239 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:40.010169 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c10e9e9-acb8-4f2b-ae14-a8fde1a01aa4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9324d1e-8rvcj\" (UID: \"9c10e9e9-acb8-4f2b-ae14-a8fde1a01aa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9324d1e-8rvcj" Nov 03 02:44:40.010381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:40.010255 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c10e9e9-acb8-4f2b-ae14-a8fde1a01aa4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9324d1e-8rvcj\" (UID: \"9c10e9e9-acb8-4f2b-ae14-a8fde1a01aa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9324d1e-8rvcj" Nov 03 02:44:40.010381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:40.010292 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frgfq\" (UniqueName: \"kubernetes.io/projected/9c10e9e9-acb8-4f2b-ae14-a8fde1a01aa4-kube-api-access-frgfq\") pod \"maas-api-revoke-sub-d9324d1e-8rvcj\" (UID: \"9c10e9e9-acb8-4f2b-ae14-a8fde1a01aa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9324d1e-8rvcj" Nov 03 02:44:40.010913 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:40.010886 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c10e9e9-acb8-4f2b-ae14-a8fde1a01aa4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9324d1e-8rvcj\" (UID: \"9c10e9e9-acb8-4f2b-ae14-a8fde1a01aa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9324d1e-8rvcj" Nov 03 02:44:40.012506 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:40.012486 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c10e9e9-acb8-4f2b-ae14-a8fde1a01aa4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9324d1e-8rvcj\" (UID: \"9c10e9e9-acb8-4f2b-ae14-a8fde1a01aa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9324d1e-8rvcj" Nov 03 02:44:40.063132 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:40.063111 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frgfq\" (UniqueName: \"kubernetes.io/projected/9c10e9e9-acb8-4f2b-ae14-a8fde1a01aa4-kube-api-access-frgfq\") pod \"maas-api-revoke-sub-d9324d1e-8rvcj\" (UID: \"9c10e9e9-acb8-4f2b-ae14-a8fde1a01aa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9324d1e-8rvcj" Nov 03 02:44:40.122921 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:40.122900 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9324d1e-8rvcj" Nov 03 02:44:40.489932 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:40.489904 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9324d1e-8rvcj"] Nov 03 02:44:40.495503 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:44:40.495476 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c10e9e9_acb8_4f2b_ae14_a8fde1a01aa4.slice/crio-65e4acff9dd94e45f368d15db8d5c474cc706868f8bb97c53137bdbbe2ff43e2 WatchSource:0}: Error finding container 65e4acff9dd94e45f368d15db8d5c474cc706868f8bb97c53137bdbbe2ff43e2: Status 404 returned error can't find the container with id 65e4acff9dd94e45f368d15db8d5c474cc706868f8bb97c53137bdbbe2ff43e2 Nov 03 02:44:40.559802 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:40.559775 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9324d1e-8rvcj" event={"ID":"9c10e9e9-acb8-4f2b-ae14-a8fde1a01aa4","Type":"ContainerStarted","Data":"65e4acff9dd94e45f368d15db8d5c474cc706868f8bb97c53137bdbbe2ff43e2"} Nov 03 02:44:40.561317 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:40.561290 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe69693-fb5t7" event={"ID":"4777275b-ba13-499d-81ef-1706e481f3f2","Type":"ContainerStarted","Data":"6e59891937a8f7c4bc8cc3503ec6b13c9e15e118f5d5d2291bdf784f09011410"} Nov 03 02:44:40.561454 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:40.561329 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe69693-fb5t7" event={"ID":"4777275b-ba13-499d-81ef-1706e481f3f2","Type":"ContainerStarted","Data":"383564b0b132bdacaa53bcf9e65990108038d3f2d59c69af3d19b967834df7a1"} Nov 03 02:44:40.670438 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:40.670396 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe69693-fb5t7" podStartSLOduration=1.6703837689999999 podStartE2EDuration="1.670383769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:44:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:44:40.668095044 +0000 UTC m=+3157.927293198" watchObservedRunningTime="2026-11-03 02:44:40.670383769 +0000 UTC m=+3157.929581924" Nov 03 02:44:41.243179 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:41.243128 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-4616ec-84b7c6449f-xf5bh" podUID="f7535c5c-da75-4fb2-8c06-94b2821a4f8b" containerName="payload-processing" containerID="cri-o://e251ddd8ef1b64097ac587c573960d1e51c8e35c5a8a5d3c2b5fc0342a0d74e9" gracePeriod=25 Nov 03 02:44:41.361172 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:41.361140 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4616ec-944fbf57f-n5n9d" podUID="00c96087-f0d3-434e-bceb-658d0f62c4e3" containerName="payload-pre-processing" containerID="cri-o://c15c4314627be745281a349061e5298f190263be5c69f9acb9bf83e6928c1877" gracePeriod=25 Nov 03 02:44:41.498101 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:41.498037 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-4616ec-84b7c6449f-xf5bh" Nov 03 02:44:41.523216 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:41.523189 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7535c5c-da75-4fb2-8c06-94b2821a4f8b-kube-api-access-hrchx\" (UniqueName: \"kubernetes.io/projected/f7535c5c-da75-4fb2-8c06-94b2821a4f8b-kube-api-access-hrchx\") pod \"f7535c5c-da75-4fb2-8c06-94b2821a4f8b\" (UID: \"f7535c5c-da75-4fb2-8c06-94b2821a4f8b\") " Nov 03 02:44:41.523447 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:41.523424 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7535c5c-da75-4fb2-8c06-94b2821a4f8b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7535c5c-da75-4fb2-8c06-94b2821a4f8b-plugins-config-volume\") pod \"f7535c5c-da75-4fb2-8c06-94b2821a4f8b\" (UID: \"f7535c5c-da75-4fb2-8c06-94b2821a4f8b\") " Nov 03 02:44:41.523800 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:41.523776 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7535c5c-da75-4fb2-8c06-94b2821a4f8b-plugins-config-volume" pod "f7535c5c-da75-4fb2-8c06-94b2821a4f8b" (UID: "f7535c5c-da75-4fb2-8c06-94b2821a4f8b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:44:41.525208 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:41.525184 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7535c5c-da75-4fb2-8c06-94b2821a4f8b-kube-api-access-hrchx" pod "f7535c5c-da75-4fb2-8c06-94b2821a4f8b" (UID: "f7535c5c-da75-4fb2-8c06-94b2821a4f8b"). InnerVolumeSpecName "kube-api-access-hrchx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:44:41.569710 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:41.569677 5605 generic.go:356] "Generic (PLEG): container finished" podID="9c10e9e9-acb8-4f2b-ae14-a8fde1a01aa4" containerID="1579b2d465a044636e5cb31c43a4dacf9d761ba20fed61153f0eb5e03c4d119c" exitCode=0 Nov 03 02:44:41.569953 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:41.569755 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9324d1e-8rvcj" event={"ID":"9c10e9e9-acb8-4f2b-ae14-a8fde1a01aa4","Type":"ContainerDied","Data":"1579b2d465a044636e5cb31c43a4dacf9d761ba20fed61153f0eb5e03c4d119c"} Nov 03 02:44:41.571734 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:41.571708 5605 generic.go:356] "Generic (PLEG): container finished" podID="00c96087-f0d3-434e-bceb-658d0f62c4e3" containerID="c15c4314627be745281a349061e5298f190263be5c69f9acb9bf83e6928c1877" exitCode=0 Nov 03 02:44:41.571899 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:41.571758 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4616ec-944fbf57f-n5n9d" event={"ID":"00c96087-f0d3-434e-bceb-658d0f62c4e3","Type":"ContainerDied","Data":"c15c4314627be745281a349061e5298f190263be5c69f9acb9bf83e6928c1877"} Nov 03 02:44:41.573693 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:41.573668 5605 generic.go:356] "Generic (PLEG): container finished" podID="4777275b-ba13-499d-81ef-1706e481f3f2" containerID="6e59891937a8f7c4bc8cc3503ec6b13c9e15e118f5d5d2291bdf784f09011410" exitCode=0 Nov 03 02:44:41.573795 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:41.573706 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe69693-fb5t7" event={"ID":"4777275b-ba13-499d-81ef-1706e481f3f2","Type":"ContainerDied","Data":"6e59891937a8f7c4bc8cc3503ec6b13c9e15e118f5d5d2291bdf784f09011410"} Nov 03 02:44:41.575218 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:41.575193 5605 generic.go:356] "Generic (PLEG): container finished" podID="f7535c5c-da75-4fb2-8c06-94b2821a4f8b" containerID="e251ddd8ef1b64097ac587c573960d1e51c8e35c5a8a5d3c2b5fc0342a0d74e9" exitCode=0 Nov 03 02:44:41.575331 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:41.575236 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-4616ec-84b7c6449f-xf5bh" Nov 03 02:44:41.575331 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:41.575238 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-4616ec-84b7c6449f-xf5bh" event={"ID":"f7535c5c-da75-4fb2-8c06-94b2821a4f8b","Type":"ContainerDied","Data":"e251ddd8ef1b64097ac587c573960d1e51c8e35c5a8a5d3c2b5fc0342a0d74e9"} Nov 03 02:44:41.575477 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:41.575386 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-4616ec-84b7c6449f-xf5bh" event={"ID":"f7535c5c-da75-4fb2-8c06-94b2821a4f8b","Type":"ContainerDied","Data":"a8f96955b04d5519d6c1e5ced00496062a38b637e5d2b787d8d115fee1e13fe8"} Nov 03 02:44:41.575477 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:41.575420 5605 scope.go:122] "RemoveContainer" containerID="e251ddd8ef1b64097ac587c573960d1e51c8e35c5a8a5d3c2b5fc0342a0d74e9" Nov 03 02:44:41.586470 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:41.586448 5605 scope.go:122] "RemoveContainer" containerID="e251ddd8ef1b64097ac587c573960d1e51c8e35c5a8a5d3c2b5fc0342a0d74e9" Nov 03 02:44:41.586754 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:44:41.586731 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e251ddd8ef1b64097ac587c573960d1e51c8e35c5a8a5d3c2b5fc0342a0d74e9\": container with ID starting with e251ddd8ef1b64097ac587c573960d1e51c8e35c5a8a5d3c2b5fc0342a0d74e9 not found: ID does not exist" containerID="e251ddd8ef1b64097ac587c573960d1e51c8e35c5a8a5d3c2b5fc0342a0d74e9" Nov 03 02:44:41.586819 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:41.586762 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e251ddd8ef1b64097ac587c573960d1e51c8e35c5a8a5d3c2b5fc0342a0d74e9"} err="failed to get container status \"e251ddd8ef1b64097ac587c573960d1e51c8e35c5a8a5d3c2b5fc0342a0d74e9\": rpc error: code = NotFound desc = could not find container \"e251ddd8ef1b64097ac587c573960d1e51c8e35c5a8a5d3c2b5fc0342a0d74e9\": container with ID starting with e251ddd8ef1b64097ac587c573960d1e51c8e35c5a8a5d3c2b5fc0342a0d74e9 not found: ID does not exist" Nov 03 02:44:41.614007 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:41.613983 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4616ec-944fbf57f-n5n9d" Nov 03 02:44:41.624342 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:41.624322 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00c96087-f0d3-434e-bceb-658d0f62c4e3-kube-api-access-wggzx\" (UniqueName: \"kubernetes.io/projected/00c96087-f0d3-434e-bceb-658d0f62c4e3-kube-api-access-wggzx\") pod \"00c96087-f0d3-434e-bceb-658d0f62c4e3\" (UID: \"00c96087-f0d3-434e-bceb-658d0f62c4e3\") " Nov 03 02:44:41.624467 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:41.624418 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00c96087-f0d3-434e-bceb-658d0f62c4e3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00c96087-f0d3-434e-bceb-658d0f62c4e3-plugins-config-volume\") pod \"00c96087-f0d3-434e-bceb-658d0f62c4e3\" (UID: \"00c96087-f0d3-434e-bceb-658d0f62c4e3\") " Nov 03 02:44:41.624846 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:41.624831 5605 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7535c5c-da75-4fb2-8c06-94b2821a4f8b-plugins-config-volume\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:44:41.624892 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:41.624859 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrchx\" (UniqueName: \"kubernetes.io/projected/f7535c5c-da75-4fb2-8c06-94b2821a4f8b-kube-api-access-hrchx\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:44:41.624892 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:41.624826 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00c96087-f0d3-434e-bceb-658d0f62c4e3-plugins-config-volume" pod "00c96087-f0d3-434e-bceb-658d0f62c4e3" (UID: "00c96087-f0d3-434e-bceb-658d0f62c4e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:44:41.626306 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:41.626289 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00c96087-f0d3-434e-bceb-658d0f62c4e3-kube-api-access-wggzx" pod "00c96087-f0d3-434e-bceb-658d0f62c4e3" (UID: "00c96087-f0d3-434e-bceb-658d0f62c4e3"). InnerVolumeSpecName "kube-api-access-wggzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:44:41.709789 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:41.709765 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-4616ec-84b7c6449f-xf5bh"] Nov 03 02:44:41.715367 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:41.715335 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-4616ec-84b7c6449f-xf5bh"] Nov 03 02:44:41.726110 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:41.726086 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wggzx\" (UniqueName: \"kubernetes.io/projected/00c96087-f0d3-434e-bceb-658d0f62c4e3-kube-api-access-wggzx\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:44:41.726205 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:41.726114 5605 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00c96087-f0d3-434e-bceb-658d0f62c4e3-plugins-config-volume\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:44:42.207052 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:42.207010 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-7273af/auth-test-model-7273af-kserve-dfd448c77-g45rq" podUID="1b495ce9-2ec7-4132-9769-8d1526622c48" containerName="main" containerID="cri-o://ae5caa9da9ece80fae3cf1bc1995cc51f6ad9d0e5b68c4b2c700e99edc1fa3b0" gracePeriod=45 Nov 03 02:44:42.463835 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:42.463781 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7273af/auth-test-model-7273af-kserve-dfd448c77-g45rq" Nov 03 02:44:42.534033 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:42.534009 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b495ce9-2ec7-4132-9769-8d1526622c48-home\" (UniqueName: \"kubernetes.io/empty-dir/1b495ce9-2ec7-4132-9769-8d1526622c48-home\") pod \"1b495ce9-2ec7-4132-9769-8d1526622c48\" (UID: \"1b495ce9-2ec7-4132-9769-8d1526622c48\") " Nov 03 02:44:42.534147 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:42.534056 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b495ce9-2ec7-4132-9769-8d1526622c48-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b495ce9-2ec7-4132-9769-8d1526622c48-dshm\") pod \"1b495ce9-2ec7-4132-9769-8d1526622c48\" (UID: \"1b495ce9-2ec7-4132-9769-8d1526622c48\") " Nov 03 02:44:42.534147 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:42.534117 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b495ce9-2ec7-4132-9769-8d1526622c48-kube-api-access-4566t\" (UniqueName: \"kubernetes.io/projected/1b495ce9-2ec7-4132-9769-8d1526622c48-kube-api-access-4566t\") pod \"1b495ce9-2ec7-4132-9769-8d1526622c48\" (UID: \"1b495ce9-2ec7-4132-9769-8d1526622c48\") " Nov 03 02:44:42.534147 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:42.534145 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b495ce9-2ec7-4132-9769-8d1526622c48-tls-certs\" (UniqueName: \"kubernetes.io/secret/1b495ce9-2ec7-4132-9769-8d1526622c48-tls-certs\") pod \"1b495ce9-2ec7-4132-9769-8d1526622c48\" (UID: \"1b495ce9-2ec7-4132-9769-8d1526622c48\") " Nov 03 02:44:42.534272 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:42.534211 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b495ce9-2ec7-4132-9769-8d1526622c48-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b495ce9-2ec7-4132-9769-8d1526622c48-model-cache\") pod \"1b495ce9-2ec7-4132-9769-8d1526622c48\" (UID: \"1b495ce9-2ec7-4132-9769-8d1526622c48\") " Nov 03 02:44:42.534272 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:42.534232 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b495ce9-2ec7-4132-9769-8d1526622c48-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b495ce9-2ec7-4132-9769-8d1526622c48-tmp-dir\") pod \"1b495ce9-2ec7-4132-9769-8d1526622c48\" (UID: \"1b495ce9-2ec7-4132-9769-8d1526622c48\") " Nov 03 02:44:42.534422 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:42.534270 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b495ce9-2ec7-4132-9769-8d1526622c48-home" pod "1b495ce9-2ec7-4132-9769-8d1526622c48" (UID: "1b495ce9-2ec7-4132-9769-8d1526622c48"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:44:42.534517 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:42.534496 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b495ce9-2ec7-4132-9769-8d1526622c48-tmp-dir" pod "1b495ce9-2ec7-4132-9769-8d1526622c48" (UID: "1b495ce9-2ec7-4132-9769-8d1526622c48"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:44:42.534587 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:42.534512 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b495ce9-2ec7-4132-9769-8d1526622c48-model-cache" pod "1b495ce9-2ec7-4132-9769-8d1526622c48" (UID: "1b495ce9-2ec7-4132-9769-8d1526622c48"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:44:42.534850 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:42.534830 5605 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b495ce9-2ec7-4132-9769-8d1526622c48-model-cache\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:44:42.534906 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:42.534858 5605 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b495ce9-2ec7-4132-9769-8d1526622c48-tmp-dir\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:44:42.534906 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:42.534873 5605 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b495ce9-2ec7-4132-9769-8d1526622c48-home\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:44:42.536271 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:42.536246 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b495ce9-2ec7-4132-9769-8d1526622c48-dshm" pod "1b495ce9-2ec7-4132-9769-8d1526622c48" (UID: "1b495ce9-2ec7-4132-9769-8d1526622c48"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:44:42.536271 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:42.536248 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b495ce9-2ec7-4132-9769-8d1526622c48-kube-api-access-4566t" pod "1b495ce9-2ec7-4132-9769-8d1526622c48" (UID: "1b495ce9-2ec7-4132-9769-8d1526622c48"). InnerVolumeSpecName "kube-api-access-4566t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:44:42.536435 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:42.536286 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b495ce9-2ec7-4132-9769-8d1526622c48-tls-certs" pod "1b495ce9-2ec7-4132-9769-8d1526622c48" (UID: "1b495ce9-2ec7-4132-9769-8d1526622c48"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:44:42.584064 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:42.584043 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4616ec-944fbf57f-n5n9d" Nov 03 02:44:42.584215 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:42.584078 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4616ec-944fbf57f-n5n9d" event={"ID":"00c96087-f0d3-434e-bceb-658d0f62c4e3","Type":"ContainerDied","Data":"c534b62bc9e8f1d6f317934e7fc8e7eae2673271244d2a891acd5ffefbea4011"} Nov 03 02:44:42.584215 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:42.584134 5605 scope.go:122] "RemoveContainer" containerID="c15c4314627be745281a349061e5298f190263be5c69f9acb9bf83e6928c1877" Nov 03 02:44:42.587016 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:42.586989 5605 generic.go:356] "Generic (PLEG): container finished" podID="1b495ce9-2ec7-4132-9769-8d1526622c48" containerID="ae5caa9da9ece80fae3cf1bc1995cc51f6ad9d0e5b68c4b2c700e99edc1fa3b0" exitCode=0 Nov 03 02:44:42.587127 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:42.587036 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7273af/auth-test-model-7273af-kserve-dfd448c77-g45rq" Nov 03 02:44:42.587127 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:42.587069 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7273af/auth-test-model-7273af-kserve-dfd448c77-g45rq" event={"ID":"1b495ce9-2ec7-4132-9769-8d1526622c48","Type":"ContainerDied","Data":"ae5caa9da9ece80fae3cf1bc1995cc51f6ad9d0e5b68c4b2c700e99edc1fa3b0"} Nov 03 02:44:42.587127 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:42.587110 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7273af/auth-test-model-7273af-kserve-dfd448c77-g45rq" event={"ID":"1b495ce9-2ec7-4132-9769-8d1526622c48","Type":"ContainerDied","Data":"1b7295a215c240503578d460ab39cccac7c1aa3c3084ac3ebd62bcbcfb0a3433"} Nov 03 02:44:42.596126 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:42.596104 5605 scope.go:122] "RemoveContainer" containerID="ae5caa9da9ece80fae3cf1bc1995cc51f6ad9d0e5b68c4b2c700e99edc1fa3b0" Nov 03 02:44:42.611455 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:42.611268 5605 scope.go:122] "RemoveContainer" containerID="ae5caa9da9ece80fae3cf1bc1995cc51f6ad9d0e5b68c4b2c700e99edc1fa3b0" Nov 03 02:44:42.611688 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:44:42.611663 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae5caa9da9ece80fae3cf1bc1995cc51f6ad9d0e5b68c4b2c700e99edc1fa3b0\": container with ID starting with ae5caa9da9ece80fae3cf1bc1995cc51f6ad9d0e5b68c4b2c700e99edc1fa3b0 not found: ID does not exist" containerID="ae5caa9da9ece80fae3cf1bc1995cc51f6ad9d0e5b68c4b2c700e99edc1fa3b0" Nov 03 02:44:42.611763 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:42.611697 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae5caa9da9ece80fae3cf1bc1995cc51f6ad9d0e5b68c4b2c700e99edc1fa3b0"} err="failed to get container status \"ae5caa9da9ece80fae3cf1bc1995cc51f6ad9d0e5b68c4b2c700e99edc1fa3b0\": rpc error: code = NotFound desc = could not find container \"ae5caa9da9ece80fae3cf1bc1995cc51f6ad9d0e5b68c4b2c700e99edc1fa3b0\": container with ID starting with ae5caa9da9ece80fae3cf1bc1995cc51f6ad9d0e5b68c4b2c700e99edc1fa3b0 not found: ID does not exist" Nov 03 02:44:42.636821 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:42.636518 5605 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b495ce9-2ec7-4132-9769-8d1526622c48-dshm\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:44:42.636821 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:42.636541 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4566t\" (UniqueName: \"kubernetes.io/projected/1b495ce9-2ec7-4132-9769-8d1526622c48-kube-api-access-4566t\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:44:42.636821 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:42.636558 5605 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b495ce9-2ec7-4132-9769-8d1526622c48-tls-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:44:42.718587 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:42.718528 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe69693-fb5t7" Nov 03 02:44:42.737114 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:42.736929 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4777275b-ba13-499d-81ef-1706e481f3f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4777275b-ba13-499d-81ef-1706e481f3f2-maas-api-openshift-ca\") pod \"4777275b-ba13-499d-81ef-1706e481f3f2\" (UID: \"4777275b-ba13-499d-81ef-1706e481f3f2\") " Nov 03 02:44:42.737114 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:42.737000 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4777275b-ba13-499d-81ef-1706e481f3f2-kube-api-access-hbpp2\" (UniqueName: \"kubernetes.io/projected/4777275b-ba13-499d-81ef-1706e481f3f2-kube-api-access-hbpp2\") pod \"4777275b-ba13-499d-81ef-1706e481f3f2\" (UID: \"4777275b-ba13-499d-81ef-1706e481f3f2\") " Nov 03 02:44:42.737114 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:42.737086 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4777275b-ba13-499d-81ef-1706e481f3f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4777275b-ba13-499d-81ef-1706e481f3f2-maas-api-xks-ca\") pod \"4777275b-ba13-499d-81ef-1706e481f3f2\" (UID: \"4777275b-ba13-499d-81ef-1706e481f3f2\") " Nov 03 02:44:42.737529 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:42.737503 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4777275b-ba13-499d-81ef-1706e481f3f2-maas-api-openshift-ca" pod "4777275b-ba13-499d-81ef-1706e481f3f2" (UID: "4777275b-ba13-499d-81ef-1706e481f3f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:44:42.739187 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:42.739162 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4777275b-ba13-499d-81ef-1706e481f3f2-maas-api-xks-ca" pod "4777275b-ba13-499d-81ef-1706e481f3f2" (UID: "4777275b-ba13-499d-81ef-1706e481f3f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:44:42.739260 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:42.739236 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4777275b-ba13-499d-81ef-1706e481f3f2-kube-api-access-hbpp2" pod "4777275b-ba13-499d-81ef-1706e481f3f2" (UID: "4777275b-ba13-499d-81ef-1706e481f3f2"). InnerVolumeSpecName "kube-api-access-hbpp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:44:42.741326 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:42.741304 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7273af/auth-test-model-7273af-kserve-dfd448c77-g45rq"] Nov 03 02:44:42.742123 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:42.742106 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9324d1e-8rvcj" Nov 03 02:44:42.743390 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:42.743346 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7273af/auth-test-model-7273af-kserve-dfd448c77-g45rq"] Nov 03 02:44:42.838828 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:42.838801 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c10e9e9-acb8-4f2b-ae14-a8fde1a01aa4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c10e9e9-acb8-4f2b-ae14-a8fde1a01aa4-maas-api-xks-ca\") pod \"9c10e9e9-acb8-4f2b-ae14-a8fde1a01aa4\" (UID: \"9c10e9e9-acb8-4f2b-ae14-a8fde1a01aa4\") " Nov 03 02:44:42.839000 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:42.838870 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c10e9e9-acb8-4f2b-ae14-a8fde1a01aa4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c10e9e9-acb8-4f2b-ae14-a8fde1a01aa4-maas-api-openshift-ca\") pod \"9c10e9e9-acb8-4f2b-ae14-a8fde1a01aa4\" (UID: \"9c10e9e9-acb8-4f2b-ae14-a8fde1a01aa4\") " Nov 03 02:44:42.839000 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:42.838921 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c10e9e9-acb8-4f2b-ae14-a8fde1a01aa4-kube-api-access-frgfq\" (UniqueName: \"kubernetes.io/projected/9c10e9e9-acb8-4f2b-ae14-a8fde1a01aa4-kube-api-access-frgfq\") pod \"9c10e9e9-acb8-4f2b-ae14-a8fde1a01aa4\" (UID: \"9c10e9e9-acb8-4f2b-ae14-a8fde1a01aa4\") " Nov 03 02:44:42.839228 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:42.839204 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c10e9e9-acb8-4f2b-ae14-a8fde1a01aa4-maas-api-openshift-ca" pod "9c10e9e9-acb8-4f2b-ae14-a8fde1a01aa4" (UID: "9c10e9e9-acb8-4f2b-ae14-a8fde1a01aa4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:44:42.839324 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:42.839305 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4777275b-ba13-499d-81ef-1706e481f3f2-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:44:42.839404 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:42.839331 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c10e9e9-acb8-4f2b-ae14-a8fde1a01aa4-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:44:42.839404 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:42.839348 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4777275b-ba13-499d-81ef-1706e481f3f2-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:44:42.839404 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:42.839391 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbpp2\" (UniqueName: \"kubernetes.io/projected/4777275b-ba13-499d-81ef-1706e481f3f2-kube-api-access-hbpp2\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:44:42.840820 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:42.840799 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c10e9e9-acb8-4f2b-ae14-a8fde1a01aa4-maas-api-xks-ca" pod "9c10e9e9-acb8-4f2b-ae14-a8fde1a01aa4" (UID: "9c10e9e9-acb8-4f2b-ae14-a8fde1a01aa4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:44:42.840876 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:42.840862 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c10e9e9-acb8-4f2b-ae14-a8fde1a01aa4-kube-api-access-frgfq" pod "9c10e9e9-acb8-4f2b-ae14-a8fde1a01aa4" (UID: "9c10e9e9-acb8-4f2b-ae14-a8fde1a01aa4"). InnerVolumeSpecName "kube-api-access-frgfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:44:42.851063 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:42.851035 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b495ce9-2ec7-4132-9769-8d1526622c48" path="/var/lib/kubelet/pods/1b495ce9-2ec7-4132-9769-8d1526622c48/volumes" Nov 03 02:44:42.851613 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:42.851595 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7535c5c-da75-4fb2-8c06-94b2821a4f8b" path="/var/lib/kubelet/pods/f7535c5c-da75-4fb2-8c06-94b2821a4f8b/volumes" Nov 03 02:44:42.852729 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:42.852679 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4616ec-944fbf57f-n5n9d"] Nov 03 02:44:42.855795 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:42.855772 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4616ec-944fbf57f-n5n9d"] Nov 03 02:44:42.940537 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:42.940456 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c10e9e9-acb8-4f2b-ae14-a8fde1a01aa4-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:44:42.940537 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:42.940488 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frgfq\" (UniqueName: \"kubernetes.io/projected/9c10e9e9-acb8-4f2b-ae14-a8fde1a01aa4-kube-api-access-frgfq\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:44:43.595678 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:43.595645 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9324d1e-8rvcj" Nov 03 02:44:43.596116 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:43.595647 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9324d1e-8rvcj" event={"ID":"9c10e9e9-acb8-4f2b-ae14-a8fde1a01aa4","Type":"ContainerDied","Data":"65e4acff9dd94e45f368d15db8d5c474cc706868f8bb97c53137bdbbe2ff43e2"} Nov 03 02:44:43.596116 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:43.595773 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65e4acff9dd94e45f368d15db8d5c474cc706868f8bb97c53137bdbbe2ff43e2" Nov 03 02:44:43.599397 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:43.598142 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe69693-fb5t7" event={"ID":"4777275b-ba13-499d-81ef-1706e481f3f2","Type":"ContainerDied","Data":"383564b0b132bdacaa53bcf9e65990108038d3f2d59c69af3d19b967834df7a1"} Nov 03 02:44:43.599397 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:43.598185 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="383564b0b132bdacaa53bcf9e65990108038d3f2d59c69af3d19b967834df7a1" Nov 03 02:44:43.599397 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:43.598241 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe69693-fb5t7" Nov 03 02:44:44.832838 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:44.832798 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-038c21-openshift-default-5d5c88569c-4r995"] Nov 03 02:44:44.833339 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:44.833237 5605 cpu_manager.go:414] "containerMap: removing container" podUID="00c96087-f0d3-434e-bceb-658d0f62c4e3" containerName="payload-pre-processing" Nov 03 02:44:44.833339 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:44.833248 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00c96087-f0d3-434e-bceb-658d0f62c4e3" containerName="payload-pre-processing" Nov 03 02:44:44.833339 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:44.833264 5605 cpu_manager.go:414] "containerMap: removing container" podUID="4777275b-ba13-499d-81ef-1706e481f3f2" containerName="revoke-keys" Nov 03 02:44:44.833339 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:44.833269 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4777275b-ba13-499d-81ef-1706e481f3f2" containerName="revoke-keys" Nov 03 02:44:44.833339 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:44.833278 5605 cpu_manager.go:414] "containerMap: removing container" podUID="9c10e9e9-acb8-4f2b-ae14-a8fde1a01aa4" containerName="revoke-keys" Nov 03 02:44:44.833339 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:44.833283 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c10e9e9-acb8-4f2b-ae14-a8fde1a01aa4" containerName="revoke-keys" Nov 03 02:44:44.833339 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:44.833294 5605 cpu_manager.go:414] "containerMap: removing container" podUID="f7535c5c-da75-4fb2-8c06-94b2821a4f8b" containerName="payload-processing" Nov 03 02:44:44.833339 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:44.833299 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7535c5c-da75-4fb2-8c06-94b2821a4f8b" containerName="payload-processing" Nov 03 02:44:44.833339 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:44.833312 5605 cpu_manager.go:414] "containerMap: removing container" podUID="1b495ce9-2ec7-4132-9769-8d1526622c48" containerName="main" Nov 03 02:44:44.833339 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:44.833318 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b495ce9-2ec7-4132-9769-8d1526622c48" containerName="main" Nov 03 02:44:44.833902 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:44.833402 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="4777275b-ba13-499d-81ef-1706e481f3f2" containerName="revoke-keys" Nov 03 02:44:44.833902 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:44.833409 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c10e9e9-acb8-4f2b-ae14-a8fde1a01aa4" containerName="revoke-keys" Nov 03 02:44:44.833902 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:44.833418 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7535c5c-da75-4fb2-8c06-94b2821a4f8b" containerName="payload-processing" Nov 03 02:44:44.833902 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:44.833427 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b495ce9-2ec7-4132-9769-8d1526622c48" containerName="main" Nov 03 02:44:44.833902 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:44.833434 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="00c96087-f0d3-434e-bceb-658d0f62c4e3" containerName="payload-pre-processing" Nov 03 02:44:44.838070 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:44.838044 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-038c21-openshift-default-5d5c88569c-4r995" Nov 03 02:44:44.853028 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:44.852999 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00c96087-f0d3-434e-bceb-658d0f62c4e3" path="/var/lib/kubelet/pods/00c96087-f0d3-434e-bceb-658d0f62c4e3/volumes" Nov 03 02:44:44.859519 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:44.859499 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7c23f7b-4a52-4a24-8fce-ba087104fca5-istio-data\") pod \"e2e-explicit-038c21-openshift-default-5d5c88569c-4r995\" (UID: \"f7c23f7b-4a52-4a24-8fce-ba087104fca5\") " pod="openshift-ingress/e2e-explicit-038c21-openshift-default-5d5c88569c-4r995" Nov 03 02:44:44.859617 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:44.859572 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7c23f7b-4a52-4a24-8fce-ba087104fca5-istio-podinfo\") pod \"e2e-explicit-038c21-openshift-default-5d5c88569c-4r995\" (UID: \"f7c23f7b-4a52-4a24-8fce-ba087104fca5\") " pod="openshift-ingress/e2e-explicit-038c21-openshift-default-5d5c88569c-4r995" Nov 03 02:44:44.859761 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:44.859742 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7c23f7b-4a52-4a24-8fce-ba087104fca5-workload-certs\") pod \"e2e-explicit-038c21-openshift-default-5d5c88569c-4r995\" (UID: \"f7c23f7b-4a52-4a24-8fce-ba087104fca5\") " pod="openshift-ingress/e2e-explicit-038c21-openshift-default-5d5c88569c-4r995" Nov 03 02:44:44.859839 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:44.859776 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7c23f7b-4a52-4a24-8fce-ba087104fca5-istio-envoy\") pod \"e2e-explicit-038c21-openshift-default-5d5c88569c-4r995\" (UID: \"f7c23f7b-4a52-4a24-8fce-ba087104fca5\") " pod="openshift-ingress/e2e-explicit-038c21-openshift-default-5d5c88569c-4r995" Nov 03 02:44:44.859910 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:44.859886 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djn5v\" (UniqueName: \"kubernetes.io/projected/f7c23f7b-4a52-4a24-8fce-ba087104fca5-kube-api-access-djn5v\") pod \"e2e-explicit-038c21-openshift-default-5d5c88569c-4r995\" (UID: \"f7c23f7b-4a52-4a24-8fce-ba087104fca5\") " pod="openshift-ingress/e2e-explicit-038c21-openshift-default-5d5c88569c-4r995" Nov 03 02:44:44.860027 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:44.859998 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7c23f7b-4a52-4a24-8fce-ba087104fca5-istiod-ca-cert\") pod \"e2e-explicit-038c21-openshift-default-5d5c88569c-4r995\" (UID: \"f7c23f7b-4a52-4a24-8fce-ba087104fca5\") " pod="openshift-ingress/e2e-explicit-038c21-openshift-default-5d5c88569c-4r995" Nov 03 02:44:44.860151 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:44.860079 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7c23f7b-4a52-4a24-8fce-ba087104fca5-credential-socket\") pod \"e2e-explicit-038c21-openshift-default-5d5c88569c-4r995\" (UID: \"f7c23f7b-4a52-4a24-8fce-ba087104fca5\") " pod="openshift-ingress/e2e-explicit-038c21-openshift-default-5d5c88569c-4r995" Nov 03 02:44:44.860214 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:44.860175 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7c23f7b-4a52-4a24-8fce-ba087104fca5-workload-socket\") pod \"e2e-explicit-038c21-openshift-default-5d5c88569c-4r995\" (UID: \"f7c23f7b-4a52-4a24-8fce-ba087104fca5\") " pod="openshift-ingress/e2e-explicit-038c21-openshift-default-5d5c88569c-4r995" Nov 03 02:44:44.860214 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:44.860202 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7c23f7b-4a52-4a24-8fce-ba087104fca5-istio-token\") pod \"e2e-explicit-038c21-openshift-default-5d5c88569c-4r995\" (UID: \"f7c23f7b-4a52-4a24-8fce-ba087104fca5\") " pod="openshift-ingress/e2e-explicit-038c21-openshift-default-5d5c88569c-4r995" Nov 03 02:44:44.910737 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:44.910704 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-038c21-openshift-default-5d5c88569c-4r995"] Nov 03 02:44:44.936339 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:44.936311 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-038c21-openshift-default-dockercfg-dz98t\"" Nov 03 02:44:44.960919 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:44.960895 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7c23f7b-4a52-4a24-8fce-ba087104fca5-istio-data\") pod \"e2e-explicit-038c21-openshift-default-5d5c88569c-4r995\" (UID: \"f7c23f7b-4a52-4a24-8fce-ba087104fca5\") " pod="openshift-ingress/e2e-explicit-038c21-openshift-default-5d5c88569c-4r995" Nov 03 02:44:44.961054 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:44.960958 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7c23f7b-4a52-4a24-8fce-ba087104fca5-istio-podinfo\") pod \"e2e-explicit-038c21-openshift-default-5d5c88569c-4r995\" (UID: \"f7c23f7b-4a52-4a24-8fce-ba087104fca5\") " pod="openshift-ingress/e2e-explicit-038c21-openshift-default-5d5c88569c-4r995" Nov 03 02:44:44.961054 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:44.960979 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7c23f7b-4a52-4a24-8fce-ba087104fca5-workload-certs\") pod \"e2e-explicit-038c21-openshift-default-5d5c88569c-4r995\" (UID: \"f7c23f7b-4a52-4a24-8fce-ba087104fca5\") " pod="openshift-ingress/e2e-explicit-038c21-openshift-default-5d5c88569c-4r995" Nov 03 02:44:44.961054 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:44.960995 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7c23f7b-4a52-4a24-8fce-ba087104fca5-istio-envoy\") pod \"e2e-explicit-038c21-openshift-default-5d5c88569c-4r995\" (UID: \"f7c23f7b-4a52-4a24-8fce-ba087104fca5\") " pod="openshift-ingress/e2e-explicit-038c21-openshift-default-5d5c88569c-4r995" Nov 03 02:44:44.961054 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:44.961017 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djn5v\" (UniqueName: \"kubernetes.io/projected/f7c23f7b-4a52-4a24-8fce-ba087104fca5-kube-api-access-djn5v\") pod \"e2e-explicit-038c21-openshift-default-5d5c88569c-4r995\" (UID: \"f7c23f7b-4a52-4a24-8fce-ba087104fca5\") " pod="openshift-ingress/e2e-explicit-038c21-openshift-default-5d5c88569c-4r995" Nov 03 02:44:44.961258 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:44.961071 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7c23f7b-4a52-4a24-8fce-ba087104fca5-istiod-ca-cert\") pod \"e2e-explicit-038c21-openshift-default-5d5c88569c-4r995\" (UID: \"f7c23f7b-4a52-4a24-8fce-ba087104fca5\") " pod="openshift-ingress/e2e-explicit-038c21-openshift-default-5d5c88569c-4r995" Nov 03 02:44:44.961258 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:44.961131 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7c23f7b-4a52-4a24-8fce-ba087104fca5-credential-socket\") pod \"e2e-explicit-038c21-openshift-default-5d5c88569c-4r995\" (UID: \"f7c23f7b-4a52-4a24-8fce-ba087104fca5\") " pod="openshift-ingress/e2e-explicit-038c21-openshift-default-5d5c88569c-4r995" Nov 03 02:44:44.961417 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:44.961396 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7c23f7b-4a52-4a24-8fce-ba087104fca5-istio-data\") pod \"e2e-explicit-038c21-openshift-default-5d5c88569c-4r995\" (UID: \"f7c23f7b-4a52-4a24-8fce-ba087104fca5\") " pod="openshift-ingress/e2e-explicit-038c21-openshift-default-5d5c88569c-4r995" Nov 03 02:44:44.961497 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:44.961443 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7c23f7b-4a52-4a24-8fce-ba087104fca5-workload-certs\") pod \"e2e-explicit-038c21-openshift-default-5d5c88569c-4r995\" (UID: \"f7c23f7b-4a52-4a24-8fce-ba087104fca5\") " pod="openshift-ingress/e2e-explicit-038c21-openshift-default-5d5c88569c-4r995" Nov 03 02:44:44.961555 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:44.961224 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7c23f7b-4a52-4a24-8fce-ba087104fca5-workload-socket\") pod \"e2e-explicit-038c21-openshift-default-5d5c88569c-4r995\" (UID: \"f7c23f7b-4a52-4a24-8fce-ba087104fca5\") " pod="openshift-ingress/e2e-explicit-038c21-openshift-default-5d5c88569c-4r995" Nov 03 02:44:44.961555 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:44.961541 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7c23f7b-4a52-4a24-8fce-ba087104fca5-istio-token\") pod \"e2e-explicit-038c21-openshift-default-5d5c88569c-4r995\" (UID: \"f7c23f7b-4a52-4a24-8fce-ba087104fca5\") " pod="openshift-ingress/e2e-explicit-038c21-openshift-default-5d5c88569c-4r995" Nov 03 02:44:44.961668 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:44.961634 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7c23f7b-4a52-4a24-8fce-ba087104fca5-credential-socket\") pod \"e2e-explicit-038c21-openshift-default-5d5c88569c-4r995\" (UID: \"f7c23f7b-4a52-4a24-8fce-ba087104fca5\") " pod="openshift-ingress/e2e-explicit-038c21-openshift-default-5d5c88569c-4r995" Nov 03 02:44:44.961765 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:44.961742 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7c23f7b-4a52-4a24-8fce-ba087104fca5-workload-socket\") pod \"e2e-explicit-038c21-openshift-default-5d5c88569c-4r995\" (UID: \"f7c23f7b-4a52-4a24-8fce-ba087104fca5\") " pod="openshift-ingress/e2e-explicit-038c21-openshift-default-5d5c88569c-4r995" Nov 03 02:44:44.961838 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:44.961820 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7c23f7b-4a52-4a24-8fce-ba087104fca5-istiod-ca-cert\") pod \"e2e-explicit-038c21-openshift-default-5d5c88569c-4r995\" (UID: \"f7c23f7b-4a52-4a24-8fce-ba087104fca5\") " pod="openshift-ingress/e2e-explicit-038c21-openshift-default-5d5c88569c-4r995" Nov 03 02:44:44.963732 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:44.963709 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7c23f7b-4a52-4a24-8fce-ba087104fca5-istio-podinfo\") pod \"e2e-explicit-038c21-openshift-default-5d5c88569c-4r995\" (UID: \"f7c23f7b-4a52-4a24-8fce-ba087104fca5\") " pod="openshift-ingress/e2e-explicit-038c21-openshift-default-5d5c88569c-4r995" Nov 03 02:44:44.963848 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:44.963739 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7c23f7b-4a52-4a24-8fce-ba087104fca5-istio-envoy\") pod \"e2e-explicit-038c21-openshift-default-5d5c88569c-4r995\" (UID: \"f7c23f7b-4a52-4a24-8fce-ba087104fca5\") " pod="openshift-ingress/e2e-explicit-038c21-openshift-default-5d5c88569c-4r995" Nov 03 02:44:45.020675 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:45.020456 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djn5v\" (UniqueName: \"kubernetes.io/projected/f7c23f7b-4a52-4a24-8fce-ba087104fca5-kube-api-access-djn5v\") pod \"e2e-explicit-038c21-openshift-default-5d5c88569c-4r995\" (UID: \"f7c23f7b-4a52-4a24-8fce-ba087104fca5\") " pod="openshift-ingress/e2e-explicit-038c21-openshift-default-5d5c88569c-4r995" Nov 03 02:44:45.025983 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:45.025953 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7c23f7b-4a52-4a24-8fce-ba087104fca5-istio-token\") pod \"e2e-explicit-038c21-openshift-default-5d5c88569c-4r995\" (UID: \"f7c23f7b-4a52-4a24-8fce-ba087104fca5\") " pod="openshift-ingress/e2e-explicit-038c21-openshift-default-5d5c88569c-4r995" Nov 03 02:44:45.151156 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:45.151079 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-038c21-openshift-default-5d5c88569c-4r995" Nov 03 02:44:45.573140 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:45.573106 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-038c21-openshift-default-5d5c88569c-4r995"] Nov 03 02:44:45.579013 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:44:45.578983 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7c23f7b_4a52_4a24_8fce_ba087104fca5.slice/crio-7c71323da9150db2af796be58db56f0b9a80c234f81b37f761a474201e2d9beb WatchSource:0}: Error finding container 7c71323da9150db2af796be58db56f0b9a80c234f81b37f761a474201e2d9beb: Status 404 returned error can't find the container with id 7c71323da9150db2af796be58db56f0b9a80c234f81b37f761a474201e2d9beb Nov 03 02:44:45.581597 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:45.581559 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:44:45.581694 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:45.581629 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:44:45.581694 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:45.581658 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:44:45.612868 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:45.612841 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-038c21-openshift-default-5d5c88569c-4r995" event={"ID":"f7c23f7b-4a52-4a24-8fce-ba087104fca5","Type":"ContainerStarted","Data":"7c71323da9150db2af796be58db56f0b9a80c234f81b37f761a474201e2d9beb"} Nov 03 02:44:46.620133 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:46.620100 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-038c21-openshift-default-5d5c88569c-4r995" event={"ID":"f7c23f7b-4a52-4a24-8fce-ba087104fca5","Type":"ContainerStarted","Data":"ea5fea9d3bd0b499ada713f2daec3443f2cf147eb3aabdc778cc85c2e515ae44"} Nov 03 02:44:46.822130 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:46.822086 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-038c21-openshift-default-5d5c88569c-4r995" podStartSLOduration=2.8220741350000003 podStartE2EDuration="2.822074135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:44:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:44:46.775452046 +0000 UTC m=+3164.034650199" watchObservedRunningTime="2026-11-03 02:44:46.822074135 +0000 UTC m=+3164.081272280" Nov 03 02:44:47.151846 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:47.151815 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-038c21-openshift-default-5d5c88569c-4r995" Nov 03 02:44:47.157534 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:47.157509 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-038c21-openshift-default-5d5c88569c-4r995" Nov 03 02:44:47.626284 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:47.626250 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-038c21-openshift-default-5d5c88569c-4r995" Nov 03 02:44:47.627077 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:47.627055 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-038c21-openshift-default-5d5c88569c-4r995" Nov 03 02:44:49.856130 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:49.856100 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6408ce5d-t5sb8"] Nov 03 02:44:49.859859 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:49.859837 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6408ce5d-t5sb8" Nov 03 02:44:49.914949 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:49.914925 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6408ce5d-t5sb8"] Nov 03 02:44:49.916325 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:49.916299 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6472c30-5f73-499e-a92f-a3950c2c40f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6408ce5d-t5sb8\" (UID: \"a6472c30-5f73-499e-a92f-a3950c2c40f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6408ce5d-t5sb8" Nov 03 02:44:49.916500 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:49.916481 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggtl9\" (UniqueName: \"kubernetes.io/projected/a6472c30-5f73-499e-a92f-a3950c2c40f0-kube-api-access-ggtl9\") pod \"maas-api-revoke-sub-6408ce5d-t5sb8\" (UID: \"a6472c30-5f73-499e-a92f-a3950c2c40f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6408ce5d-t5sb8" Nov 03 02:44:49.916574 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:49.916555 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6472c30-5f73-499e-a92f-a3950c2c40f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6408ce5d-t5sb8\" (UID: \"a6472c30-5f73-499e-a92f-a3950c2c40f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6408ce5d-t5sb8" Nov 03 02:44:49.958284 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:49.958258 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:44:49.958399 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:49.958295 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:44:49.958399 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:49.958310 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:44:49.958399 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:49.958348 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:44:50.018122 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:50.018099 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6472c30-5f73-499e-a92f-a3950c2c40f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6408ce5d-t5sb8\" (UID: \"a6472c30-5f73-499e-a92f-a3950c2c40f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6408ce5d-t5sb8" Nov 03 02:44:50.018228 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:50.018187 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6472c30-5f73-499e-a92f-a3950c2c40f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6408ce5d-t5sb8\" (UID: \"a6472c30-5f73-499e-a92f-a3950c2c40f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6408ce5d-t5sb8" Nov 03 02:44:50.019253 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:50.018307 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggtl9\" (UniqueName: \"kubernetes.io/projected/a6472c30-5f73-499e-a92f-a3950c2c40f0-kube-api-access-ggtl9\") pod \"maas-api-revoke-sub-6408ce5d-t5sb8\" (UID: \"a6472c30-5f73-499e-a92f-a3950c2c40f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6408ce5d-t5sb8" Nov 03 02:44:50.021939 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:50.019563 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6472c30-5f73-499e-a92f-a3950c2c40f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6408ce5d-t5sb8\" (UID: \"a6472c30-5f73-499e-a92f-a3950c2c40f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6408ce5d-t5sb8" Nov 03 02:44:50.028194 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:50.028173 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6472c30-5f73-499e-a92f-a3950c2c40f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6408ce5d-t5sb8\" (UID: \"a6472c30-5f73-499e-a92f-a3950c2c40f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6408ce5d-t5sb8" Nov 03 02:44:50.071954 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:50.071933 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggtl9\" (UniqueName: \"kubernetes.io/projected/a6472c30-5f73-499e-a92f-a3950c2c40f0-kube-api-access-ggtl9\") pod \"maas-api-revoke-sub-6408ce5d-t5sb8\" (UID: \"a6472c30-5f73-499e-a92f-a3950c2c40f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6408ce5d-t5sb8" Nov 03 02:44:50.173568 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:50.173513 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6408ce5d-t5sb8" Nov 03 02:44:50.539408 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:50.539323 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6408ce5d-t5sb8"] Nov 03 02:44:50.544543 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:44:50.544513 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6472c30_5f73_499e_a92f_a3950c2c40f0.slice/crio-062eddde9484af6daac1d405d0a03ef2151667b71a9771124e2ce2599ad5533c WatchSource:0}: Error finding container 062eddde9484af6daac1d405d0a03ef2151667b71a9771124e2ce2599ad5533c: Status 404 returned error can't find the container with id 062eddde9484af6daac1d405d0a03ef2151667b71a9771124e2ce2599ad5533c Nov 03 02:44:50.568505 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:50.568481 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79f36937-fgnzb"] Nov 03 02:44:50.573377 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:50.573336 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f36937-fgnzb" Nov 03 02:44:50.628069 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:50.628043 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sbzd\" (UniqueName: \"kubernetes.io/projected/d1a4aff6-eb0a-4234-9642-e5bbe77e3102-kube-api-access-2sbzd\") pod \"maas-api-revoke-sub-79f36937-fgnzb\" (UID: \"d1a4aff6-eb0a-4234-9642-e5bbe77e3102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f36937-fgnzb" Nov 03 02:44:50.628169 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:50.628112 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1a4aff6-eb0a-4234-9642-e5bbe77e3102-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79f36937-fgnzb\" (UID: \"d1a4aff6-eb0a-4234-9642-e5bbe77e3102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f36937-fgnzb" Nov 03 02:44:50.628212 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:50.628179 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1a4aff6-eb0a-4234-9642-e5bbe77e3102-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79f36937-fgnzb\" (UID: \"d1a4aff6-eb0a-4234-9642-e5bbe77e3102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f36937-fgnzb" Nov 03 02:44:50.644167 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:50.644066 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79f36937-fgnzb"] Nov 03 02:44:50.645539 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:50.645512 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6408ce5d-t5sb8" event={"ID":"a6472c30-5f73-499e-a92f-a3950c2c40f0","Type":"ContainerStarted","Data":"062eddde9484af6daac1d405d0a03ef2151667b71a9771124e2ce2599ad5533c"} Nov 03 02:44:50.729792 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:50.729766 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1a4aff6-eb0a-4234-9642-e5bbe77e3102-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79f36937-fgnzb\" (UID: \"d1a4aff6-eb0a-4234-9642-e5bbe77e3102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f36937-fgnzb" Nov 03 02:44:50.729895 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:50.729800 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1a4aff6-eb0a-4234-9642-e5bbe77e3102-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79f36937-fgnzb\" (UID: \"d1a4aff6-eb0a-4234-9642-e5bbe77e3102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f36937-fgnzb" Nov 03 02:44:50.729947 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:50.729898 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sbzd\" (UniqueName: \"kubernetes.io/projected/d1a4aff6-eb0a-4234-9642-e5bbe77e3102-kube-api-access-2sbzd\") pod \"maas-api-revoke-sub-79f36937-fgnzb\" (UID: \"d1a4aff6-eb0a-4234-9642-e5bbe77e3102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f36937-fgnzb" Nov 03 02:44:50.730569 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:50.730551 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1a4aff6-eb0a-4234-9642-e5bbe77e3102-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79f36937-fgnzb\" (UID: \"d1a4aff6-eb0a-4234-9642-e5bbe77e3102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f36937-fgnzb" Nov 03 02:44:50.732063 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:50.732045 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1a4aff6-eb0a-4234-9642-e5bbe77e3102-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79f36937-fgnzb\" (UID: \"d1a4aff6-eb0a-4234-9642-e5bbe77e3102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f36937-fgnzb" Nov 03 02:44:50.855041 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:50.855019 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sbzd\" (UniqueName: \"kubernetes.io/projected/d1a4aff6-eb0a-4234-9642-e5bbe77e3102-kube-api-access-2sbzd\") pod \"maas-api-revoke-sub-79f36937-fgnzb\" (UID: \"d1a4aff6-eb0a-4234-9642-e5bbe77e3102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f36937-fgnzb" Nov 03 02:44:50.885127 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:50.885109 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f36937-fgnzb" Nov 03 02:44:51.254679 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:51.254587 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79f36937-fgnzb"] Nov 03 02:44:51.260532 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:44:51.260504 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1a4aff6_eb0a_4234_9642_e5bbe77e3102.slice/crio-6facc4b7d85f916417ec3b7158e103ba400ce6ab13a987f272ce34b2e6779f2d WatchSource:0}: Error finding container 6facc4b7d85f916417ec3b7158e103ba400ce6ab13a987f272ce34b2e6779f2d: Status 404 returned error can't find the container with id 6facc4b7d85f916417ec3b7158e103ba400ce6ab13a987f272ce34b2e6779f2d Nov 03 02:44:51.652414 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:51.652378 5605 generic.go:356] "Generic (PLEG): container finished" podID="a6472c30-5f73-499e-a92f-a3950c2c40f0" containerID="b87d9f4ef1f14a1a59f243ef9bc86a9a2238345d84d9fa41e0c1ede853bb2aab" exitCode=0 Nov 03 02:44:51.652584 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:51.652448 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6408ce5d-t5sb8" event={"ID":"a6472c30-5f73-499e-a92f-a3950c2c40f0","Type":"ContainerDied","Data":"b87d9f4ef1f14a1a59f243ef9bc86a9a2238345d84d9fa41e0c1ede853bb2aab"} Nov 03 02:44:51.653950 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:51.653931 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f36937-fgnzb" event={"ID":"d1a4aff6-eb0a-4234-9642-e5bbe77e3102","Type":"ContainerStarted","Data":"a6fd954fae62f72abacbba2baf5fe64325520528c8dc2ca55dadeb6c1d3585e5"} Nov 03 02:44:51.654054 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:51.653964 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f36937-fgnzb" event={"ID":"d1a4aff6-eb0a-4234-9642-e5bbe77e3102","Type":"ContainerStarted","Data":"6facc4b7d85f916417ec3b7158e103ba400ce6ab13a987f272ce34b2e6779f2d"} Nov 03 02:44:51.863563 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:51.863451 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f36937-fgnzb" podStartSLOduration=1.863439757 podStartE2EDuration="1.863439757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:44:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:44:51.862437415 +0000 UTC m=+3169.121635581" watchObservedRunningTime="2026-11-03 02:44:51.863439757 +0000 UTC m=+3169.122637941" Nov 03 02:44:52.661231 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:52.661193 5605 generic.go:356] "Generic (PLEG): container finished" podID="d1a4aff6-eb0a-4234-9642-e5bbe77e3102" containerID="a6fd954fae62f72abacbba2baf5fe64325520528c8dc2ca55dadeb6c1d3585e5" exitCode=0 Nov 03 02:44:52.661630 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:52.661274 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f36937-fgnzb" event={"ID":"d1a4aff6-eb0a-4234-9642-e5bbe77e3102","Type":"ContainerDied","Data":"a6fd954fae62f72abacbba2baf5fe64325520528c8dc2ca55dadeb6c1d3585e5"} Nov 03 02:44:52.833676 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:52.833647 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6408ce5d-t5sb8" Nov 03 02:44:52.952748 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:52.952664 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6472c30-5f73-499e-a92f-a3950c2c40f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6472c30-5f73-499e-a92f-a3950c2c40f0-maas-api-xks-ca\") pod \"a6472c30-5f73-499e-a92f-a3950c2c40f0\" (UID: \"a6472c30-5f73-499e-a92f-a3950c2c40f0\") " Nov 03 02:44:52.952902 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:52.952749 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6472c30-5f73-499e-a92f-a3950c2c40f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6472c30-5f73-499e-a92f-a3950c2c40f0-maas-api-openshift-ca\") pod \"a6472c30-5f73-499e-a92f-a3950c2c40f0\" (UID: \"a6472c30-5f73-499e-a92f-a3950c2c40f0\") " Nov 03 02:44:52.952902 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:52.952826 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6472c30-5f73-499e-a92f-a3950c2c40f0-kube-api-access-ggtl9\" (UniqueName: \"kubernetes.io/projected/a6472c30-5f73-499e-a92f-a3950c2c40f0-kube-api-access-ggtl9\") pod \"a6472c30-5f73-499e-a92f-a3950c2c40f0\" (UID: \"a6472c30-5f73-499e-a92f-a3950c2c40f0\") " Nov 03 02:44:52.953066 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:52.953041 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6472c30-5f73-499e-a92f-a3950c2c40f0-maas-api-openshift-ca" pod "a6472c30-5f73-499e-a92f-a3950c2c40f0" (UID: "a6472c30-5f73-499e-a92f-a3950c2c40f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:44:52.953296 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:52.953279 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6472c30-5f73-499e-a92f-a3950c2c40f0-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:44:52.954922 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:52.954898 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6472c30-5f73-499e-a92f-a3950c2c40f0-maas-api-xks-ca" pod "a6472c30-5f73-499e-a92f-a3950c2c40f0" (UID: "a6472c30-5f73-499e-a92f-a3950c2c40f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:44:52.955380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:52.955317 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6472c30-5f73-499e-a92f-a3950c2c40f0-kube-api-access-ggtl9" pod "a6472c30-5f73-499e-a92f-a3950c2c40f0" (UID: "a6472c30-5f73-499e-a92f-a3950c2c40f0"). InnerVolumeSpecName "kube-api-access-ggtl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:44:53.054426 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:53.054392 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggtl9\" (UniqueName: \"kubernetes.io/projected/a6472c30-5f73-499e-a92f-a3950c2c40f0-kube-api-access-ggtl9\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:44:53.054426 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:53.054417 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6472c30-5f73-499e-a92f-a3950c2c40f0-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:44:53.433377 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:53.433033 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-038c21-c54876867-hxprn"] Nov 03 02:44:53.436141 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:53.433734 5605 cpu_manager.go:414] "containerMap: removing container" podUID="a6472c30-5f73-499e-a92f-a3950c2c40f0" containerName="revoke-keys" Nov 03 02:44:53.436141 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:53.433758 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6472c30-5f73-499e-a92f-a3950c2c40f0" containerName="revoke-keys" Nov 03 02:44:53.436141 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:53.433882 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6472c30-5f73-499e-a92f-a3950c2c40f0" containerName="revoke-keys" Nov 03 02:44:53.441392 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:53.437460 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-038c21-c54876867-hxprn" Nov 03 02:44:53.489972 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:53.489941 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-038c21-c54876867-hxprn"] Nov 03 02:44:53.533505 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:53.533482 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-038c21\"" Nov 03 02:44:53.533505 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:53.533489 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-038c21-dockercfg-5tfvq\"" Nov 03 02:44:53.560201 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:53.560178 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d63996c8-1493-441a-927c-09b9f7394f12-plugins-config-volume\") pod \"payload-processing-e2e-explicit-038c21-c54876867-hxprn\" (UID: \"d63996c8-1493-441a-927c-09b9f7394f12\") " pod="openshift-ingress/payload-processing-e2e-explicit-038c21-c54876867-hxprn" Nov 03 02:44:53.560343 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:53.560325 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l48rw\" (UniqueName: \"kubernetes.io/projected/d63996c8-1493-441a-927c-09b9f7394f12-kube-api-access-l48rw\") pod \"payload-processing-e2e-explicit-038c21-c54876867-hxprn\" (UID: \"d63996c8-1493-441a-927c-09b9f7394f12\") " pod="openshift-ingress/payload-processing-e2e-explicit-038c21-c54876867-hxprn" Nov 03 02:44:53.627039 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:53.627013 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-038c21-5d649bd5f-tcdq5"] Nov 03 02:44:53.631765 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:53.631740 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-038c21-5d649bd5f-tcdq5" Nov 03 02:44:53.661139 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:53.661115 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d63996c8-1493-441a-927c-09b9f7394f12-plugins-config-volume\") pod \"payload-processing-e2e-explicit-038c21-c54876867-hxprn\" (UID: \"d63996c8-1493-441a-927c-09b9f7394f12\") " pod="openshift-ingress/payload-processing-e2e-explicit-038c21-c54876867-hxprn" Nov 03 02:44:53.661276 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:53.661258 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l48rw\" (UniqueName: \"kubernetes.io/projected/d63996c8-1493-441a-927c-09b9f7394f12-kube-api-access-l48rw\") pod \"payload-processing-e2e-explicit-038c21-c54876867-hxprn\" (UID: \"d63996c8-1493-441a-927c-09b9f7394f12\") " pod="openshift-ingress/payload-processing-e2e-explicit-038c21-c54876867-hxprn" Nov 03 02:44:53.661852 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:53.661835 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d63996c8-1493-441a-927c-09b9f7394f12-plugins-config-volume\") pod \"payload-processing-e2e-explicit-038c21-c54876867-hxprn\" (UID: \"d63996c8-1493-441a-927c-09b9f7394f12\") " pod="openshift-ingress/payload-processing-e2e-explicit-038c21-c54876867-hxprn" Nov 03 02:44:53.667618 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:53.667598 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6408ce5d-t5sb8" Nov 03 02:44:53.667706 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:53.667628 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6408ce5d-t5sb8" event={"ID":"a6472c30-5f73-499e-a92f-a3950c2c40f0","Type":"ContainerDied","Data":"062eddde9484af6daac1d405d0a03ef2151667b71a9771124e2ce2599ad5533c"} Nov 03 02:44:53.667706 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:53.667687 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="062eddde9484af6daac1d405d0a03ef2151667b71a9771124e2ce2599ad5533c" Nov 03 02:44:53.685521 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:53.685465 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-038c21-5d649bd5f-tcdq5"] Nov 03 02:44:53.715131 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:53.715099 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l48rw\" (UniqueName: \"kubernetes.io/projected/d63996c8-1493-441a-927c-09b9f7394f12-kube-api-access-l48rw\") pod \"payload-processing-e2e-explicit-038c21-c54876867-hxprn\" (UID: \"d63996c8-1493-441a-927c-09b9f7394f12\") " pod="openshift-ingress/payload-processing-e2e-explicit-038c21-c54876867-hxprn" Nov 03 02:44:53.757317 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:53.757291 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-038c21-c54876867-hxprn" Nov 03 02:44:53.762128 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:53.762099 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5s6x\" (UniqueName: \"kubernetes.io/projected/9b5077e6-2db7-45f4-85e4-6fca6f27b117-kube-api-access-r5s6x\") pod \"payload-pre-processing-e2e-explicit-038c21-5d649bd5f-tcdq5\" (UID: \"9b5077e6-2db7-45f4-85e4-6fca6f27b117\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-038c21-5d649bd5f-tcdq5" Nov 03 02:44:53.762276 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:53.762232 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b5077e6-2db7-45f4-85e4-6fca6f27b117-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-038c21-5d649bd5f-tcdq5\" (UID: \"9b5077e6-2db7-45f4-85e4-6fca6f27b117\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-038c21-5d649bd5f-tcdq5" Nov 03 02:44:53.847905 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:53.847877 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f36937-fgnzb" Nov 03 02:44:53.863972 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:53.863947 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5s6x\" (UniqueName: \"kubernetes.io/projected/9b5077e6-2db7-45f4-85e4-6fca6f27b117-kube-api-access-r5s6x\") pod \"payload-pre-processing-e2e-explicit-038c21-5d649bd5f-tcdq5\" (UID: \"9b5077e6-2db7-45f4-85e4-6fca6f27b117\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-038c21-5d649bd5f-tcdq5" Nov 03 02:44:53.864125 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:53.864064 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b5077e6-2db7-45f4-85e4-6fca6f27b117-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-038c21-5d649bd5f-tcdq5\" (UID: \"9b5077e6-2db7-45f4-85e4-6fca6f27b117\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-038c21-5d649bd5f-tcdq5" Nov 03 02:44:53.865001 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:53.864979 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b5077e6-2db7-45f4-85e4-6fca6f27b117-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-038c21-5d649bd5f-tcdq5\" (UID: \"9b5077e6-2db7-45f4-85e4-6fca6f27b117\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-038c21-5d649bd5f-tcdq5" Nov 03 02:44:53.914081 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:44:53.914047 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:44:53.917985 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:53.917956 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5s6x\" (UniqueName: \"kubernetes.io/projected/9b5077e6-2db7-45f4-85e4-6fca6f27b117-kube-api-access-r5s6x\") pod \"payload-pre-processing-e2e-explicit-038c21-5d649bd5f-tcdq5\" (UID: \"9b5077e6-2db7-45f4-85e4-6fca6f27b117\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-038c21-5d649bd5f-tcdq5" Nov 03 02:44:53.947124 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:53.947061 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-038c21-5d649bd5f-tcdq5" Nov 03 02:44:53.965289 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:53.965258 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1a4aff6-eb0a-4234-9642-e5bbe77e3102-kube-api-access-2sbzd\" (UniqueName: \"kubernetes.io/projected/d1a4aff6-eb0a-4234-9642-e5bbe77e3102-kube-api-access-2sbzd\") pod \"d1a4aff6-eb0a-4234-9642-e5bbe77e3102\" (UID: \"d1a4aff6-eb0a-4234-9642-e5bbe77e3102\") " Nov 03 02:44:53.965430 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:53.965404 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1a4aff6-eb0a-4234-9642-e5bbe77e3102-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1a4aff6-eb0a-4234-9642-e5bbe77e3102-maas-api-openshift-ca\") pod \"d1a4aff6-eb0a-4234-9642-e5bbe77e3102\" (UID: \"d1a4aff6-eb0a-4234-9642-e5bbe77e3102\") " Nov 03 02:44:53.965501 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:53.965461 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1a4aff6-eb0a-4234-9642-e5bbe77e3102-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1a4aff6-eb0a-4234-9642-e5bbe77e3102-maas-api-xks-ca\") pod \"d1a4aff6-eb0a-4234-9642-e5bbe77e3102\" (UID: \"d1a4aff6-eb0a-4234-9642-e5bbe77e3102\") " Nov 03 02:44:53.966724 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:53.966695 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1a4aff6-eb0a-4234-9642-e5bbe77e3102-maas-api-openshift-ca" pod "d1a4aff6-eb0a-4234-9642-e5bbe77e3102" (UID: "d1a4aff6-eb0a-4234-9642-e5bbe77e3102"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:44:53.968390 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:53.968339 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1a4aff6-eb0a-4234-9642-e5bbe77e3102-maas-api-xks-ca" pod "d1a4aff6-eb0a-4234-9642-e5bbe77e3102" (UID: "d1a4aff6-eb0a-4234-9642-e5bbe77e3102"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:44:53.968604 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:53.968575 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1a4aff6-eb0a-4234-9642-e5bbe77e3102-kube-api-access-2sbzd" pod "d1a4aff6-eb0a-4234-9642-e5bbe77e3102" (UID: "d1a4aff6-eb0a-4234-9642-e5bbe77e3102"). InnerVolumeSpecName "kube-api-access-2sbzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:44:54.066783 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:54.066750 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sbzd\" (UniqueName: \"kubernetes.io/projected/d1a4aff6-eb0a-4234-9642-e5bbe77e3102-kube-api-access-2sbzd\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:44:54.066783 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:54.066781 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1a4aff6-eb0a-4234-9642-e5bbe77e3102-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:44:54.066957 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:54.066793 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1a4aff6-eb0a-4234-9642-e5bbe77e3102-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:44:54.130405 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:54.130375 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-038c21-c54876867-hxprn"] Nov 03 02:44:54.135341 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:44:54.135316 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd63996c8_1493_441a_927c_09b9f7394f12.slice/crio-eb1faff8fd540331ec443aab7cd91938009b00ce614486f66382c51b88960641 WatchSource:0}: Error finding container eb1faff8fd540331ec443aab7cd91938009b00ce614486f66382c51b88960641: Status 404 returned error can't find the container with id eb1faff8fd540331ec443aab7cd91938009b00ce614486f66382c51b88960641 Nov 03 02:44:54.318953 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:54.318917 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-038c21-5d649bd5f-tcdq5"] Nov 03 02:44:54.324124 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:44:54.324098 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b5077e6_2db7_45f4_85e4_6fca6f27b117.slice/crio-84a272a259f7c3aa57cd33d99dd462ee13bc300e38eb60ce37c900e8bbc99822 WatchSource:0}: Error finding container 84a272a259f7c3aa57cd33d99dd462ee13bc300e38eb60ce37c900e8bbc99822: Status 404 returned error can't find the container with id 84a272a259f7c3aa57cd33d99dd462ee13bc300e38eb60ce37c900e8bbc99822 Nov 03 02:44:54.675967 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:54.675883 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-038c21-5d649bd5f-tcdq5" event={"ID":"9b5077e6-2db7-45f4-85e4-6fca6f27b117","Type":"ContainerStarted","Data":"9cd99fcf0b614a01120461f7025e7a07f81a7a46986b65e94ad3859d42d347d6"} Nov 03 02:44:54.675967 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:54.675935 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-038c21-5d649bd5f-tcdq5" event={"ID":"9b5077e6-2db7-45f4-85e4-6fca6f27b117","Type":"ContainerStarted","Data":"84a272a259f7c3aa57cd33d99dd462ee13bc300e38eb60ce37c900e8bbc99822"} Nov 03 02:44:54.676470 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:54.676030 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-038c21-5d649bd5f-tcdq5" Nov 03 02:44:54.677410 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:54.677389 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f36937-fgnzb" Nov 03 02:44:54.677540 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:54.677389 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f36937-fgnzb" event={"ID":"d1a4aff6-eb0a-4234-9642-e5bbe77e3102","Type":"ContainerDied","Data":"6facc4b7d85f916417ec3b7158e103ba400ce6ab13a987f272ce34b2e6779f2d"} Nov 03 02:44:54.677540 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:54.677522 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6facc4b7d85f916417ec3b7158e103ba400ce6ab13a987f272ce34b2e6779f2d" Nov 03 02:44:54.678960 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:54.678940 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-038c21-c54876867-hxprn" event={"ID":"d63996c8-1493-441a-927c-09b9f7394f12","Type":"ContainerStarted","Data":"a1688a5c0a6b6349d11020da63822bc309e670752119a52058ee5062739a6a47"} Nov 03 02:44:54.679058 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:54.678973 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-038c21-c54876867-hxprn" event={"ID":"d63996c8-1493-441a-927c-09b9f7394f12","Type":"ContainerStarted","Data":"eb1faff8fd540331ec443aab7cd91938009b00ce614486f66382c51b88960641"} Nov 03 02:44:54.679120 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:54.679084 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-038c21-c54876867-hxprn" Nov 03 02:44:54.783094 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:54.783050 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-038c21-5d649bd5f-tcdq5" podStartSLOduration=1.783038392 podStartE2EDuration="1.783038392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:44:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:44:54.782226405 +0000 UTC m=+3172.041424560" watchObservedRunningTime="2026-11-03 02:44:54.783038392 +0000 UTC m=+3172.042236545" Nov 03 02:44:54.888586 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:44:54.888546 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-038c21-c54876867-hxprn" podStartSLOduration=1.888534749 podStartE2EDuration="1.888534749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:44:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:44:54.88741325 +0000 UTC m=+3172.146611404" watchObservedRunningTime="2026-11-03 02:44:54.888534749 +0000 UTC m=+3172.147732903" Nov 03 02:45:00.154338 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.154305 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894565-g2zcb"] Nov 03 02:45:00.154796 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.154782 5605 cpu_manager.go:414] "containerMap: removing container" podUID="d1a4aff6-eb0a-4234-9642-e5bbe77e3102" containerName="revoke-keys" Nov 03 02:45:00.154843 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.154796 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1a4aff6-eb0a-4234-9642-e5bbe77e3102" containerName="revoke-keys" Nov 03 02:45:00.154896 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.154887 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1a4aff6-eb0a-4234-9642-e5bbe77e3102" containerName="revoke-keys" Nov 03 02:45:00.158436 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.158417 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894565-66m7h"] Nov 03 02:45:00.158560 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.158540 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894565-g2zcb" Nov 03 02:45:00.162274 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.162254 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894565-lvznm"] Nov 03 02:45:00.162397 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.162379 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894565-66m7h" Nov 03 02:45:00.165877 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.165857 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894565-lvznm" Nov 03 02:45:00.200975 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.200953 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894565-kdmq9"] Nov 03 02:45:00.210337 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.210313 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:45:00.211852 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.211828 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894565-tqf6b"] Nov 03 02:45:00.212052 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.211925 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894565-kdmq9" Nov 03 02:45:00.217218 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.217199 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894565-fz2lx"] Nov 03 02:45:00.217342 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.217325 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894565-tqf6b" Nov 03 02:45:00.220650 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.220632 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894565-6wfpx"] Nov 03 02:45:00.220760 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.220744 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894565-fz2lx" Nov 03 02:45:00.224091 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.224072 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894565-47qxs"] Nov 03 02:45:00.224193 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.224159 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894565-6wfpx" Nov 03 02:45:00.227637 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.227616 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894565-47qxs" Nov 03 02:45:00.234548 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.234526 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe40cb78-4bfe-4975-acbd-867c3366f3ae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894565-66m7h\" (UID: \"fe40cb78-4bfe-4975-acbd-867c3366f3ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894565-66m7h" Nov 03 02:45:00.234634 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.234567 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn2lq\" (UniqueName: \"kubernetes.io/projected/fe40cb78-4bfe-4975-acbd-867c3366f3ae-kube-api-access-dn2lq\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894565-66m7h\" (UID: \"fe40cb78-4bfe-4975-acbd-867c3366f3ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894565-66m7h" Nov 03 02:45:00.234634 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.234590 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb82f\" (UniqueName: \"kubernetes.io/projected/aeb1d4c3-d1f5-49f4-9e3c-47ff2b640ac9-kube-api-access-sb82f\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894565-g2zcb\" (UID: \"aeb1d4c3-d1f5-49f4-9e3c-47ff2b640ac9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894565-g2zcb" Nov 03 02:45:00.234725 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.234706 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt8sh\" (UniqueName: \"kubernetes.io/projected/ed96e13a-3942-46e6-846f-027f5d1718cc-kube-api-access-tt8sh\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894565-lvznm\" (UID: \"ed96e13a-3942-46e6-846f-027f5d1718cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894565-lvznm" Nov 03 02:45:00.234758 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.234743 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aeb1d4c3-d1f5-49f4-9e3c-47ff2b640ac9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894565-g2zcb\" (UID: \"aeb1d4c3-d1f5-49f4-9e3c-47ff2b640ac9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894565-g2zcb" Nov 03 02:45:00.234892 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.234876 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed96e13a-3942-46e6-846f-027f5d1718cc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894565-lvznm\" (UID: \"ed96e13a-3942-46e6-846f-027f5d1718cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894565-lvznm" Nov 03 02:45:00.245515 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.245496 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:45:00.245515 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.245496 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:45:00.259919 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.259896 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894565-66m7h"] Nov 03 02:45:00.261965 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.261950 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894565-g2zcb"] Nov 03 02:45:00.264338 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.264320 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894565-lvznm"] Nov 03 02:45:00.297787 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.297764 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894565-kdmq9"] Nov 03 02:45:00.303661 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.303637 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894565-fz2lx"] Nov 03 02:45:00.336318 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.336291 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e94bb7db-1559-4c16-857f-3d20e0a7a008-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894565-kdmq9\" (UID: \"e94bb7db-1559-4c16-857f-3d20e0a7a008\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894565-kdmq9" Nov 03 02:45:00.336507 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.336384 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe40cb78-4bfe-4975-acbd-867c3366f3ae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894565-66m7h\" (UID: \"fe40cb78-4bfe-4975-acbd-867c3366f3ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894565-66m7h" Nov 03 02:45:00.336507 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.336481 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/660b74cc-c09a-48c2-b447-d869e6f3efa2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894565-6wfpx\" (UID: \"660b74cc-c09a-48c2-b447-d869e6f3efa2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894565-6wfpx" Nov 03 02:45:00.336635 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.336514 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn2lq\" (UniqueName: \"kubernetes.io/projected/fe40cb78-4bfe-4975-acbd-867c3366f3ae-kube-api-access-dn2lq\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894565-66m7h\" (UID: \"fe40cb78-4bfe-4975-acbd-867c3366f3ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894565-66m7h" Nov 03 02:45:00.336635 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.336540 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb82f\" (UniqueName: \"kubernetes.io/projected/aeb1d4c3-d1f5-49f4-9e3c-47ff2b640ac9-kube-api-access-sb82f\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894565-g2zcb\" (UID: \"aeb1d4c3-d1f5-49f4-9e3c-47ff2b640ac9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894565-g2zcb" Nov 03 02:45:00.336635 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.336569 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b8b50df-b64b-4249-864d-841839156ae5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894565-47qxs\" (UID: \"6b8b50df-b64b-4249-864d-841839156ae5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894565-47qxs" Nov 03 02:45:00.336635 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.336598 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt8sh\" (UniqueName: \"kubernetes.io/projected/ed96e13a-3942-46e6-846f-027f5d1718cc-kube-api-access-tt8sh\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894565-lvznm\" (UID: \"ed96e13a-3942-46e6-846f-027f5d1718cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894565-lvznm" Nov 03 02:45:00.336635 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.336630 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aeb1d4c3-d1f5-49f4-9e3c-47ff2b640ac9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894565-g2zcb\" (UID: \"aeb1d4c3-d1f5-49f4-9e3c-47ff2b640ac9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894565-g2zcb" Nov 03 02:45:00.336893 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.336689 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dc6be8c-60cf-4429-9ced-90ca7ed49656-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c7d758-29894565-fz2lx\" (UID: \"0dc6be8c-60cf-4429-9ced-90ca7ed49656\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894565-fz2lx" Nov 03 02:45:00.336893 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.336717 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm586\" (UniqueName: \"kubernetes.io/projected/0dc6be8c-60cf-4429-9ced-90ca7ed49656-kube-api-access-cm586\") pod \"maas-api-key-cleanup-e2e-worker-w4-c7d758-29894565-fz2lx\" (UID: \"0dc6be8c-60cf-4429-9ced-90ca7ed49656\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894565-fz2lx" Nov 03 02:45:00.336893 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.336749 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp4cn\" (UniqueName: \"kubernetes.io/projected/6b8b50df-b64b-4249-864d-841839156ae5-kube-api-access-jp4cn\") pod \"maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894565-47qxs\" (UID: \"6b8b50df-b64b-4249-864d-841839156ae5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894565-47qxs" Nov 03 02:45:00.336893 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.336816 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lmhd\" (UniqueName: \"kubernetes.io/projected/e94bb7db-1559-4c16-857f-3d20e0a7a008-kube-api-access-4lmhd\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894565-kdmq9\" (UID: \"e94bb7db-1559-4c16-857f-3d20e0a7a008\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894565-kdmq9" Nov 03 02:45:00.336893 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.336886 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed96e13a-3942-46e6-846f-027f5d1718cc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894565-lvznm\" (UID: \"ed96e13a-3942-46e6-846f-027f5d1718cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894565-lvznm" Nov 03 02:45:00.337224 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.336925 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57mqg\" (UniqueName: \"kubernetes.io/projected/305b0cb8-2348-4517-86a5-f77bfe6a3814-kube-api-access-57mqg\") pod \"maas-api-key-cleanup-e2e-worker-w2-521f1f-29894565-tqf6b\" (UID: \"305b0cb8-2348-4517-86a5-f77bfe6a3814\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894565-tqf6b" Nov 03 02:45:00.337224 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.336968 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxb9h\" (UniqueName: \"kubernetes.io/projected/660b74cc-c09a-48c2-b447-d869e6f3efa2-kube-api-access-wxb9h\") pod \"maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894565-6wfpx\" (UID: \"660b74cc-c09a-48c2-b447-d869e6f3efa2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894565-6wfpx" Nov 03 02:45:00.337224 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.337002 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/305b0cb8-2348-4517-86a5-f77bfe6a3814-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-521f1f-29894565-tqf6b\" (UID: \"305b0cb8-2348-4517-86a5-f77bfe6a3814\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894565-tqf6b" Nov 03 02:45:00.337747 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.337724 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe40cb78-4bfe-4975-acbd-867c3366f3ae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894565-66m7h\" (UID: \"fe40cb78-4bfe-4975-acbd-867c3366f3ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894565-66m7h" Nov 03 02:45:00.338021 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.337999 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aeb1d4c3-d1f5-49f4-9e3c-47ff2b640ac9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894565-g2zcb\" (UID: \"aeb1d4c3-d1f5-49f4-9e3c-47ff2b640ac9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894565-g2zcb" Nov 03 02:45:00.338888 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.338866 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed96e13a-3942-46e6-846f-027f5d1718cc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894565-lvznm\" (UID: \"ed96e13a-3942-46e6-846f-027f5d1718cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894565-lvznm" Nov 03 02:45:00.345140 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.344409 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894565-6wfpx"] Nov 03 02:45:00.348117 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.347313 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894565-47qxs"] Nov 03 02:45:00.360031 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.360007 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894565-tqf6b"] Nov 03 02:45:00.391053 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.391024 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt8sh\" (UniqueName: \"kubernetes.io/projected/ed96e13a-3942-46e6-846f-027f5d1718cc-kube-api-access-tt8sh\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894565-lvznm\" (UID: \"ed96e13a-3942-46e6-846f-027f5d1718cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894565-lvznm" Nov 03 02:45:00.391182 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.391035 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn2lq\" (UniqueName: \"kubernetes.io/projected/fe40cb78-4bfe-4975-acbd-867c3366f3ae-kube-api-access-dn2lq\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894565-66m7h\" (UID: \"fe40cb78-4bfe-4975-acbd-867c3366f3ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894565-66m7h" Nov 03 02:45:00.391182 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.391170 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb82f\" (UniqueName: \"kubernetes.io/projected/aeb1d4c3-d1f5-49f4-9e3c-47ff2b640ac9-kube-api-access-sb82f\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894565-g2zcb\" (UID: \"aeb1d4c3-d1f5-49f4-9e3c-47ff2b640ac9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894565-g2zcb" Nov 03 02:45:00.438226 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.438157 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b8b50df-b64b-4249-864d-841839156ae5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894565-47qxs\" (UID: \"6b8b50df-b64b-4249-864d-841839156ae5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894565-47qxs" Nov 03 02:45:00.438226 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.438212 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dc6be8c-60cf-4429-9ced-90ca7ed49656-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c7d758-29894565-fz2lx\" (UID: \"0dc6be8c-60cf-4429-9ced-90ca7ed49656\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894565-fz2lx" Nov 03 02:45:00.438415 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.438394 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm586\" (UniqueName: \"kubernetes.io/projected/0dc6be8c-60cf-4429-9ced-90ca7ed49656-kube-api-access-cm586\") pod \"maas-api-key-cleanup-e2e-worker-w4-c7d758-29894565-fz2lx\" (UID: \"0dc6be8c-60cf-4429-9ced-90ca7ed49656\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894565-fz2lx" Nov 03 02:45:00.438476 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.438440 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp4cn\" (UniqueName: \"kubernetes.io/projected/6b8b50df-b64b-4249-864d-841839156ae5-kube-api-access-jp4cn\") pod \"maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894565-47qxs\" (UID: \"6b8b50df-b64b-4249-864d-841839156ae5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894565-47qxs" Nov 03 02:45:00.438531 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.438509 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lmhd\" (UniqueName: \"kubernetes.io/projected/e94bb7db-1559-4c16-857f-3d20e0a7a008-kube-api-access-4lmhd\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894565-kdmq9\" (UID: \"e94bb7db-1559-4c16-857f-3d20e0a7a008\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894565-kdmq9" Nov 03 02:45:00.438623 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.438592 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57mqg\" (UniqueName: \"kubernetes.io/projected/305b0cb8-2348-4517-86a5-f77bfe6a3814-kube-api-access-57mqg\") pod \"maas-api-key-cleanup-e2e-worker-w2-521f1f-29894565-tqf6b\" (UID: \"305b0cb8-2348-4517-86a5-f77bfe6a3814\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894565-tqf6b" Nov 03 02:45:00.438693 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.438643 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxb9h\" (UniqueName: \"kubernetes.io/projected/660b74cc-c09a-48c2-b447-d869e6f3efa2-kube-api-access-wxb9h\") pod \"maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894565-6wfpx\" (UID: \"660b74cc-c09a-48c2-b447-d869e6f3efa2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894565-6wfpx" Nov 03 02:45:00.438693 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.438678 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/305b0cb8-2348-4517-86a5-f77bfe6a3814-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-521f1f-29894565-tqf6b\" (UID: \"305b0cb8-2348-4517-86a5-f77bfe6a3814\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894565-tqf6b" Nov 03 02:45:00.438803 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.438749 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e94bb7db-1559-4c16-857f-3d20e0a7a008-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894565-kdmq9\" (UID: \"e94bb7db-1559-4c16-857f-3d20e0a7a008\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894565-kdmq9" Nov 03 02:45:00.438864 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.438850 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/660b74cc-c09a-48c2-b447-d869e6f3efa2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894565-6wfpx\" (UID: \"660b74cc-c09a-48c2-b447-d869e6f3efa2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894565-6wfpx" Nov 03 02:45:00.438919 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.438893 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b8b50df-b64b-4249-864d-841839156ae5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894565-47qxs\" (UID: \"6b8b50df-b64b-4249-864d-841839156ae5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894565-47qxs" Nov 03 02:45:00.439001 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.438979 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dc6be8c-60cf-4429-9ced-90ca7ed49656-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c7d758-29894565-fz2lx\" (UID: \"0dc6be8c-60cf-4429-9ced-90ca7ed49656\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894565-fz2lx" Nov 03 02:45:00.439464 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.439443 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e94bb7db-1559-4c16-857f-3d20e0a7a008-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894565-kdmq9\" (UID: \"e94bb7db-1559-4c16-857f-3d20e0a7a008\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894565-kdmq9" Nov 03 02:45:00.439574 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.439509 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/305b0cb8-2348-4517-86a5-f77bfe6a3814-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-521f1f-29894565-tqf6b\" (UID: \"305b0cb8-2348-4517-86a5-f77bfe6a3814\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894565-tqf6b" Nov 03 02:45:00.439574 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.439533 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/660b74cc-c09a-48c2-b447-d869e6f3efa2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894565-6wfpx\" (UID: \"660b74cc-c09a-48c2-b447-d869e6f3efa2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894565-6wfpx" Nov 03 02:45:00.473310 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.473279 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894565-g2zcb" Nov 03 02:45:00.480585 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.480563 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894565-66m7h" Nov 03 02:45:00.486901 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.486879 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894565-lvznm" Nov 03 02:45:00.492302 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.492276 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxb9h\" (UniqueName: \"kubernetes.io/projected/660b74cc-c09a-48c2-b447-d869e6f3efa2-kube-api-access-wxb9h\") pod \"maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894565-6wfpx\" (UID: \"660b74cc-c09a-48c2-b447-d869e6f3efa2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894565-6wfpx" Nov 03 02:45:00.492916 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.492892 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lmhd\" (UniqueName: \"kubernetes.io/projected/e94bb7db-1559-4c16-857f-3d20e0a7a008-kube-api-access-4lmhd\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894565-kdmq9\" (UID: \"e94bb7db-1559-4c16-857f-3d20e0a7a008\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894565-kdmq9" Nov 03 02:45:00.493023 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.492962 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57mqg\" (UniqueName: \"kubernetes.io/projected/305b0cb8-2348-4517-86a5-f77bfe6a3814-kube-api-access-57mqg\") pod \"maas-api-key-cleanup-e2e-worker-w2-521f1f-29894565-tqf6b\" (UID: \"305b0cb8-2348-4517-86a5-f77bfe6a3814\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894565-tqf6b" Nov 03 02:45:00.494137 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.494112 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm586\" (UniqueName: \"kubernetes.io/projected/0dc6be8c-60cf-4429-9ced-90ca7ed49656-kube-api-access-cm586\") pod \"maas-api-key-cleanup-e2e-worker-w4-c7d758-29894565-fz2lx\" (UID: \"0dc6be8c-60cf-4429-9ced-90ca7ed49656\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894565-fz2lx" Nov 03 02:45:00.494570 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.494550 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp4cn\" (UniqueName: \"kubernetes.io/projected/6b8b50df-b64b-4249-864d-841839156ae5-kube-api-access-jp4cn\") pod \"maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894565-47qxs\" (UID: \"6b8b50df-b64b-4249-864d-841839156ae5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894565-47qxs" Nov 03 02:45:00.526656 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.526627 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894565-kdmq9" Nov 03 02:45:00.534598 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.534571 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894565-tqf6b" Nov 03 02:45:00.542333 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.542183 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894565-fz2lx" Nov 03 02:45:00.551940 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.551720 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894565-6wfpx" Nov 03 02:45:00.567438 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.567017 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894565-47qxs" Nov 03 02:45:00.861957 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.861926 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894565-g2zcb"] Nov 03 02:45:00.867490 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:45:00.867464 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeb1d4c3_d1f5_49f4_9e3c_47ff2b640ac9.slice/crio-46536676785b1461233e7b99677f9ec73c3487672c841c4fda7667ac85d73a0c WatchSource:0}: Error finding container 46536676785b1461233e7b99677f9ec73c3487672c841c4fda7667ac85d73a0c: Status 404 returned error can't find the container with id 46536676785b1461233e7b99677f9ec73c3487672c841c4fda7667ac85d73a0c Nov 03 02:45:00.913391 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.913347 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894565-lvznm"] Nov 03 02:45:00.917344 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:45:00.917317 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded96e13a_3942_46e6_846f_027f5d1718cc.slice/crio-4eedb02384d1d3e75f8b5cfab773881df7ec995f13bc723163534877dfc47275 WatchSource:0}: Error finding container 4eedb02384d1d3e75f8b5cfab773881df7ec995f13bc723163534877dfc47275: Status 404 returned error can't find the container with id 4eedb02384d1d3e75f8b5cfab773881df7ec995f13bc723163534877dfc47275 Nov 03 02:45:00.977908 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:00.977883 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894565-tqf6b"] Nov 03 02:45:00.982845 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:45:00.982817 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod305b0cb8_2348_4517_86a5_f77bfe6a3814.slice/crio-6ff257238610e0b194e47cdcfc1a3cb56160b0702fb599f4210e807cf8b10853 WatchSource:0}: Error finding container 6ff257238610e0b194e47cdcfc1a3cb56160b0702fb599f4210e807cf8b10853: Status 404 returned error can't find the container with id 6ff257238610e0b194e47cdcfc1a3cb56160b0702fb599f4210e807cf8b10853 Nov 03 02:45:01.013151 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:01.013124 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894565-47qxs"] Nov 03 02:45:01.015910 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:01.015871 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894565-kdmq9"] Nov 03 02:45:01.019031 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:01.019010 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894565-6wfpx"] Nov 03 02:45:01.019888 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:45:01.019866 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod660b74cc_c09a_48c2_b447_d869e6f3efa2.slice/crio-37dac07c2a91a5cef26b0ab3f4503bd3890d4a1296713e589ae6f38ff1329846 WatchSource:0}: Error finding container 37dac07c2a91a5cef26b0ab3f4503bd3890d4a1296713e589ae6f38ff1329846: Status 404 returned error can't find the container with id 37dac07c2a91a5cef26b0ab3f4503bd3890d4a1296713e589ae6f38ff1329846 Nov 03 02:45:01.020536 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:45:01.020511 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b8b50df_b64b_4249_864d_841839156ae5.slice/crio-13aaedb2064f55195c263df9063dce8c02679233aacfa0ae48c3263b9984b407 WatchSource:0}: Error finding container 13aaedb2064f55195c263df9063dce8c02679233aacfa0ae48c3263b9984b407: Status 404 returned error can't find the container with id 13aaedb2064f55195c263df9063dce8c02679233aacfa0ae48c3263b9984b407 Nov 03 02:45:01.021418 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:45:01.021394 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode94bb7db_1559_4c16_857f_3d20e0a7a008.slice/crio-df35027e69de80bea9fd6290a6b37897bf8057ec4590a27017248fc47bb4994c WatchSource:0}: Error finding container df35027e69de80bea9fd6290a6b37897bf8057ec4590a27017248fc47bb4994c: Status 404 returned error can't find the container with id df35027e69de80bea9fd6290a6b37897bf8057ec4590a27017248fc47bb4994c Nov 03 02:45:01.021999 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:01.021974 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894565-fz2lx"] Nov 03 02:45:01.026588 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:45:01.026566 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dc6be8c_60cf_4429_9ced_90ca7ed49656.slice/crio-d69e188f6b7b72ed9d6c990b02647f40915251db7dc88b6d7bba2240f4379d24 WatchSource:0}: Error finding container d69e188f6b7b72ed9d6c990b02647f40915251db7dc88b6d7bba2240f4379d24: Status 404 returned error can't find the container with id d69e188f6b7b72ed9d6c990b02647f40915251db7dc88b6d7bba2240f4379d24 Nov 03 02:45:01.033766 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:01.033743 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894565-66m7h"] Nov 03 02:45:01.047612 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:45:01.045704 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe40cb78_4bfe_4975_acbd_867c3366f3ae.slice/crio-92a5a63648bcbebe22847c16658ea5e1457e3a5e864c48073fbd7a09c4dda24a WatchSource:0}: Error finding container 92a5a63648bcbebe22847c16658ea5e1457e3a5e864c48073fbd7a09c4dda24a: Status 404 returned error can't find the container with id 92a5a63648bcbebe22847c16658ea5e1457e3a5e864c48073fbd7a09c4dda24a Nov 03 02:45:01.737716 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:01.737671 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894565-fz2lx" event={"ID":"0dc6be8c-60cf-4429-9ced-90ca7ed49656","Type":"ContainerStarted","Data":"62181e77f4d7ac961dcc2c9764393c111c8ffa5d8e6a1599e8696a8dba4d6727"} Nov 03 02:45:01.738261 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:01.737775 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894565-fz2lx" event={"ID":"0dc6be8c-60cf-4429-9ced-90ca7ed49656","Type":"ContainerStarted","Data":"d69e188f6b7b72ed9d6c990b02647f40915251db7dc88b6d7bba2240f4379d24"} Nov 03 02:45:01.739271 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:01.739224 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894565-lvznm" event={"ID":"ed96e13a-3942-46e6-846f-027f5d1718cc","Type":"ContainerStarted","Data":"e7dd244e7c8084f07a23e6b316a0754ad4914819626bc7908f43d02b9c99070e"} Nov 03 02:45:01.739394 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:01.739313 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894565-lvznm" event={"ID":"ed96e13a-3942-46e6-846f-027f5d1718cc","Type":"ContainerStarted","Data":"4eedb02384d1d3e75f8b5cfab773881df7ec995f13bc723163534877dfc47275"} Nov 03 02:45:01.740765 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:01.740739 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894565-47qxs" event={"ID":"6b8b50df-b64b-4249-864d-841839156ae5","Type":"ContainerStarted","Data":"68133302a1643f8d9cff9682bab544052b015e6e82af32e22ca238840c2d70e1"} Nov 03 02:45:01.740846 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:01.740789 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894565-47qxs" event={"ID":"6b8b50df-b64b-4249-864d-841839156ae5","Type":"ContainerStarted","Data":"13aaedb2064f55195c263df9063dce8c02679233aacfa0ae48c3263b9984b407"} Nov 03 02:45:01.742196 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:01.742172 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894565-6wfpx" event={"ID":"660b74cc-c09a-48c2-b447-d869e6f3efa2","Type":"ContainerStarted","Data":"7f5f58b3ddef2d4b11c26cb0e3d57be00f73403f1daffe6350a020bfda58209d"} Nov 03 02:45:01.742286 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:01.742222 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894565-6wfpx" event={"ID":"660b74cc-c09a-48c2-b447-d869e6f3efa2","Type":"ContainerStarted","Data":"37dac07c2a91a5cef26b0ab3f4503bd3890d4a1296713e589ae6f38ff1329846"} Nov 03 02:45:01.743726 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:01.743702 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894565-kdmq9" event={"ID":"e94bb7db-1559-4c16-857f-3d20e0a7a008","Type":"ContainerStarted","Data":"4075aeaaa88420c4fe9e30c372a8d0a7e0ac19b9a35acf2ef4ad18bef6e3e95e"} Nov 03 02:45:01.743818 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:01.743743 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894565-kdmq9" event={"ID":"e94bb7db-1559-4c16-857f-3d20e0a7a008","Type":"ContainerStarted","Data":"df35027e69de80bea9fd6290a6b37897bf8057ec4590a27017248fc47bb4994c"} Nov 03 02:45:01.745242 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:01.745220 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894565-tqf6b" event={"ID":"305b0cb8-2348-4517-86a5-f77bfe6a3814","Type":"ContainerStarted","Data":"3e9b6b71f7c249c21f2008ab5c1f443208252f00cc331acad4dd07ef89687220"} Nov 03 02:45:01.745333 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:01.745262 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894565-tqf6b" event={"ID":"305b0cb8-2348-4517-86a5-f77bfe6a3814","Type":"ContainerStarted","Data":"6ff257238610e0b194e47cdcfc1a3cb56160b0702fb599f4210e807cf8b10853"} Nov 03 02:45:01.746741 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:01.746722 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894565-66m7h" event={"ID":"fe40cb78-4bfe-4975-acbd-867c3366f3ae","Type":"ContainerStarted","Data":"20ded7c668fadd8208cce9fc2a4809a8f4300a2597e4dc53f4bb5594fd59cfe4"} Nov 03 02:45:01.746801 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:01.746760 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894565-66m7h" event={"ID":"fe40cb78-4bfe-4975-acbd-867c3366f3ae","Type":"ContainerStarted","Data":"92a5a63648bcbebe22847c16658ea5e1457e3a5e864c48073fbd7a09c4dda24a"} Nov 03 02:45:01.748020 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:01.748001 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894565-g2zcb" event={"ID":"aeb1d4c3-d1f5-49f4-9e3c-47ff2b640ac9","Type":"ContainerStarted","Data":"5f87109f7e1234ee3dafd0e04f5e5d23f606f3e2aa8bb52aedc90af0710f4314"} Nov 03 02:45:01.748085 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:01.748035 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894565-g2zcb" event={"ID":"aeb1d4c3-d1f5-49f4-9e3c-47ff2b640ac9","Type":"ContainerStarted","Data":"46536676785b1461233e7b99677f9ec73c3487672c841c4fda7667ac85d73a0c"} Nov 03 02:45:01.845345 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:01.845244 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894565-fz2lx" podStartSLOduration=1.8452310600000001 podStartE2EDuration="1.84523106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:45:01.843540033 +0000 UTC m=+3179.102738190" watchObservedRunningTime="2026-11-03 02:45:01.84523106 +0000 UTC m=+3179.104429213" Nov 03 02:45:01.949710 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:01.949660 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894565-g2zcb" podStartSLOduration=1.94964714 podStartE2EDuration="1.94964714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:45:01.947830589 +0000 UTC m=+3179.207028743" watchObservedRunningTime="2026-11-03 02:45:01.94964714 +0000 UTC m=+3179.208845293" Nov 03 02:45:02.053755 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:02.053708 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894565-kdmq9" podStartSLOduration=2.053694569 podStartE2EDuration="2.053694569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:45:02.053116157 +0000 UTC m=+3179.312314323" watchObservedRunningTime="2026-11-03 02:45:02.053694569 +0000 UTC m=+3179.312892723" Nov 03 02:45:02.161115 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:02.161069 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894565-66m7h" podStartSLOduration=2.161053898 podStartE2EDuration="2.161053898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:45:02.157455853 +0000 UTC m=+3179.416654007" watchObservedRunningTime="2026-11-03 02:45:02.161053898 +0000 UTC m=+3179.420252052" Nov 03 02:45:02.264162 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:02.264116 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894565-tqf6b" podStartSLOduration=2.264102625 podStartE2EDuration="2.264102625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:45:02.26181856 +0000 UTC m=+3179.521016715" watchObservedRunningTime="2026-11-03 02:45:02.264102625 +0000 UTC m=+3179.523300779" Nov 03 02:45:02.367735 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:02.367685 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894565-lvznm" podStartSLOduration=2.367671 podStartE2EDuration="2.367671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:45:02.36686934 +0000 UTC m=+3179.626067493" watchObservedRunningTime="2026-11-03 02:45:02.367671 +0000 UTC m=+3179.626869153" Nov 03 02:45:02.474886 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:02.474777 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894565-6wfpx" podStartSLOduration=2.474763763 podStartE2EDuration="2.474763763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:45:02.473888733 +0000 UTC m=+3179.733086888" watchObservedRunningTime="2026-11-03 02:45:02.474763763 +0000 UTC m=+3179.733961915" Nov 03 02:45:02.583060 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:02.583006 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894565-47qxs" podStartSLOduration=2.58299324 podStartE2EDuration="2.58299324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:45:02.580613816 +0000 UTC m=+3179.839811970" watchObservedRunningTime="2026-11-03 02:45:02.58299324 +0000 UTC m=+3179.842191393" Nov 03 02:45:02.756076 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:02.755978 5605 generic.go:356] "Generic (PLEG): container finished" podID="0dc6be8c-60cf-4429-9ced-90ca7ed49656" containerID="62181e77f4d7ac961dcc2c9764393c111c8ffa5d8e6a1599e8696a8dba4d6727" exitCode=60 Nov 03 02:45:02.756076 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:02.756059 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894565-fz2lx" event={"ID":"0dc6be8c-60cf-4429-9ced-90ca7ed49656","Type":"ContainerDied","Data":"62181e77f4d7ac961dcc2c9764393c111c8ffa5d8e6a1599e8696a8dba4d6727"} Nov 03 02:45:02.756582 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:02.756333 5605 scope.go:122] "RemoveContainer" containerID="62181e77f4d7ac961dcc2c9764393c111c8ffa5d8e6a1599e8696a8dba4d6727" Nov 03 02:45:02.757868 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:02.757847 5605 generic.go:356] "Generic (PLEG): container finished" podID="ed96e13a-3942-46e6-846f-027f5d1718cc" containerID="e7dd244e7c8084f07a23e6b316a0754ad4914819626bc7908f43d02b9c99070e" exitCode=60 Nov 03 02:45:02.757999 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:02.757945 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894565-lvznm" event={"ID":"ed96e13a-3942-46e6-846f-027f5d1718cc","Type":"ContainerDied","Data":"e7dd244e7c8084f07a23e6b316a0754ad4914819626bc7908f43d02b9c99070e"} Nov 03 02:45:02.758239 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:02.758224 5605 scope.go:122] "RemoveContainer" containerID="e7dd244e7c8084f07a23e6b316a0754ad4914819626bc7908f43d02b9c99070e" Nov 03 02:45:02.760138 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:02.759981 5605 generic.go:356] "Generic (PLEG): container finished" podID="6b8b50df-b64b-4249-864d-841839156ae5" containerID="68133302a1643f8d9cff9682bab544052b015e6e82af32e22ca238840c2d70e1" exitCode=60 Nov 03 02:45:02.760138 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:02.760007 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894565-47qxs" event={"ID":"6b8b50df-b64b-4249-864d-841839156ae5","Type":"ContainerDied","Data":"68133302a1643f8d9cff9682bab544052b015e6e82af32e22ca238840c2d70e1"} Nov 03 02:45:02.760285 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:02.760267 5605 scope.go:122] "RemoveContainer" containerID="68133302a1643f8d9cff9682bab544052b015e6e82af32e22ca238840c2d70e1" Nov 03 02:45:02.761917 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:02.761896 5605 generic.go:356] "Generic (PLEG): container finished" podID="660b74cc-c09a-48c2-b447-d869e6f3efa2" containerID="7f5f58b3ddef2d4b11c26cb0e3d57be00f73403f1daffe6350a020bfda58209d" exitCode=60 Nov 03 02:45:02.762022 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:02.761947 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894565-6wfpx" event={"ID":"660b74cc-c09a-48c2-b447-d869e6f3efa2","Type":"ContainerDied","Data":"7f5f58b3ddef2d4b11c26cb0e3d57be00f73403f1daffe6350a020bfda58209d"} Nov 03 02:45:02.762204 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:02.762184 5605 scope.go:122] "RemoveContainer" containerID="7f5f58b3ddef2d4b11c26cb0e3d57be00f73403f1daffe6350a020bfda58209d" Nov 03 02:45:02.763940 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:02.763914 5605 generic.go:356] "Generic (PLEG): container finished" podID="e94bb7db-1559-4c16-857f-3d20e0a7a008" containerID="4075aeaaa88420c4fe9e30c372a8d0a7e0ac19b9a35acf2ef4ad18bef6e3e95e" exitCode=60 Nov 03 02:45:02.764052 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:02.763991 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894565-kdmq9" event={"ID":"e94bb7db-1559-4c16-857f-3d20e0a7a008","Type":"ContainerDied","Data":"4075aeaaa88420c4fe9e30c372a8d0a7e0ac19b9a35acf2ef4ad18bef6e3e95e"} Nov 03 02:45:02.764177 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:02.764162 5605 scope.go:122] "RemoveContainer" containerID="4075aeaaa88420c4fe9e30c372a8d0a7e0ac19b9a35acf2ef4ad18bef6e3e95e" Nov 03 02:45:02.765887 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:02.765673 5605 generic.go:356] "Generic (PLEG): container finished" podID="305b0cb8-2348-4517-86a5-f77bfe6a3814" containerID="3e9b6b71f7c249c21f2008ab5c1f443208252f00cc331acad4dd07ef89687220" exitCode=60 Nov 03 02:45:02.765887 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:02.765746 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894565-tqf6b" event={"ID":"305b0cb8-2348-4517-86a5-f77bfe6a3814","Type":"ContainerDied","Data":"3e9b6b71f7c249c21f2008ab5c1f443208252f00cc331acad4dd07ef89687220"} Nov 03 02:45:02.766048 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:02.765933 5605 scope.go:122] "RemoveContainer" containerID="3e9b6b71f7c249c21f2008ab5c1f443208252f00cc331acad4dd07ef89687220" Nov 03 02:45:02.767892 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:02.767875 5605 generic.go:356] "Generic (PLEG): container finished" podID="fe40cb78-4bfe-4975-acbd-867c3366f3ae" containerID="20ded7c668fadd8208cce9fc2a4809a8f4300a2597e4dc53f4bb5594fd59cfe4" exitCode=60 Nov 03 02:45:02.767991 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:02.767972 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894565-66m7h" event={"ID":"fe40cb78-4bfe-4975-acbd-867c3366f3ae","Type":"ContainerDied","Data":"20ded7c668fadd8208cce9fc2a4809a8f4300a2597e4dc53f4bb5594fd59cfe4"} Nov 03 02:45:02.768152 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:02.768137 5605 scope.go:122] "RemoveContainer" containerID="20ded7c668fadd8208cce9fc2a4809a8f4300a2597e4dc53f4bb5594fd59cfe4" Nov 03 02:45:02.769542 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:02.769522 5605 generic.go:356] "Generic (PLEG): container finished" podID="aeb1d4c3-d1f5-49f4-9e3c-47ff2b640ac9" containerID="5f87109f7e1234ee3dafd0e04f5e5d23f606f3e2aa8bb52aedc90af0710f4314" exitCode=60 Nov 03 02:45:02.769635 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:02.769589 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894565-g2zcb" event={"ID":"aeb1d4c3-d1f5-49f4-9e3c-47ff2b640ac9","Type":"ContainerDied","Data":"5f87109f7e1234ee3dafd0e04f5e5d23f606f3e2aa8bb52aedc90af0710f4314"} Nov 03 02:45:02.769752 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:02.769738 5605 scope.go:122] "RemoveContainer" containerID="5f87109f7e1234ee3dafd0e04f5e5d23f606f3e2aa8bb52aedc90af0710f4314" Nov 03 02:45:03.778841 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:03.778807 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894565-tqf6b" event={"ID":"305b0cb8-2348-4517-86a5-f77bfe6a3814","Type":"ContainerStarted","Data":"7e661e2ebada1e85d38e1f3e02c725603734ccfbc35c345767c5dcc1f58691b6"} Nov 03 02:45:03.780786 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:03.780757 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894565-66m7h" event={"ID":"fe40cb78-4bfe-4975-acbd-867c3366f3ae","Type":"ContainerStarted","Data":"293dc09cc0805a7deb1ea3c30c6de1bb25695776091d1e6c82371b2245f4b39c"} Nov 03 02:45:03.782571 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:03.782542 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894565-g2zcb" event={"ID":"aeb1d4c3-d1f5-49f4-9e3c-47ff2b640ac9","Type":"ContainerStarted","Data":"980e377aff3fe8f7f5b6ab00d7efb6f56762e2d8989fad5b027dd2942feb84b8"} Nov 03 02:45:03.784484 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:03.784448 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894565-fz2lx" event={"ID":"0dc6be8c-60cf-4429-9ced-90ca7ed49656","Type":"ContainerStarted","Data":"0251f4773108794bef5aea1de80e64a54796724aae1cc766fbc6e74e830a9396"} Nov 03 02:45:03.786171 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:03.786149 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894565-lvznm" event={"ID":"ed96e13a-3942-46e6-846f-027f5d1718cc","Type":"ContainerStarted","Data":"bca2678301e57a59607de50092323ce0b452d6e6d2021257a50626ea8db65799"} Nov 03 02:45:03.787937 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:03.787913 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894565-47qxs" event={"ID":"6b8b50df-b64b-4249-864d-841839156ae5","Type":"ContainerStarted","Data":"0c02b3b6b02ac222513614255d74abe97aa2d7fccd3f69e97bd6c8c7092f055a"} Nov 03 02:45:03.789562 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:03.789544 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894565-6wfpx" event={"ID":"660b74cc-c09a-48c2-b447-d869e6f3efa2","Type":"ContainerStarted","Data":"72ebd9217d0b101236d3b5293c3da9c52f2391b1f0370679183dd2581f61d7a4"} Nov 03 02:45:03.791253 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:03.791235 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894565-kdmq9" event={"ID":"e94bb7db-1559-4c16-857f-3d20e0a7a008","Type":"ContainerStarted","Data":"14b0be68f7dcd5c79ebc1de13244ffe5751740e8ec884141301ee6576b63b447"} Nov 03 02:45:04.800235 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:04.800197 5605 generic.go:356] "Generic (PLEG): container finished" podID="6b8b50df-b64b-4249-864d-841839156ae5" containerID="0c02b3b6b02ac222513614255d74abe97aa2d7fccd3f69e97bd6c8c7092f055a" exitCode=60 Nov 03 02:45:04.800730 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:04.800278 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894565-47qxs" event={"ID":"6b8b50df-b64b-4249-864d-841839156ae5","Type":"ContainerDied","Data":"0c02b3b6b02ac222513614255d74abe97aa2d7fccd3f69e97bd6c8c7092f055a"} Nov 03 02:45:04.800730 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:04.800351 5605 scope.go:122] "RemoveContainer" containerID="68133302a1643f8d9cff9682bab544052b015e6e82af32e22ca238840c2d70e1" Nov 03 02:45:04.800730 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:04.800571 5605 scope.go:122] "RemoveContainer" containerID="0c02b3b6b02ac222513614255d74abe97aa2d7fccd3f69e97bd6c8c7092f055a" Nov 03 02:45:04.800911 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:45:04.800881 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894565-47qxs_odh-ai-gateway-infra(6b8b50df-b64b-4249-864d-841839156ae5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894565-47qxs" podUID="6b8b50df-b64b-4249-864d-841839156ae5" Nov 03 02:45:04.802205 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:04.802185 5605 generic.go:356] "Generic (PLEG): container finished" podID="660b74cc-c09a-48c2-b447-d869e6f3efa2" containerID="72ebd9217d0b101236d3b5293c3da9c52f2391b1f0370679183dd2581f61d7a4" exitCode=60 Nov 03 02:45:04.802302 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:04.802267 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894565-6wfpx" event={"ID":"660b74cc-c09a-48c2-b447-d869e6f3efa2","Type":"ContainerDied","Data":"72ebd9217d0b101236d3b5293c3da9c52f2391b1f0370679183dd2581f61d7a4"} Nov 03 02:45:04.802486 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:04.802470 5605 scope.go:122] "RemoveContainer" containerID="72ebd9217d0b101236d3b5293c3da9c52f2391b1f0370679183dd2581f61d7a4" Nov 03 02:45:04.802752 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:45:04.802728 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894565-6wfpx_odh-ai-gateway-infra(660b74cc-c09a-48c2-b447-d869e6f3efa2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894565-6wfpx" podUID="660b74cc-c09a-48c2-b447-d869e6f3efa2" Nov 03 02:45:04.804113 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:04.804095 5605 generic.go:356] "Generic (PLEG): container finished" podID="e94bb7db-1559-4c16-857f-3d20e0a7a008" containerID="14b0be68f7dcd5c79ebc1de13244ffe5751740e8ec884141301ee6576b63b447" exitCode=60 Nov 03 02:45:04.804230 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:04.804178 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894565-kdmq9" event={"ID":"e94bb7db-1559-4c16-857f-3d20e0a7a008","Type":"ContainerDied","Data":"14b0be68f7dcd5c79ebc1de13244ffe5751740e8ec884141301ee6576b63b447"} Nov 03 02:45:04.804412 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:04.804384 5605 scope.go:122] "RemoveContainer" containerID="14b0be68f7dcd5c79ebc1de13244ffe5751740e8ec884141301ee6576b63b447" Nov 03 02:45:04.804654 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:45:04.804633 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894565-kdmq9_odh-ai-gateway-infra(e94bb7db-1559-4c16-857f-3d20e0a7a008)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894565-kdmq9" podUID="e94bb7db-1559-4c16-857f-3d20e0a7a008" Nov 03 02:45:04.806044 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:04.806025 5605 generic.go:356] "Generic (PLEG): container finished" podID="305b0cb8-2348-4517-86a5-f77bfe6a3814" containerID="7e661e2ebada1e85d38e1f3e02c725603734ccfbc35c345767c5dcc1f58691b6" exitCode=60 Nov 03 02:45:04.806128 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:04.806094 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894565-tqf6b" event={"ID":"305b0cb8-2348-4517-86a5-f77bfe6a3814","Type":"ContainerDied","Data":"7e661e2ebada1e85d38e1f3e02c725603734ccfbc35c345767c5dcc1f58691b6"} Nov 03 02:45:04.806333 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:04.806317 5605 scope.go:122] "RemoveContainer" containerID="7e661e2ebada1e85d38e1f3e02c725603734ccfbc35c345767c5dcc1f58691b6" Nov 03 02:45:04.806580 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:45:04.806560 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-521f1f-29894565-tqf6b_odh-ai-gateway-infra(305b0cb8-2348-4517-86a5-f77bfe6a3814)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894565-tqf6b" podUID="305b0cb8-2348-4517-86a5-f77bfe6a3814" Nov 03 02:45:04.808017 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:04.807999 5605 generic.go:356] "Generic (PLEG): container finished" podID="fe40cb78-4bfe-4975-acbd-867c3366f3ae" containerID="293dc09cc0805a7deb1ea3c30c6de1bb25695776091d1e6c82371b2245f4b39c" exitCode=60 Nov 03 02:45:04.808101 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:04.808088 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894565-66m7h" event={"ID":"fe40cb78-4bfe-4975-acbd-867c3366f3ae","Type":"ContainerDied","Data":"293dc09cc0805a7deb1ea3c30c6de1bb25695776091d1e6c82371b2245f4b39c"} Nov 03 02:45:04.808305 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:04.808286 5605 scope.go:122] "RemoveContainer" containerID="293dc09cc0805a7deb1ea3c30c6de1bb25695776091d1e6c82371b2245f4b39c" Nov 03 02:45:04.808543 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:45:04.808512 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894565-66m7h_odh-ai-gateway-infra(fe40cb78-4bfe-4975-acbd-867c3366f3ae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894565-66m7h" podUID="fe40cb78-4bfe-4975-acbd-867c3366f3ae" Nov 03 02:45:04.810136 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:04.810116 5605 generic.go:356] "Generic (PLEG): container finished" podID="aeb1d4c3-d1f5-49f4-9e3c-47ff2b640ac9" containerID="980e377aff3fe8f7f5b6ab00d7efb6f56762e2d8989fad5b027dd2942feb84b8" exitCode=60 Nov 03 02:45:04.810232 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:04.810218 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894565-g2zcb" event={"ID":"aeb1d4c3-d1f5-49f4-9e3c-47ff2b640ac9","Type":"ContainerDied","Data":"980e377aff3fe8f7f5b6ab00d7efb6f56762e2d8989fad5b027dd2942feb84b8"} Nov 03 02:45:04.810502 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:04.810480 5605 scope.go:122] "RemoveContainer" containerID="980e377aff3fe8f7f5b6ab00d7efb6f56762e2d8989fad5b027dd2942feb84b8" Nov 03 02:45:04.810748 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:45:04.810726 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894565-g2zcb_odh-ai-gateway-infra(aeb1d4c3-d1f5-49f4-9e3c-47ff2b640ac9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894565-g2zcb" podUID="aeb1d4c3-d1f5-49f4-9e3c-47ff2b640ac9" Nov 03 02:45:04.812318 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:04.812295 5605 generic.go:356] "Generic (PLEG): container finished" podID="0dc6be8c-60cf-4429-9ced-90ca7ed49656" containerID="0251f4773108794bef5aea1de80e64a54796724aae1cc766fbc6e74e830a9396" exitCode=60 Nov 03 02:45:04.812441 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:04.812399 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894565-fz2lx" event={"ID":"0dc6be8c-60cf-4429-9ced-90ca7ed49656","Type":"ContainerDied","Data":"0251f4773108794bef5aea1de80e64a54796724aae1cc766fbc6e74e830a9396"} Nov 03 02:45:04.812599 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:04.812573 5605 scope.go:122] "RemoveContainer" containerID="0251f4773108794bef5aea1de80e64a54796724aae1cc766fbc6e74e830a9396" Nov 03 02:45:04.812799 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:45:04.812781 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-c7d758-29894565-fz2lx_odh-ai-gateway-infra(0dc6be8c-60cf-4429-9ced-90ca7ed49656)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894565-fz2lx" podUID="0dc6be8c-60cf-4429-9ced-90ca7ed49656" Nov 03 02:45:04.814384 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:04.814335 5605 generic.go:356] "Generic (PLEG): container finished" podID="ed96e13a-3942-46e6-846f-027f5d1718cc" containerID="bca2678301e57a59607de50092323ce0b452d6e6d2021257a50626ea8db65799" exitCode=60 Nov 03 02:45:04.814481 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:04.814404 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894565-lvznm" event={"ID":"ed96e13a-3942-46e6-846f-027f5d1718cc","Type":"ContainerDied","Data":"bca2678301e57a59607de50092323ce0b452d6e6d2021257a50626ea8db65799"} Nov 03 02:45:04.814602 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:04.814588 5605 scope.go:122] "RemoveContainer" containerID="bca2678301e57a59607de50092323ce0b452d6e6d2021257a50626ea8db65799" Nov 03 02:45:04.814770 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:45:04.814754 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894565-lvznm_odh-ai-gateway-infra(ed96e13a-3942-46e6-846f-027f5d1718cc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894565-lvznm" podUID="ed96e13a-3942-46e6-846f-027f5d1718cc" Nov 03 02:45:04.816459 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:04.816439 5605 scope.go:122] "RemoveContainer" containerID="7f5f58b3ddef2d4b11c26cb0e3d57be00f73403f1daffe6350a020bfda58209d" Nov 03 02:45:04.828837 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:04.828819 5605 scope.go:122] "RemoveContainer" containerID="4075aeaaa88420c4fe9e30c372a8d0a7e0ac19b9a35acf2ef4ad18bef6e3e95e" Nov 03 02:45:04.840592 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:04.840575 5605 scope.go:122] "RemoveContainer" containerID="3e9b6b71f7c249c21f2008ab5c1f443208252f00cc331acad4dd07ef89687220" Nov 03 02:45:04.852875 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:04.852858 5605 scope.go:122] "RemoveContainer" containerID="20ded7c668fadd8208cce9fc2a4809a8f4300a2597e4dc53f4bb5594fd59cfe4" Nov 03 02:45:04.865073 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:04.865055 5605 scope.go:122] "RemoveContainer" containerID="5f87109f7e1234ee3dafd0e04f5e5d23f606f3e2aa8bb52aedc90af0710f4314" Nov 03 02:45:04.876762 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:04.876747 5605 scope.go:122] "RemoveContainer" containerID="62181e77f4d7ac961dcc2c9764393c111c8ffa5d8e6a1599e8696a8dba4d6727" Nov 03 02:45:04.888272 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:04.888257 5605 scope.go:122] "RemoveContainer" containerID="e7dd244e7c8084f07a23e6b316a0754ad4914819626bc7908f43d02b9c99070e" Nov 03 02:45:05.689185 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:05.689156 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-038c21-5d649bd5f-tcdq5" Nov 03 02:45:05.689339 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:05.689227 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-038c21-c54876867-hxprn" Nov 03 02:45:05.823728 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:05.823700 5605 scope.go:122] "RemoveContainer" containerID="980e377aff3fe8f7f5b6ab00d7efb6f56762e2d8989fad5b027dd2942feb84b8" Nov 03 02:45:05.824149 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:45:05.823953 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894565-g2zcb_odh-ai-gateway-infra(aeb1d4c3-d1f5-49f4-9e3c-47ff2b640ac9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894565-g2zcb" podUID="aeb1d4c3-d1f5-49f4-9e3c-47ff2b640ac9" Nov 03 02:45:05.825542 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:05.825521 5605 scope.go:122] "RemoveContainer" containerID="0251f4773108794bef5aea1de80e64a54796724aae1cc766fbc6e74e830a9396" Nov 03 02:45:05.825784 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:45:05.825758 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-c7d758-29894565-fz2lx_odh-ai-gateway-infra(0dc6be8c-60cf-4429-9ced-90ca7ed49656)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894565-fz2lx" podUID="0dc6be8c-60cf-4429-9ced-90ca7ed49656" Nov 03 02:45:05.827281 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:05.827264 5605 scope.go:122] "RemoveContainer" containerID="bca2678301e57a59607de50092323ce0b452d6e6d2021257a50626ea8db65799" Nov 03 02:45:05.827529 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:45:05.827501 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894565-lvznm_odh-ai-gateway-infra(ed96e13a-3942-46e6-846f-027f5d1718cc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894565-lvznm" podUID="ed96e13a-3942-46e6-846f-027f5d1718cc" Nov 03 02:45:05.829072 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:05.829051 5605 scope.go:122] "RemoveContainer" containerID="0c02b3b6b02ac222513614255d74abe97aa2d7fccd3f69e97bd6c8c7092f055a" Nov 03 02:45:05.829312 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:45:05.829284 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894565-47qxs_odh-ai-gateway-infra(6b8b50df-b64b-4249-864d-841839156ae5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894565-47qxs" podUID="6b8b50df-b64b-4249-864d-841839156ae5" Nov 03 02:45:05.830563 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:05.830544 5605 scope.go:122] "RemoveContainer" containerID="72ebd9217d0b101236d3b5293c3da9c52f2391b1f0370679183dd2581f61d7a4" Nov 03 02:45:05.830757 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:45:05.830736 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894565-6wfpx_odh-ai-gateway-infra(660b74cc-c09a-48c2-b447-d869e6f3efa2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894565-6wfpx" podUID="660b74cc-c09a-48c2-b447-d869e6f3efa2" Nov 03 02:45:05.832182 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:05.832164 5605 scope.go:122] "RemoveContainer" containerID="14b0be68f7dcd5c79ebc1de13244ffe5751740e8ec884141301ee6576b63b447" Nov 03 02:45:05.832400 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:45:05.832381 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894565-kdmq9_odh-ai-gateway-infra(e94bb7db-1559-4c16-857f-3d20e0a7a008)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894565-kdmq9" podUID="e94bb7db-1559-4c16-857f-3d20e0a7a008" Nov 03 02:45:05.833868 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:05.833849 5605 scope.go:122] "RemoveContainer" containerID="7e661e2ebada1e85d38e1f3e02c725603734ccfbc35c345767c5dcc1f58691b6" Nov 03 02:45:05.834027 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:45:05.834009 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-521f1f-29894565-tqf6b_odh-ai-gateway-infra(305b0cb8-2348-4517-86a5-f77bfe6a3814)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894565-tqf6b" podUID="305b0cb8-2348-4517-86a5-f77bfe6a3814" Nov 03 02:45:05.835515 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:05.835498 5605 scope.go:122] "RemoveContainer" containerID="293dc09cc0805a7deb1ea3c30c6de1bb25695776091d1e6c82371b2245f4b39c" Nov 03 02:45:05.835674 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:45:05.835659 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894565-66m7h_odh-ai-gateway-infra(fe40cb78-4bfe-4975-acbd-867c3366f3ae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894565-66m7h" podUID="fe40cb78-4bfe-4975-acbd-867c3366f3ae" Nov 03 02:45:12.343550 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:12.343510 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-f735f4/auth-test-model-f735f4-kserve-c545848fc-glzwl"] Nov 03 02:45:14.850664 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:14.850633 5605 scope.go:122] "RemoveContainer" containerID="14b0be68f7dcd5c79ebc1de13244ffe5751740e8ec884141301ee6576b63b447" Nov 03 02:45:15.843276 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:15.843243 5605 scope.go:122] "RemoveContainer" containerID="293dc09cc0805a7deb1ea3c30c6de1bb25695776091d1e6c82371b2245f4b39c" Nov 03 02:45:15.843467 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:15.843371 5605 scope.go:122] "RemoveContainer" containerID="0251f4773108794bef5aea1de80e64a54796724aae1cc766fbc6e74e830a9396" Nov 03 02:45:15.889645 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:15.889618 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7229-8d33da"] Nov 03 02:45:15.894742 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:15.894721 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7229-8d33da" Nov 03 02:45:15.902130 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:15.902108 5605 generic.go:356] "Generic (PLEG): container finished" podID="e94bb7db-1559-4c16-857f-3d20e0a7a008" containerID="c36ba1e298bceb4fc382921f6d66af34295c51697da8342394b94d5ac8fb3235" exitCode=60 Nov 03 02:45:15.902242 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:15.902182 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894565-kdmq9" event={"ID":"e94bb7db-1559-4c16-857f-3d20e0a7a008","Type":"ContainerDied","Data":"c36ba1e298bceb4fc382921f6d66af34295c51697da8342394b94d5ac8fb3235"} Nov 03 02:45:15.902242 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:15.902237 5605 scope.go:122] "RemoveContainer" containerID="14b0be68f7dcd5c79ebc1de13244ffe5751740e8ec884141301ee6576b63b447" Nov 03 02:45:15.902696 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:15.902677 5605 scope.go:122] "RemoveContainer" containerID="c36ba1e298bceb4fc382921f6d66af34295c51697da8342394b94d5ac8fb3235" Nov 03 02:45:15.903009 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:45:15.902981 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894565-kdmq9_odh-ai-gateway-infra(e94bb7db-1559-4c16-857f-3d20e0a7a008)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894565-kdmq9" podUID="e94bb7db-1559-4c16-857f-3d20e0a7a008" Nov 03 02:45:15.926727 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:15.926699 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbvsj\" (UniqueName: \"kubernetes.io/projected/2e096621-72ec-4cd2-aeac-1c434feea16d-kube-api-access-nbvsj\") pod \"test-curl-7229-8d33da\" (UID: \"2e096621-72ec-4cd2-aeac-1c434feea16d\") " pod="opendatahub/test-curl-7229-8d33da" Nov 03 02:45:15.946561 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:15.946535 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7229-8d33da"] Nov 03 02:45:15.989261 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:15.989230 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-579jc\"" Nov 03 02:45:16.028128 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:16.028047 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbvsj\" (UniqueName: \"kubernetes.io/projected/2e096621-72ec-4cd2-aeac-1c434feea16d-kube-api-access-nbvsj\") pod \"test-curl-7229-8d33da\" (UID: \"2e096621-72ec-4cd2-aeac-1c434feea16d\") " pod="opendatahub/test-curl-7229-8d33da" Nov 03 02:45:16.155234 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:16.155189 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbvsj\" (UniqueName: \"kubernetes.io/projected/2e096621-72ec-4cd2-aeac-1c434feea16d-kube-api-access-nbvsj\") pod \"test-curl-7229-8d33da\" (UID: \"2e096621-72ec-4cd2-aeac-1c434feea16d\") " pod="opendatahub/test-curl-7229-8d33da" Nov 03 02:45:16.212710 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:16.212670 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7229-8d33da" Nov 03 02:45:16.583555 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:16.583515 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7229-8d33da"] Nov 03 02:45:16.589189 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:45:16.589157 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e096621_72ec_4cd2_aeac_1c434feea16d.slice/crio-aae53d753ca0858e91119fc4d0fff70e45cb46e2cc7838afaf63a715d07a5c08 WatchSource:0}: Error finding container aae53d753ca0858e91119fc4d0fff70e45cb46e2cc7838afaf63a715d07a5c08: Status 404 returned error can't find the container with id aae53d753ca0858e91119fc4d0fff70e45cb46e2cc7838afaf63a715d07a5c08 Nov 03 02:45:16.851157 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:16.851082 5605 scope.go:122] "RemoveContainer" containerID="bca2678301e57a59607de50092323ce0b452d6e6d2021257a50626ea8db65799" Nov 03 02:45:16.851157 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:16.851121 5605 scope.go:122] "RemoveContainer" containerID="980e377aff3fe8f7f5b6ab00d7efb6f56762e2d8989fad5b027dd2942feb84b8" Nov 03 02:45:16.912288 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:16.912252 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7229-8d33da" event={"ID":"2e096621-72ec-4cd2-aeac-1c434feea16d","Type":"ContainerStarted","Data":"40107c4c8de8bff977662746fa58c9fc1389eb6a75c6bed32b0121a8bd04ccd6"} Nov 03 02:45:16.912791 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:16.912300 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7229-8d33da" event={"ID":"2e096621-72ec-4cd2-aeac-1c434feea16d","Type":"ContainerStarted","Data":"aae53d753ca0858e91119fc4d0fff70e45cb46e2cc7838afaf63a715d07a5c08"} Nov 03 02:45:16.914270 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:16.914244 5605 generic.go:356] "Generic (PLEG): container finished" podID="fe40cb78-4bfe-4975-acbd-867c3366f3ae" containerID="098ffbfd37ab052cbb015ebaad9d5ffc4dd13a69588157859ff9a87706ec71e1" exitCode=60 Nov 03 02:45:16.914429 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:16.914314 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894565-66m7h" event={"ID":"fe40cb78-4bfe-4975-acbd-867c3366f3ae","Type":"ContainerDied","Data":"098ffbfd37ab052cbb015ebaad9d5ffc4dd13a69588157859ff9a87706ec71e1"} Nov 03 02:45:16.914429 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:16.914403 5605 scope.go:122] "RemoveContainer" containerID="293dc09cc0805a7deb1ea3c30c6de1bb25695776091d1e6c82371b2245f4b39c" Nov 03 02:45:16.914802 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:16.914777 5605 scope.go:122] "RemoveContainer" containerID="098ffbfd37ab052cbb015ebaad9d5ffc4dd13a69588157859ff9a87706ec71e1" Nov 03 02:45:16.915111 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:45:16.915085 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894565-66m7h_odh-ai-gateway-infra(fe40cb78-4bfe-4975-acbd-867c3366f3ae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894565-66m7h" podUID="fe40cb78-4bfe-4975-acbd-867c3366f3ae" Nov 03 02:45:16.916593 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:16.916573 5605 generic.go:356] "Generic (PLEG): container finished" podID="0dc6be8c-60cf-4429-9ced-90ca7ed49656" containerID="df2b472892eaf0b9e698411e405db52fbc8bb3be37e26fe5f175586149b19806" exitCode=60 Nov 03 02:45:16.916696 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:16.916642 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894565-fz2lx" event={"ID":"0dc6be8c-60cf-4429-9ced-90ca7ed49656","Type":"ContainerDied","Data":"df2b472892eaf0b9e698411e405db52fbc8bb3be37e26fe5f175586149b19806"} Nov 03 02:45:16.917070 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:16.917053 5605 scope.go:122] "RemoveContainer" containerID="df2b472892eaf0b9e698411e405db52fbc8bb3be37e26fe5f175586149b19806" Nov 03 02:45:16.917296 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:45:16.917276 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-c7d758-29894565-fz2lx_odh-ai-gateway-infra(0dc6be8c-60cf-4429-9ced-90ca7ed49656)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894565-fz2lx" podUID="0dc6be8c-60cf-4429-9ced-90ca7ed49656" Nov 03 02:45:16.928280 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:16.928260 5605 scope.go:122] "RemoveContainer" containerID="0251f4773108794bef5aea1de80e64a54796724aae1cc766fbc6e74e830a9396" Nov 03 02:45:17.017492 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:17.017435 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7229-8d33da" podStartSLOduration=2.017416872 podStartE2EDuration="2.017416872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:45:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:45:17.016112982 +0000 UTC m=+3194.275311135" watchObservedRunningTime="2026-11-03 02:45:17.017416872 +0000 UTC m=+3194.276615023" Nov 03 02:45:17.115846 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:17.115760 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894565-kdmq9"] Nov 03 02:45:17.278499 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:17.278469 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894565-kdmq9" Nov 03 02:45:17.344368 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:17.344321 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e94bb7db-1559-4c16-857f-3d20e0a7a008-kube-api-access-4lmhd\" (UniqueName: \"kubernetes.io/projected/e94bb7db-1559-4c16-857f-3d20e0a7a008-kube-api-access-4lmhd\") pod \"e94bb7db-1559-4c16-857f-3d20e0a7a008\" (UID: \"e94bb7db-1559-4c16-857f-3d20e0a7a008\") " Nov 03 02:45:17.344571 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:17.344407 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e94bb7db-1559-4c16-857f-3d20e0a7a008-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e94bb7db-1559-4c16-857f-3d20e0a7a008-maas-api-service-ca\") pod \"e94bb7db-1559-4c16-857f-3d20e0a7a008\" (UID: \"e94bb7db-1559-4c16-857f-3d20e0a7a008\") " Nov 03 02:45:17.344797 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:17.344769 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e94bb7db-1559-4c16-857f-3d20e0a7a008-maas-api-service-ca" pod "e94bb7db-1559-4c16-857f-3d20e0a7a008" (UID: "e94bb7db-1559-4c16-857f-3d20e0a7a008"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:45:17.344909 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:17.344892 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e94bb7db-1559-4c16-857f-3d20e0a7a008-maas-api-service-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:17.346487 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:17.346465 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e94bb7db-1559-4c16-857f-3d20e0a7a008-kube-api-access-4lmhd" pod "e94bb7db-1559-4c16-857f-3d20e0a7a008" (UID: "e94bb7db-1559-4c16-857f-3d20e0a7a008"). InnerVolumeSpecName "kube-api-access-4lmhd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:45:17.446332 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:17.446239 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lmhd\" (UniqueName: \"kubernetes.io/projected/e94bb7db-1559-4c16-857f-3d20e0a7a008-kube-api-access-4lmhd\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:17.843870 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:17.843839 5605 scope.go:122] "RemoveContainer" containerID="0c02b3b6b02ac222513614255d74abe97aa2d7fccd3f69e97bd6c8c7092f055a" Nov 03 02:45:17.844048 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:17.843930 5605 scope.go:122] "RemoveContainer" containerID="72ebd9217d0b101236d3b5293c3da9c52f2391b1f0370679183dd2581f61d7a4" Nov 03 02:45:17.924831 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:17.924801 5605 generic.go:356] "Generic (PLEG): container finished" podID="aeb1d4c3-d1f5-49f4-9e3c-47ff2b640ac9" containerID="9f1de971624a88ab08901e6748fa066b9b7feac486f4e00fcf99a1b7752f7ea7" exitCode=60 Nov 03 02:45:17.925272 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:17.924870 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894565-g2zcb" event={"ID":"aeb1d4c3-d1f5-49f4-9e3c-47ff2b640ac9","Type":"ContainerDied","Data":"9f1de971624a88ab08901e6748fa066b9b7feac486f4e00fcf99a1b7752f7ea7"} Nov 03 02:45:17.925272 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:17.924933 5605 scope.go:122] "RemoveContainer" containerID="980e377aff3fe8f7f5b6ab00d7efb6f56762e2d8989fad5b027dd2942feb84b8" Nov 03 02:45:17.925424 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:17.925301 5605 scope.go:122] "RemoveContainer" containerID="9f1de971624a88ab08901e6748fa066b9b7feac486f4e00fcf99a1b7752f7ea7" Nov 03 02:45:17.925668 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:45:17.925642 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894565-g2zcb_odh-ai-gateway-infra(aeb1d4c3-d1f5-49f4-9e3c-47ff2b640ac9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894565-g2zcb" podUID="aeb1d4c3-d1f5-49f4-9e3c-47ff2b640ac9" Nov 03 02:45:17.928389 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:17.928350 5605 generic.go:356] "Generic (PLEG): container finished" podID="ed96e13a-3942-46e6-846f-027f5d1718cc" containerID="6e0e7015a3bd59d73dd0ce6b123398da4594317e82d7d3f331967c4e7dac3870" exitCode=60 Nov 03 02:45:17.928490 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:17.928384 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894565-lvznm" event={"ID":"ed96e13a-3942-46e6-846f-027f5d1718cc","Type":"ContainerDied","Data":"6e0e7015a3bd59d73dd0ce6b123398da4594317e82d7d3f331967c4e7dac3870"} Nov 03 02:45:17.928775 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:17.928755 5605 scope.go:122] "RemoveContainer" containerID="6e0e7015a3bd59d73dd0ce6b123398da4594317e82d7d3f331967c4e7dac3870" Nov 03 02:45:17.929039 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:45:17.929014 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894565-lvznm_odh-ai-gateway-infra(ed96e13a-3942-46e6-846f-027f5d1718cc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894565-lvznm" podUID="ed96e13a-3942-46e6-846f-027f5d1718cc" Nov 03 02:45:17.929744 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:17.929726 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894565-kdmq9" Nov 03 02:45:17.929835 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:17.929729 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894565-kdmq9" event={"ID":"e94bb7db-1559-4c16-857f-3d20e0a7a008","Type":"ContainerDied","Data":"df35027e69de80bea9fd6290a6b37897bf8057ec4590a27017248fc47bb4994c"} Nov 03 02:45:17.939541 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:17.939517 5605 scope.go:122] "RemoveContainer" containerID="bca2678301e57a59607de50092323ce0b452d6e6d2021257a50626ea8db65799" Nov 03 02:45:17.953136 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:17.953116 5605 scope.go:122] "RemoveContainer" containerID="c36ba1e298bceb4fc382921f6d66af34295c51697da8342394b94d5ac8fb3235" Nov 03 02:45:18.134253 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.134222 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894565-66m7h"] Nov 03 02:45:18.240809 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.240778 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894565-fz2lx"] Nov 03 02:45:18.248551 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.248521 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894565-kdmq9"] Nov 03 02:45:18.252324 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.252293 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894565-kdmq9"] Nov 03 02:45:18.300734 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.300703 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894565-66m7h" Nov 03 02:45:18.358365 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.358263 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe40cb78-4bfe-4975-acbd-867c3366f3ae-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe40cb78-4bfe-4975-acbd-867c3366f3ae-maas-api-service-ca\") pod \"fe40cb78-4bfe-4975-acbd-867c3366f3ae\" (UID: \"fe40cb78-4bfe-4975-acbd-867c3366f3ae\") " Nov 03 02:45:18.358977 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.358530 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe40cb78-4bfe-4975-acbd-867c3366f3ae-kube-api-access-dn2lq\" (UniqueName: \"kubernetes.io/projected/fe40cb78-4bfe-4975-acbd-867c3366f3ae-kube-api-access-dn2lq\") pod \"fe40cb78-4bfe-4975-acbd-867c3366f3ae\" (UID: \"fe40cb78-4bfe-4975-acbd-867c3366f3ae\") " Nov 03 02:45:18.359536 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.359496 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe40cb78-4bfe-4975-acbd-867c3366f3ae-maas-api-service-ca" pod "fe40cb78-4bfe-4975-acbd-867c3366f3ae" (UID: "fe40cb78-4bfe-4975-acbd-867c3366f3ae"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:45:18.361677 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.361649 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe40cb78-4bfe-4975-acbd-867c3366f3ae-kube-api-access-dn2lq" pod "fe40cb78-4bfe-4975-acbd-867c3366f3ae" (UID: "fe40cb78-4bfe-4975-acbd-867c3366f3ae"). InnerVolumeSpecName "kube-api-access-dn2lq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:45:18.423943 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.423912 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7229-8d33da"] Nov 03 02:45:18.427213 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.427163 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894565-fz2lx" Nov 03 02:45:18.427842 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.427820 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7229-8d33da"] Nov 03 02:45:18.459603 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.459562 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dc6be8c-60cf-4429-9ced-90ca7ed49656-kube-api-access-cm586\" (UniqueName: \"kubernetes.io/projected/0dc6be8c-60cf-4429-9ced-90ca7ed49656-kube-api-access-cm586\") pod \"0dc6be8c-60cf-4429-9ced-90ca7ed49656\" (UID: \"0dc6be8c-60cf-4429-9ced-90ca7ed49656\") " Nov 03 02:45:18.459814 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.459792 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dc6be8c-60cf-4429-9ced-90ca7ed49656-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dc6be8c-60cf-4429-9ced-90ca7ed49656-maas-api-service-ca\") pod \"0dc6be8c-60cf-4429-9ced-90ca7ed49656\" (UID: \"0dc6be8c-60cf-4429-9ced-90ca7ed49656\") " Nov 03 02:45:18.460137 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.460113 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dc6be8c-60cf-4429-9ced-90ca7ed49656-maas-api-service-ca" pod "0dc6be8c-60cf-4429-9ced-90ca7ed49656" (UID: "0dc6be8c-60cf-4429-9ced-90ca7ed49656"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:45:18.460277 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.460239 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dn2lq\" (UniqueName: \"kubernetes.io/projected/fe40cb78-4bfe-4975-acbd-867c3366f3ae-kube-api-access-dn2lq\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:18.460373 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.460292 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe40cb78-4bfe-4975-acbd-867c3366f3ae-maas-api-service-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:18.461874 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.461847 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dc6be8c-60cf-4429-9ced-90ca7ed49656-kube-api-access-cm586" pod "0dc6be8c-60cf-4429-9ced-90ca7ed49656" (UID: "0dc6be8c-60cf-4429-9ced-90ca7ed49656"). InnerVolumeSpecName "kube-api-access-cm586". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:45:18.561776 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.561737 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dc6be8c-60cf-4429-9ced-90ca7ed49656-maas-api-service-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:18.561776 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.561770 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm586\" (UniqueName: \"kubernetes.io/projected/0dc6be8c-60cf-4429-9ced-90ca7ed49656-kube-api-access-cm586\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:18.696528 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.696490 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-876aed4a-66qz6"] Nov 03 02:45:18.697137 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.697099 5605 cpu_manager.go:414] "containerMap: removing container" podUID="e94bb7db-1559-4c16-857f-3d20e0a7a008" containerName="cleanup" Nov 03 02:45:18.697137 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.697120 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e94bb7db-1559-4c16-857f-3d20e0a7a008" containerName="cleanup" Nov 03 02:45:18.697137 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.697137 5605 cpu_manager.go:414] "containerMap: removing container" podUID="0dc6be8c-60cf-4429-9ced-90ca7ed49656" containerName="cleanup" Nov 03 02:45:18.697394 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.697145 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dc6be8c-60cf-4429-9ced-90ca7ed49656" containerName="cleanup" Nov 03 02:45:18.697394 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.697179 5605 cpu_manager.go:414] "containerMap: removing container" podUID="fe40cb78-4bfe-4975-acbd-867c3366f3ae" containerName="cleanup" Nov 03 02:45:18.697394 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.697186 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe40cb78-4bfe-4975-acbd-867c3366f3ae" containerName="cleanup" Nov 03 02:45:18.697394 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.697216 5605 cpu_manager.go:414] "containerMap: removing container" podUID="fe40cb78-4bfe-4975-acbd-867c3366f3ae" containerName="cleanup" Nov 03 02:45:18.697394 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.697223 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe40cb78-4bfe-4975-acbd-867c3366f3ae" containerName="cleanup" Nov 03 02:45:18.697394 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.697246 5605 cpu_manager.go:414] "containerMap: removing container" podUID="0dc6be8c-60cf-4429-9ced-90ca7ed49656" containerName="cleanup" Nov 03 02:45:18.697394 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.697253 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dc6be8c-60cf-4429-9ced-90ca7ed49656" containerName="cleanup" Nov 03 02:45:18.697394 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.697267 5605 cpu_manager.go:414] "containerMap: removing container" podUID="e94bb7db-1559-4c16-857f-3d20e0a7a008" containerName="cleanup" Nov 03 02:45:18.697394 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.697274 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e94bb7db-1559-4c16-857f-3d20e0a7a008" containerName="cleanup" Nov 03 02:45:18.697394 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.697287 5605 cpu_manager.go:414] "containerMap: removing container" podUID="2e096621-72ec-4cd2-aeac-1c434feea16d" containerName="test-curl-7229-8d33da" Nov 03 02:45:18.697394 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.697295 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e096621-72ec-4cd2-aeac-1c434feea16d" containerName="test-curl-7229-8d33da" Nov 03 02:45:18.697394 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.697311 5605 cpu_manager.go:414] "containerMap: removing container" podUID="fe40cb78-4bfe-4975-acbd-867c3366f3ae" containerName="cleanup" Nov 03 02:45:18.697394 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.697318 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe40cb78-4bfe-4975-acbd-867c3366f3ae" containerName="cleanup" Nov 03 02:45:18.698041 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.697475 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe40cb78-4bfe-4975-acbd-867c3366f3ae" containerName="cleanup" Nov 03 02:45:18.698041 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.697494 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dc6be8c-60cf-4429-9ced-90ca7ed49656" containerName="cleanup" Nov 03 02:45:18.698041 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.697504 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="e94bb7db-1559-4c16-857f-3d20e0a7a008" containerName="cleanup" Nov 03 02:45:18.698041 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.697516 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="e94bb7db-1559-4c16-857f-3d20e0a7a008" containerName="cleanup" Nov 03 02:45:18.698041 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.697527 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e096621-72ec-4cd2-aeac-1c434feea16d" containerName="test-curl-7229-8d33da" Nov 03 02:45:18.698041 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.697540 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe40cb78-4bfe-4975-acbd-867c3366f3ae" containerName="cleanup" Nov 03 02:45:18.698041 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.697549 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="e94bb7db-1559-4c16-857f-3d20e0a7a008" containerName="cleanup" Nov 03 02:45:18.698041 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.697558 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dc6be8c-60cf-4429-9ced-90ca7ed49656" containerName="cleanup" Nov 03 02:45:18.701211 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.701188 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-876aed4a-66qz6" Nov 03 02:45:18.750268 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.750236 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:45:18.753433 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.753388 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-876aed4a-66qz6"] Nov 03 02:45:18.764529 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.764496 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a07adb9-2325-4532-97d0-00bdd2c8d6f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-876aed4a-66qz6\" (UID: \"6a07adb9-2325-4532-97d0-00bdd2c8d6f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-876aed4a-66qz6" Nov 03 02:45:18.764693 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.764671 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a07adb9-2325-4532-97d0-00bdd2c8d6f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-876aed4a-66qz6\" (UID: \"6a07adb9-2325-4532-97d0-00bdd2c8d6f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-876aed4a-66qz6" Nov 03 02:45:18.764776 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.764734 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6c9n\" (UniqueName: \"kubernetes.io/projected/6a07adb9-2325-4532-97d0-00bdd2c8d6f0-kube-api-access-p6c9n\") pod \"maas-api-revoke-sub-876aed4a-66qz6\" (UID: \"6a07adb9-2325-4532-97d0-00bdd2c8d6f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-876aed4a-66qz6" Nov 03 02:45:18.851062 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.851028 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e94bb7db-1559-4c16-857f-3d20e0a7a008" path="/var/lib/kubelet/pods/e94bb7db-1559-4c16-857f-3d20e0a7a008/volumes" Nov 03 02:45:18.852099 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.852077 5605 scope.go:122] "RemoveContainer" containerID="7e661e2ebada1e85d38e1f3e02c725603734ccfbc35c345767c5dcc1f58691b6" Nov 03 02:45:18.865483 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.865457 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a07adb9-2325-4532-97d0-00bdd2c8d6f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-876aed4a-66qz6\" (UID: \"6a07adb9-2325-4532-97d0-00bdd2c8d6f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-876aed4a-66qz6" Nov 03 02:45:18.865626 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.865501 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6c9n\" (UniqueName: \"kubernetes.io/projected/6a07adb9-2325-4532-97d0-00bdd2c8d6f0-kube-api-access-p6c9n\") pod \"maas-api-revoke-sub-876aed4a-66qz6\" (UID: \"6a07adb9-2325-4532-97d0-00bdd2c8d6f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-876aed4a-66qz6" Nov 03 02:45:18.865626 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.865601 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a07adb9-2325-4532-97d0-00bdd2c8d6f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-876aed4a-66qz6\" (UID: \"6a07adb9-2325-4532-97d0-00bdd2c8d6f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-876aed4a-66qz6" Nov 03 02:45:18.866147 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.866122 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a07adb9-2325-4532-97d0-00bdd2c8d6f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-876aed4a-66qz6\" (UID: \"6a07adb9-2325-4532-97d0-00bdd2c8d6f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-876aed4a-66qz6" Nov 03 02:45:18.868062 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.868042 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a07adb9-2325-4532-97d0-00bdd2c8d6f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-876aed4a-66qz6\" (UID: \"6a07adb9-2325-4532-97d0-00bdd2c8d6f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-876aed4a-66qz6" Nov 03 02:45:18.918945 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.918869 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6c9n\" (UniqueName: \"kubernetes.io/projected/6a07adb9-2325-4532-97d0-00bdd2c8d6f0-kube-api-access-p6c9n\") pod \"maas-api-revoke-sub-876aed4a-66qz6\" (UID: \"6a07adb9-2325-4532-97d0-00bdd2c8d6f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-876aed4a-66qz6" Nov 03 02:45:18.940039 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.940004 5605 generic.go:356] "Generic (PLEG): container finished" podID="6b8b50df-b64b-4249-864d-841839156ae5" containerID="0499499074ad7c01e277705cd094bef71129819bdb8c58f38525d3836d96f03a" exitCode=60 Nov 03 02:45:18.940547 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.940075 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894565-47qxs" event={"ID":"6b8b50df-b64b-4249-864d-841839156ae5","Type":"ContainerDied","Data":"0499499074ad7c01e277705cd094bef71129819bdb8c58f38525d3836d96f03a"} Nov 03 02:45:18.940547 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.940137 5605 scope.go:122] "RemoveContainer" containerID="0c02b3b6b02ac222513614255d74abe97aa2d7fccd3f69e97bd6c8c7092f055a" Nov 03 02:45:18.940629 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.940604 5605 scope.go:122] "RemoveContainer" containerID="0499499074ad7c01e277705cd094bef71129819bdb8c58f38525d3836d96f03a" Nov 03 02:45:18.940952 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:45:18.940926 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894565-47qxs_odh-ai-gateway-infra(6b8b50df-b64b-4249-864d-841839156ae5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894565-47qxs" podUID="6b8b50df-b64b-4249-864d-841839156ae5" Nov 03 02:45:18.942736 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.942712 5605 generic.go:356] "Generic (PLEG): container finished" podID="660b74cc-c09a-48c2-b447-d869e6f3efa2" containerID="175b76fb7f6d18f415b2c28ccd8c8161d773c5bfa4d1fd21eeba5674221f194f" exitCode=60 Nov 03 02:45:18.942843 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.942793 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894565-6wfpx" event={"ID":"660b74cc-c09a-48c2-b447-d869e6f3efa2","Type":"ContainerDied","Data":"175b76fb7f6d18f415b2c28ccd8c8161d773c5bfa4d1fd21eeba5674221f194f"} Nov 03 02:45:18.943255 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.943234 5605 scope.go:122] "RemoveContainer" containerID="175b76fb7f6d18f415b2c28ccd8c8161d773c5bfa4d1fd21eeba5674221f194f" Nov 03 02:45:18.943607 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:45:18.943579 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894565-6wfpx_odh-ai-gateway-infra(660b74cc-c09a-48c2-b447-d869e6f3efa2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894565-6wfpx" podUID="660b74cc-c09a-48c2-b447-d869e6f3efa2" Nov 03 02:45:18.945235 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.945216 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894565-66m7h" Nov 03 02:45:18.945337 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.945245 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894565-66m7h" event={"ID":"fe40cb78-4bfe-4975-acbd-867c3366f3ae","Type":"ContainerDied","Data":"92a5a63648bcbebe22847c16658ea5e1457e3a5e864c48073fbd7a09c4dda24a"} Nov 03 02:45:18.948896 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.948871 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894565-fz2lx" event={"ID":"0dc6be8c-60cf-4429-9ced-90ca7ed49656","Type":"ContainerDied","Data":"d69e188f6b7b72ed9d6c990b02647f40915251db7dc88b6d7bba2240f4379d24"} Nov 03 02:45:18.948973 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.948961 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894565-fz2lx" Nov 03 02:45:18.950998 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.950971 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7229-8d33da" podUID="2e096621-72ec-4cd2-aeac-1c434feea16d" containerName="test-curl-7229-8d33da" containerID="cri-o://40107c4c8de8bff977662746fa58c9fc1389eb6a75c6bed32b0121a8bd04ccd6" gracePeriod=2 Nov 03 02:45:18.958435 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.958395 5605 scope.go:122] "RemoveContainer" containerID="72ebd9217d0b101236d3b5293c3da9c52f2391b1f0370679183dd2581f61d7a4" Nov 03 02:45:18.973666 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.973638 5605 scope.go:122] "RemoveContainer" containerID="098ffbfd37ab052cbb015ebaad9d5ffc4dd13a69588157859ff9a87706ec71e1" Nov 03 02:45:18.986734 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:18.986707 5605 scope.go:122] "RemoveContainer" containerID="df2b472892eaf0b9e698411e405db52fbc8bb3be37e26fe5f175586149b19806" Nov 03 02:45:19.015175 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:19.015138 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-876aed4a-66qz6" Nov 03 02:45:19.042184 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:19.042142 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894565-g2zcb"] Nov 03 02:45:19.148915 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:19.148858 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894565-lvznm"] Nov 03 02:45:19.160936 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:19.160876 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894565-fz2lx"] Nov 03 02:45:19.173206 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:19.173127 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894565-fz2lx"] Nov 03 02:45:19.245508 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:19.245477 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894565-g2zcb" Nov 03 02:45:19.274232 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:19.273459 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894565-66m7h"] Nov 03 02:45:19.276977 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:19.276936 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894565-66m7h"] Nov 03 02:45:19.300979 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:19.300953 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894565-lvznm" Nov 03 02:45:19.372317 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:19.372294 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aeb1d4c3-d1f5-49f4-9e3c-47ff2b640ac9-kube-api-access-sb82f\" (UniqueName: \"kubernetes.io/projected/aeb1d4c3-d1f5-49f4-9e3c-47ff2b640ac9-kube-api-access-sb82f\") pod \"aeb1d4c3-d1f5-49f4-9e3c-47ff2b640ac9\" (UID: \"aeb1d4c3-d1f5-49f4-9e3c-47ff2b640ac9\") " Nov 03 02:45:19.372481 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:19.372347 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aeb1d4c3-d1f5-49f4-9e3c-47ff2b640ac9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aeb1d4c3-d1f5-49f4-9e3c-47ff2b640ac9-maas-api-service-ca\") pod \"aeb1d4c3-d1f5-49f4-9e3c-47ff2b640ac9\" (UID: \"aeb1d4c3-d1f5-49f4-9e3c-47ff2b640ac9\") " Nov 03 02:45:19.372560 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:19.372542 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed96e13a-3942-46e6-846f-027f5d1718cc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed96e13a-3942-46e6-846f-027f5d1718cc-maas-api-service-ca\") pod \"ed96e13a-3942-46e6-846f-027f5d1718cc\" (UID: \"ed96e13a-3942-46e6-846f-027f5d1718cc\") " Nov 03 02:45:19.372614 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:19.372603 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed96e13a-3942-46e6-846f-027f5d1718cc-kube-api-access-tt8sh\" (UniqueName: \"kubernetes.io/projected/ed96e13a-3942-46e6-846f-027f5d1718cc-kube-api-access-tt8sh\") pod \"ed96e13a-3942-46e6-846f-027f5d1718cc\" (UID: \"ed96e13a-3942-46e6-846f-027f5d1718cc\") " Nov 03 02:45:19.372976 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:19.372691 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aeb1d4c3-d1f5-49f4-9e3c-47ff2b640ac9-maas-api-service-ca" pod "aeb1d4c3-d1f5-49f4-9e3c-47ff2b640ac9" (UID: "aeb1d4c3-d1f5-49f4-9e3c-47ff2b640ac9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:45:19.372976 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:19.372919 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed96e13a-3942-46e6-846f-027f5d1718cc-maas-api-service-ca" pod "ed96e13a-3942-46e6-846f-027f5d1718cc" (UID: "ed96e13a-3942-46e6-846f-027f5d1718cc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:45:19.373142 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:19.373067 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed96e13a-3942-46e6-846f-027f5d1718cc-maas-api-service-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:19.373142 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:19.373088 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aeb1d4c3-d1f5-49f4-9e3c-47ff2b640ac9-maas-api-service-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:19.374673 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:19.374654 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed96e13a-3942-46e6-846f-027f5d1718cc-kube-api-access-tt8sh" pod "ed96e13a-3942-46e6-846f-027f5d1718cc" (UID: "ed96e13a-3942-46e6-846f-027f5d1718cc"). InnerVolumeSpecName "kube-api-access-tt8sh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:45:19.374839 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:19.374815 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeb1d4c3-d1f5-49f4-9e3c-47ff2b640ac9-kube-api-access-sb82f" pod "aeb1d4c3-d1f5-49f4-9e3c-47ff2b640ac9" (UID: "aeb1d4c3-d1f5-49f4-9e3c-47ff2b640ac9"). InnerVolumeSpecName "kube-api-access-sb82f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:45:19.396576 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:19.396538 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-876aed4a-66qz6"] Nov 03 02:45:19.402542 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:45:19.402505 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a07adb9_2325_4532_97d0_00bdd2c8d6f0.slice/crio-70c086c1e51297bb6139913d31ffa9a54f4338c7b24c4365d1d45ec165306759 WatchSource:0}: Error finding container 70c086c1e51297bb6139913d31ffa9a54f4338c7b24c4365d1d45ec165306759: Status 404 returned error can't find the container with id 70c086c1e51297bb6139913d31ffa9a54f4338c7b24c4365d1d45ec165306759 Nov 03 02:45:19.474908 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:19.474867 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt8sh\" (UniqueName: \"kubernetes.io/projected/ed96e13a-3942-46e6-846f-027f5d1718cc-kube-api-access-tt8sh\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:19.474908 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:19.474910 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sb82f\" (UniqueName: \"kubernetes.io/projected/aeb1d4c3-d1f5-49f4-9e3c-47ff2b640ac9-kube-api-access-sb82f\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:19.959785 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:19.959757 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894565-g2zcb" Nov 03 02:45:19.960268 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:19.959797 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894565-g2zcb" event={"ID":"aeb1d4c3-d1f5-49f4-9e3c-47ff2b640ac9","Type":"ContainerDied","Data":"46536676785b1461233e7b99677f9ec73c3487672c841c4fda7667ac85d73a0c"} Nov 03 02:45:19.960268 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:19.959862 5605 scope.go:122] "RemoveContainer" containerID="9f1de971624a88ab08901e6748fa066b9b7feac486f4e00fcf99a1b7752f7ea7" Nov 03 02:45:19.962324 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:19.962304 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894565-lvznm" Nov 03 02:45:19.962452 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:19.962335 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894565-lvznm" event={"ID":"ed96e13a-3942-46e6-846f-027f5d1718cc","Type":"ContainerDied","Data":"4eedb02384d1d3e75f8b5cfab773881df7ec995f13bc723163534877dfc47275"} Nov 03 02:45:19.964021 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:19.963995 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-876aed4a-66qz6" event={"ID":"6a07adb9-2325-4532-97d0-00bdd2c8d6f0","Type":"ContainerStarted","Data":"d003b2e0b5f4ffed049635adc90a9b3dfe4aa4a5ef90505f79ed1b91c682b896"} Nov 03 02:45:19.964152 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:19.964050 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-876aed4a-66qz6" event={"ID":"6a07adb9-2325-4532-97d0-00bdd2c8d6f0","Type":"ContainerStarted","Data":"70c086c1e51297bb6139913d31ffa9a54f4338c7b24c4365d1d45ec165306759"} Nov 03 02:45:19.971823 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:19.971797 5605 generic.go:356] "Generic (PLEG): container finished" podID="305b0cb8-2348-4517-86a5-f77bfe6a3814" containerID="b3338ec469ad3f5367068d87a86a41212ec461a3119d2ca8e8c556db3d93a8b8" exitCode=60 Nov 03 02:45:19.971958 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:19.971833 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894565-tqf6b" event={"ID":"305b0cb8-2348-4517-86a5-f77bfe6a3814","Type":"ContainerDied","Data":"b3338ec469ad3f5367068d87a86a41212ec461a3119d2ca8e8c556db3d93a8b8"} Nov 03 02:45:19.972187 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:19.972172 5605 scope.go:122] "RemoveContainer" containerID="b3338ec469ad3f5367068d87a86a41212ec461a3119d2ca8e8c556db3d93a8b8" Nov 03 02:45:19.972793 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:45:19.972488 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-521f1f-29894565-tqf6b_odh-ai-gateway-infra(305b0cb8-2348-4517-86a5-f77bfe6a3814)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894565-tqf6b" podUID="305b0cb8-2348-4517-86a5-f77bfe6a3814" Nov 03 02:45:19.972953 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:19.972827 5605 scope.go:122] "RemoveContainer" containerID="6e0e7015a3bd59d73dd0ce6b123398da4594317e82d7d3f331967c4e7dac3870" Nov 03 02:45:19.982860 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:19.982836 5605 scope.go:122] "RemoveContainer" containerID="7e661e2ebada1e85d38e1f3e02c725603734ccfbc35c345767c5dcc1f58691b6" Nov 03 02:45:20.059639 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:20.059608 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894565-47qxs"] Nov 03 02:45:20.202128 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:20.202102 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894565-47qxs" Nov 03 02:45:20.224230 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:20.224197 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894565-lvznm"] Nov 03 02:45:20.230064 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:20.230036 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894565-lvznm"] Nov 03 02:45:20.284307 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:20.284272 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b8b50df-b64b-4249-864d-841839156ae5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b8b50df-b64b-4249-864d-841839156ae5-maas-api-service-ca\") pod \"6b8b50df-b64b-4249-864d-841839156ae5\" (UID: \"6b8b50df-b64b-4249-864d-841839156ae5\") " Nov 03 02:45:20.284534 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:20.284350 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b8b50df-b64b-4249-864d-841839156ae5-kube-api-access-jp4cn\" (UniqueName: \"kubernetes.io/projected/6b8b50df-b64b-4249-864d-841839156ae5-kube-api-access-jp4cn\") pod \"6b8b50df-b64b-4249-864d-841839156ae5\" (UID: \"6b8b50df-b64b-4249-864d-841839156ae5\") " Nov 03 02:45:20.284737 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:20.284710 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b8b50df-b64b-4249-864d-841839156ae5-maas-api-service-ca" pod "6b8b50df-b64b-4249-864d-841839156ae5" (UID: "6b8b50df-b64b-4249-864d-841839156ae5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:45:20.284889 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:20.284875 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b8b50df-b64b-4249-864d-841839156ae5-maas-api-service-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:20.286581 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:20.286561 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b8b50df-b64b-4249-864d-841839156ae5-kube-api-access-jp4cn" pod "6b8b50df-b64b-4249-864d-841839156ae5" (UID: "6b8b50df-b64b-4249-864d-841839156ae5"). InnerVolumeSpecName "kube-api-access-jp4cn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:45:20.331680 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:20.331644 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894565-g2zcb"] Nov 03 02:45:20.335687 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:20.335554 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894565-g2zcb"] Nov 03 02:45:20.386425 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:20.386308 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jp4cn\" (UniqueName: \"kubernetes.io/projected/6b8b50df-b64b-4249-864d-841839156ae5-kube-api-access-jp4cn\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:20.393876 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:20.392489 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894565-6wfpx"] Nov 03 02:45:20.442903 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:20.442851 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-876aed4a-66qz6" podStartSLOduration=2.442836899 podStartE2EDuration="2.442836899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:45:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:45:20.439260427 +0000 UTC m=+3197.698458582" watchObservedRunningTime="2026-11-03 02:45:20.442836899 +0000 UTC m=+3197.702035052" Nov 03 02:45:20.521602 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:20.521574 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894565-6wfpx" Nov 03 02:45:20.589088 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:20.589047 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/660b74cc-c09a-48c2-b447-d869e6f3efa2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/660b74cc-c09a-48c2-b447-d869e6f3efa2-maas-api-service-ca\") pod \"660b74cc-c09a-48c2-b447-d869e6f3efa2\" (UID: \"660b74cc-c09a-48c2-b447-d869e6f3efa2\") " Nov 03 02:45:20.589244 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:20.589196 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/660b74cc-c09a-48c2-b447-d869e6f3efa2-kube-api-access-wxb9h\" (UniqueName: \"kubernetes.io/projected/660b74cc-c09a-48c2-b447-d869e6f3efa2-kube-api-access-wxb9h\") pod \"660b74cc-c09a-48c2-b447-d869e6f3efa2\" (UID: \"660b74cc-c09a-48c2-b447-d869e6f3efa2\") " Nov 03 02:45:20.589425 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:20.589391 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/660b74cc-c09a-48c2-b447-d869e6f3efa2-maas-api-service-ca" pod "660b74cc-c09a-48c2-b447-d869e6f3efa2" (UID: "660b74cc-c09a-48c2-b447-d869e6f3efa2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:45:20.589673 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:20.589655 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/660b74cc-c09a-48c2-b447-d869e6f3efa2-maas-api-service-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:20.591332 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:20.591310 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/660b74cc-c09a-48c2-b447-d869e6f3efa2-kube-api-access-wxb9h" pod "660b74cc-c09a-48c2-b447-d869e6f3efa2" (UID: "660b74cc-c09a-48c2-b447-d869e6f3efa2"). InnerVolumeSpecName "kube-api-access-wxb9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:45:20.690575 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:20.690484 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxb9h\" (UniqueName: \"kubernetes.io/projected/660b74cc-c09a-48c2-b447-d869e6f3efa2-kube-api-access-wxb9h\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:20.850090 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:20.850055 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dc6be8c-60cf-4429-9ced-90ca7ed49656" path="/var/lib/kubelet/pods/0dc6be8c-60cf-4429-9ced-90ca7ed49656/volumes" Nov 03 02:45:20.850525 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:20.850512 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aeb1d4c3-d1f5-49f4-9e3c-47ff2b640ac9" path="/var/lib/kubelet/pods/aeb1d4c3-d1f5-49f4-9e3c-47ff2b640ac9/volumes" Nov 03 02:45:20.850882 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:20.850870 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed96e13a-3942-46e6-846f-027f5d1718cc" path="/var/lib/kubelet/pods/ed96e13a-3942-46e6-846f-027f5d1718cc/volumes" Nov 03 02:45:20.851562 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:20.851547 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe40cb78-4bfe-4975-acbd-867c3366f3ae" path="/var/lib/kubelet/pods/fe40cb78-4bfe-4975-acbd-867c3366f3ae/volumes" Nov 03 02:45:20.983001 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:20.982968 5605 generic.go:356] "Generic (PLEG): container finished" podID="6a07adb9-2325-4532-97d0-00bdd2c8d6f0" containerID="d003b2e0b5f4ffed049635adc90a9b3dfe4aa4a5ef90505f79ed1b91c682b896" exitCode=0 Nov 03 02:45:20.983422 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:20.983051 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-876aed4a-66qz6" event={"ID":"6a07adb9-2325-4532-97d0-00bdd2c8d6f0","Type":"ContainerDied","Data":"d003b2e0b5f4ffed049635adc90a9b3dfe4aa4a5ef90505f79ed1b91c682b896"} Nov 03 02:45:20.986815 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:20.984464 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894565-47qxs" event={"ID":"6b8b50df-b64b-4249-864d-841839156ae5","Type":"ContainerDied","Data":"13aaedb2064f55195c263df9063dce8c02679233aacfa0ae48c3263b9984b407"} Nov 03 02:45:20.986815 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:20.984515 5605 scope.go:122] "RemoveContainer" containerID="0499499074ad7c01e277705cd094bef71129819bdb8c58f38525d3836d96f03a" Nov 03 02:45:20.986815 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:20.984589 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894565-47qxs" Nov 03 02:45:20.986815 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:20.986056 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894565-6wfpx" event={"ID":"660b74cc-c09a-48c2-b447-d869e6f3efa2","Type":"ContainerDied","Data":"37dac07c2a91a5cef26b0ab3f4503bd3890d4a1296713e589ae6f38ff1329846"} Nov 03 02:45:20.986815 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:20.986118 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894565-6wfpx" Nov 03 02:45:21.012975 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:21.012951 5605 scope.go:122] "RemoveContainer" containerID="175b76fb7f6d18f415b2c28ccd8c8161d773c5bfa4d1fd21eeba5674221f194f" Nov 03 02:45:21.128998 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:21.128972 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894565-tqf6b"] Nov 03 02:45:21.213991 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:21.213386 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894565-6wfpx"] Nov 03 02:45:21.213991 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:21.213443 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894565-6wfpx"] Nov 03 02:45:21.253182 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:21.253152 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7229-8d33da" Nov 03 02:45:21.309345 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:21.309316 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894565-47qxs"] Nov 03 02:45:21.311286 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:21.311264 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894565-tqf6b" Nov 03 02:45:21.311737 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:21.311718 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894565-47qxs"] Nov 03 02:45:21.416223 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:21.416191 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/305b0cb8-2348-4517-86a5-f77bfe6a3814-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/305b0cb8-2348-4517-86a5-f77bfe6a3814-maas-api-service-ca\") pod \"305b0cb8-2348-4517-86a5-f77bfe6a3814\" (UID: \"305b0cb8-2348-4517-86a5-f77bfe6a3814\") " Nov 03 02:45:21.416414 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:21.416290 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e096621-72ec-4cd2-aeac-1c434feea16d-kube-api-access-nbvsj\" (UniqueName: \"kubernetes.io/projected/2e096621-72ec-4cd2-aeac-1c434feea16d-kube-api-access-nbvsj\") pod \"2e096621-72ec-4cd2-aeac-1c434feea16d\" (UID: \"2e096621-72ec-4cd2-aeac-1c434feea16d\") " Nov 03 02:45:21.416481 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:21.416429 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/305b0cb8-2348-4517-86a5-f77bfe6a3814-kube-api-access-57mqg\" (UniqueName: \"kubernetes.io/projected/305b0cb8-2348-4517-86a5-f77bfe6a3814-kube-api-access-57mqg\") pod \"305b0cb8-2348-4517-86a5-f77bfe6a3814\" (UID: \"305b0cb8-2348-4517-86a5-f77bfe6a3814\") " Nov 03 02:45:21.416613 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:21.416576 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/305b0cb8-2348-4517-86a5-f77bfe6a3814-maas-api-service-ca" pod "305b0cb8-2348-4517-86a5-f77bfe6a3814" (UID: "305b0cb8-2348-4517-86a5-f77bfe6a3814"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:45:21.416866 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:21.416848 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/305b0cb8-2348-4517-86a5-f77bfe6a3814-maas-api-service-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:21.418596 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:21.418572 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/305b0cb8-2348-4517-86a5-f77bfe6a3814-kube-api-access-57mqg" pod "305b0cb8-2348-4517-86a5-f77bfe6a3814" (UID: "305b0cb8-2348-4517-86a5-f77bfe6a3814"). InnerVolumeSpecName "kube-api-access-57mqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:45:21.418692 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:21.418590 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e096621-72ec-4cd2-aeac-1c434feea16d-kube-api-access-nbvsj" pod "2e096621-72ec-4cd2-aeac-1c434feea16d" (UID: "2e096621-72ec-4cd2-aeac-1c434feea16d"). InnerVolumeSpecName "kube-api-access-nbvsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:45:21.518428 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:21.518331 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbvsj\" (UniqueName: \"kubernetes.io/projected/2e096621-72ec-4cd2-aeac-1c434feea16d-kube-api-access-nbvsj\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:21.518428 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:21.518379 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57mqg\" (UniqueName: \"kubernetes.io/projected/305b0cb8-2348-4517-86a5-f77bfe6a3814-kube-api-access-57mqg\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:21.995060 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:21.995025 5605 generic.go:356] "Generic (PLEG): container finished" podID="2e096621-72ec-4cd2-aeac-1c434feea16d" containerID="40107c4c8de8bff977662746fa58c9fc1389eb6a75c6bed32b0121a8bd04ccd6" exitCode=137 Nov 03 02:45:21.995562 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:21.995067 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7229-8d33da" Nov 03 02:45:21.995562 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:21.995094 5605 scope.go:122] "RemoveContainer" containerID="40107c4c8de8bff977662746fa58c9fc1389eb6a75c6bed32b0121a8bd04ccd6" Nov 03 02:45:21.996499 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:21.996469 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894565-tqf6b" event={"ID":"305b0cb8-2348-4517-86a5-f77bfe6a3814","Type":"ContainerDied","Data":"6ff257238610e0b194e47cdcfc1a3cb56160b0702fb599f4210e807cf8b10853"} Nov 03 02:45:21.996499 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:21.996489 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894565-tqf6b" Nov 03 02:45:22.006132 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:22.006114 5605 scope.go:122] "RemoveContainer" containerID="40107c4c8de8bff977662746fa58c9fc1389eb6a75c6bed32b0121a8bd04ccd6" Nov 03 02:45:22.006576 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:45:22.006542 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40107c4c8de8bff977662746fa58c9fc1389eb6a75c6bed32b0121a8bd04ccd6\": container with ID starting with 40107c4c8de8bff977662746fa58c9fc1389eb6a75c6bed32b0121a8bd04ccd6 not found: ID does not exist" containerID="40107c4c8de8bff977662746fa58c9fc1389eb6a75c6bed32b0121a8bd04ccd6" Nov 03 02:45:22.006704 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:22.006584 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40107c4c8de8bff977662746fa58c9fc1389eb6a75c6bed32b0121a8bd04ccd6"} err="failed to get container status \"40107c4c8de8bff977662746fa58c9fc1389eb6a75c6bed32b0121a8bd04ccd6\": rpc error: code = NotFound desc = could not find container \"40107c4c8de8bff977662746fa58c9fc1389eb6a75c6bed32b0121a8bd04ccd6\": container with ID starting with 40107c4c8de8bff977662746fa58c9fc1389eb6a75c6bed32b0121a8bd04ccd6 not found: ID does not exist" Nov 03 02:45:22.006704 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:22.006608 5605 scope.go:122] "RemoveContainer" containerID="b3338ec469ad3f5367068d87a86a41212ec461a3119d2ca8e8c556db3d93a8b8" Nov 03 02:45:22.094320 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:45:22.094282 5605 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7229-8d33da\" is forbidden: User \"system:node:ip-10-0-140-140.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-140.us-west-2.compute.internal' and this object" podUID="2e096621-72ec-4cd2-aeac-1c434feea16d" pod="opendatahub/test-curl-7229-8d33da" Nov 03 02:45:22.141610 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:22.141579 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-876aed4a-66qz6" Nov 03 02:45:22.199883 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:22.199793 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894565-tqf6b"] Nov 03 02:45:22.202872 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:22.202845 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894565-tqf6b"] Nov 03 02:45:22.226797 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:22.226765 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a07adb9-2325-4532-97d0-00bdd2c8d6f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a07adb9-2325-4532-97d0-00bdd2c8d6f0-maas-api-openshift-ca\") pod \"6a07adb9-2325-4532-97d0-00bdd2c8d6f0\" (UID: \"6a07adb9-2325-4532-97d0-00bdd2c8d6f0\") " Nov 03 02:45:22.226926 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:22.226834 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a07adb9-2325-4532-97d0-00bdd2c8d6f0-kube-api-access-p6c9n\" (UniqueName: \"kubernetes.io/projected/6a07adb9-2325-4532-97d0-00bdd2c8d6f0-kube-api-access-p6c9n\") pod \"6a07adb9-2325-4532-97d0-00bdd2c8d6f0\" (UID: \"6a07adb9-2325-4532-97d0-00bdd2c8d6f0\") " Nov 03 02:45:22.226926 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:22.226861 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a07adb9-2325-4532-97d0-00bdd2c8d6f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a07adb9-2325-4532-97d0-00bdd2c8d6f0-maas-api-xks-ca\") pod \"6a07adb9-2325-4532-97d0-00bdd2c8d6f0\" (UID: \"6a07adb9-2325-4532-97d0-00bdd2c8d6f0\") " Nov 03 02:45:22.227227 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:22.227196 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a07adb9-2325-4532-97d0-00bdd2c8d6f0-maas-api-openshift-ca" pod "6a07adb9-2325-4532-97d0-00bdd2c8d6f0" (UID: "6a07adb9-2325-4532-97d0-00bdd2c8d6f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:45:22.229051 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:22.229025 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a07adb9-2325-4532-97d0-00bdd2c8d6f0-maas-api-xks-ca" pod "6a07adb9-2325-4532-97d0-00bdd2c8d6f0" (UID: "6a07adb9-2325-4532-97d0-00bdd2c8d6f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:45:22.229051 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:22.229036 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a07adb9-2325-4532-97d0-00bdd2c8d6f0-kube-api-access-p6c9n" pod "6a07adb9-2325-4532-97d0-00bdd2c8d6f0" (UID: "6a07adb9-2325-4532-97d0-00bdd2c8d6f0"). InnerVolumeSpecName "kube-api-access-p6c9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:45:22.327898 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:22.327858 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6c9n\" (UniqueName: \"kubernetes.io/projected/6a07adb9-2325-4532-97d0-00bdd2c8d6f0-kube-api-access-p6c9n\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:22.327898 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:22.327891 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a07adb9-2325-4532-97d0-00bdd2c8d6f0-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:22.327898 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:22.327902 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a07adb9-2325-4532-97d0-00bdd2c8d6f0-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:22.354617 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:45:22.354581 5605 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7229-8d33da\" is forbidden: User \"system:node:ip-10-0-140-140.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-140.us-west-2.compute.internal' and this object" podUID="2e096621-72ec-4cd2-aeac-1c434feea16d" pod="opendatahub/test-curl-7229-8d33da" Nov 03 02:45:22.850022 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:22.849986 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e096621-72ec-4cd2-aeac-1c434feea16d" path="/var/lib/kubelet/pods/2e096621-72ec-4cd2-aeac-1c434feea16d/volumes" Nov 03 02:45:22.850521 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:22.850501 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="305b0cb8-2348-4517-86a5-f77bfe6a3814" path="/var/lib/kubelet/pods/305b0cb8-2348-4517-86a5-f77bfe6a3814/volumes" Nov 03 02:45:22.850941 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:22.850923 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="660b74cc-c09a-48c2-b447-d869e6f3efa2" path="/var/lib/kubelet/pods/660b74cc-c09a-48c2-b447-d869e6f3efa2/volumes" Nov 03 02:45:22.851500 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:22.851477 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b8b50df-b64b-4249-864d-841839156ae5" path="/var/lib/kubelet/pods/6b8b50df-b64b-4249-864d-841839156ae5/volumes" Nov 03 02:45:22.893613 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:45:22.893575 5605 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7229-8d33da\" is forbidden: User \"system:node:ip-10-0-140-140.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-140.us-west-2.compute.internal' and this object" podUID="2e096621-72ec-4cd2-aeac-1c434feea16d" pod="opendatahub/test-curl-7229-8d33da" Nov 03 02:45:23.005125 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:23.005098 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-876aed4a-66qz6" Nov 03 02:45:23.005125 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:23.005121 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-876aed4a-66qz6" event={"ID":"6a07adb9-2325-4532-97d0-00bdd2c8d6f0","Type":"ContainerDied","Data":"70c086c1e51297bb6139913d31ffa9a54f4338c7b24c4365d1d45ec165306759"} Nov 03 02:45:23.005615 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:23.005168 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70c086c1e51297bb6139913d31ffa9a54f4338c7b24c4365d1d45ec165306759" Nov 03 02:45:23.919597 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:45:23.919554 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:45:27.370175 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:27.370108 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-f735f4/auth-test-model-f735f4-kserve-c545848fc-glzwl" podUID="757daf90-e81c-4b1d-86ff-c6a6afb9d4e4" containerName="main" containerID="cri-o://4d1c87c8aae1c676743b90e006d7fd96f24bf038c0da1c90cd2a3571191ecdda" gracePeriod=45 Nov 03 02:45:27.638896 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:27.638818 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-f735f4/auth-test-model-f735f4-kserve-c545848fc-glzwl" Nov 03 02:45:27.791008 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:27.790972 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/757daf90-e81c-4b1d-86ff-c6a6afb9d4e4-tls-certs\" (UniqueName: \"kubernetes.io/secret/757daf90-e81c-4b1d-86ff-c6a6afb9d4e4-tls-certs\") pod \"757daf90-e81c-4b1d-86ff-c6a6afb9d4e4\" (UID: \"757daf90-e81c-4b1d-86ff-c6a6afb9d4e4\") " Nov 03 02:45:27.791008 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:27.791013 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/757daf90-e81c-4b1d-86ff-c6a6afb9d4e4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/757daf90-e81c-4b1d-86ff-c6a6afb9d4e4-dshm\") pod \"757daf90-e81c-4b1d-86ff-c6a6afb9d4e4\" (UID: \"757daf90-e81c-4b1d-86ff-c6a6afb9d4e4\") " Nov 03 02:45:27.791237 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:27.791035 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/757daf90-e81c-4b1d-86ff-c6a6afb9d4e4-kube-api-access-69rwr\" (UniqueName: \"kubernetes.io/projected/757daf90-e81c-4b1d-86ff-c6a6afb9d4e4-kube-api-access-69rwr\") pod \"757daf90-e81c-4b1d-86ff-c6a6afb9d4e4\" (UID: \"757daf90-e81c-4b1d-86ff-c6a6afb9d4e4\") " Nov 03 02:45:27.791237 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:27.791145 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/757daf90-e81c-4b1d-86ff-c6a6afb9d4e4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/757daf90-e81c-4b1d-86ff-c6a6afb9d4e4-tmp-dir\") pod \"757daf90-e81c-4b1d-86ff-c6a6afb9d4e4\" (UID: \"757daf90-e81c-4b1d-86ff-c6a6afb9d4e4\") " Nov 03 02:45:27.791237 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:27.791193 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/757daf90-e81c-4b1d-86ff-c6a6afb9d4e4-home\" (UniqueName: \"kubernetes.io/empty-dir/757daf90-e81c-4b1d-86ff-c6a6afb9d4e4-home\") pod \"757daf90-e81c-4b1d-86ff-c6a6afb9d4e4\" (UID: \"757daf90-e81c-4b1d-86ff-c6a6afb9d4e4\") " Nov 03 02:45:27.791409 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:27.791236 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/757daf90-e81c-4b1d-86ff-c6a6afb9d4e4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/757daf90-e81c-4b1d-86ff-c6a6afb9d4e4-model-cache\") pod \"757daf90-e81c-4b1d-86ff-c6a6afb9d4e4\" (UID: \"757daf90-e81c-4b1d-86ff-c6a6afb9d4e4\") " Nov 03 02:45:27.791481 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:27.791451 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/757daf90-e81c-4b1d-86ff-c6a6afb9d4e4-tmp-dir" pod "757daf90-e81c-4b1d-86ff-c6a6afb9d4e4" (UID: "757daf90-e81c-4b1d-86ff-c6a6afb9d4e4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:45:27.791551 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:27.791493 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/757daf90-e81c-4b1d-86ff-c6a6afb9d4e4-home" pod "757daf90-e81c-4b1d-86ff-c6a6afb9d4e4" (UID: "757daf90-e81c-4b1d-86ff-c6a6afb9d4e4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:45:27.791710 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:27.791676 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/757daf90-e81c-4b1d-86ff-c6a6afb9d4e4-model-cache" pod "757daf90-e81c-4b1d-86ff-c6a6afb9d4e4" (UID: "757daf90-e81c-4b1d-86ff-c6a6afb9d4e4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:45:27.791911 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:27.791889 5605 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/757daf90-e81c-4b1d-86ff-c6a6afb9d4e4-tmp-dir\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:27.791985 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:27.791914 5605 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/757daf90-e81c-4b1d-86ff-c6a6afb9d4e4-home\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:27.791985 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:27.791928 5605 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/757daf90-e81c-4b1d-86ff-c6a6afb9d4e4-model-cache\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:27.793499 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:27.793474 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/757daf90-e81c-4b1d-86ff-c6a6afb9d4e4-tls-certs" pod "757daf90-e81c-4b1d-86ff-c6a6afb9d4e4" (UID: "757daf90-e81c-4b1d-86ff-c6a6afb9d4e4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:45:27.793575 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:27.793511 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/757daf90-e81c-4b1d-86ff-c6a6afb9d4e4-dshm" pod "757daf90-e81c-4b1d-86ff-c6a6afb9d4e4" (UID: "757daf90-e81c-4b1d-86ff-c6a6afb9d4e4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:45:27.793625 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:27.793568 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/757daf90-e81c-4b1d-86ff-c6a6afb9d4e4-kube-api-access-69rwr" pod "757daf90-e81c-4b1d-86ff-c6a6afb9d4e4" (UID: "757daf90-e81c-4b1d-86ff-c6a6afb9d4e4"). InnerVolumeSpecName "kube-api-access-69rwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:45:27.893396 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:27.893260 5605 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/757daf90-e81c-4b1d-86ff-c6a6afb9d4e4-tls-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:27.893396 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:27.893295 5605 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/757daf90-e81c-4b1d-86ff-c6a6afb9d4e4-dshm\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:27.893396 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:27.893307 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69rwr\" (UniqueName: \"kubernetes.io/projected/757daf90-e81c-4b1d-86ff-c6a6afb9d4e4-kube-api-access-69rwr\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:28.041654 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:28.041616 5605 generic.go:356] "Generic (PLEG): container finished" podID="757daf90-e81c-4b1d-86ff-c6a6afb9d4e4" containerID="4d1c87c8aae1c676743b90e006d7fd96f24bf038c0da1c90cd2a3571191ecdda" exitCode=0 Nov 03 02:45:28.041845 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:28.041667 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-f735f4/auth-test-model-f735f4-kserve-c545848fc-glzwl" Nov 03 02:45:28.041845 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:28.041673 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-f735f4/auth-test-model-f735f4-kserve-c545848fc-glzwl" event={"ID":"757daf90-e81c-4b1d-86ff-c6a6afb9d4e4","Type":"ContainerDied","Data":"4d1c87c8aae1c676743b90e006d7fd96f24bf038c0da1c90cd2a3571191ecdda"} Nov 03 02:45:28.041845 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:28.041714 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-f735f4/auth-test-model-f735f4-kserve-c545848fc-glzwl" event={"ID":"757daf90-e81c-4b1d-86ff-c6a6afb9d4e4","Type":"ContainerDied","Data":"c0b3ceb91705244058b117f6aba4f597f0552770ecd085034be521b90b0cf178"} Nov 03 02:45:28.041845 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:28.041736 5605 scope.go:122] "RemoveContainer" containerID="4d1c87c8aae1c676743b90e006d7fd96f24bf038c0da1c90cd2a3571191ecdda" Nov 03 02:45:28.061862 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:28.061838 5605 scope.go:122] "RemoveContainer" containerID="4d1c87c8aae1c676743b90e006d7fd96f24bf038c0da1c90cd2a3571191ecdda" Nov 03 02:45:28.062181 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:45:28.062161 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d1c87c8aae1c676743b90e006d7fd96f24bf038c0da1c90cd2a3571191ecdda\": container with ID starting with 4d1c87c8aae1c676743b90e006d7fd96f24bf038c0da1c90cd2a3571191ecdda not found: ID does not exist" containerID="4d1c87c8aae1c676743b90e006d7fd96f24bf038c0da1c90cd2a3571191ecdda" Nov 03 02:45:28.062258 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:28.062189 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d1c87c8aae1c676743b90e006d7fd96f24bf038c0da1c90cd2a3571191ecdda"} err="failed to get container status \"4d1c87c8aae1c676743b90e006d7fd96f24bf038c0da1c90cd2a3571191ecdda\": rpc error: code = NotFound desc = could not find container \"4d1c87c8aae1c676743b90e006d7fd96f24bf038c0da1c90cd2a3571191ecdda\": container with ID starting with 4d1c87c8aae1c676743b90e006d7fd96f24bf038c0da1c90cd2a3571191ecdda not found: ID does not exist" Nov 03 02:45:28.204652 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:28.204559 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-f735f4/auth-test-model-f735f4-kserve-c545848fc-glzwl"] Nov 03 02:45:28.219404 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:28.219372 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-f735f4/auth-test-model-f735f4-kserve-c545848fc-glzwl"] Nov 03 02:45:28.593483 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:28.590234 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-f735f4/auth-test-model-f735f4-kserve-c545848fc-glzwl" podUID="757daf90-e81c-4b1d-86ff-c6a6afb9d4e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.154:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:45:28.594007 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:28.593447 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-957b33/auth-test-model-957b33-kserve-c8778cdbf-dlqkf"] Nov 03 02:45:28.854764 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:28.854681 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="757daf90-e81c-4b1d-86ff-c6a6afb9d4e4" path="/var/lib/kubelet/pods/757daf90-e81c-4b1d-86ff-c6a6afb9d4e4/volumes" Nov 03 02:45:29.437491 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.437460 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d2c4664-tjjzg"] Nov 03 02:45:29.437908 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.437896 5605 cpu_manager.go:414] "containerMap: removing container" podUID="660b74cc-c09a-48c2-b447-d869e6f3efa2" containerName="cleanup" Nov 03 02:45:29.437961 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.437910 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="660b74cc-c09a-48c2-b447-d869e6f3efa2" containerName="cleanup" Nov 03 02:45:29.437961 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.437920 5605 cpu_manager.go:414] "containerMap: removing container" podUID="6b8b50df-b64b-4249-864d-841839156ae5" containerName="cleanup" Nov 03 02:45:29.437961 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.437925 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b8b50df-b64b-4249-864d-841839156ae5" containerName="cleanup" Nov 03 02:45:29.437961 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.437934 5605 cpu_manager.go:414] "containerMap: removing container" podUID="6b8b50df-b64b-4249-864d-841839156ae5" containerName="cleanup" Nov 03 02:45:29.437961 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.437939 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b8b50df-b64b-4249-864d-841839156ae5" containerName="cleanup" Nov 03 02:45:29.437961 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.437949 5605 cpu_manager.go:414] "containerMap: removing container" podUID="305b0cb8-2348-4517-86a5-f77bfe6a3814" containerName="cleanup" Nov 03 02:45:29.437961 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.437954 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="305b0cb8-2348-4517-86a5-f77bfe6a3814" containerName="cleanup" Nov 03 02:45:29.438253 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.437965 5605 cpu_manager.go:414] "containerMap: removing container" podUID="6b8b50df-b64b-4249-864d-841839156ae5" containerName="cleanup" Nov 03 02:45:29.438253 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.437969 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b8b50df-b64b-4249-864d-841839156ae5" containerName="cleanup" Nov 03 02:45:29.438253 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.437978 5605 cpu_manager.go:414] "containerMap: removing container" podUID="aeb1d4c3-d1f5-49f4-9e3c-47ff2b640ac9" containerName="cleanup" Nov 03 02:45:29.438253 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.437982 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aeb1d4c3-d1f5-49f4-9e3c-47ff2b640ac9" containerName="cleanup" Nov 03 02:45:29.438253 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.437988 5605 cpu_manager.go:414] "containerMap: removing container" podUID="aeb1d4c3-d1f5-49f4-9e3c-47ff2b640ac9" containerName="cleanup" Nov 03 02:45:29.438253 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.437993 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aeb1d4c3-d1f5-49f4-9e3c-47ff2b640ac9" containerName="cleanup" Nov 03 02:45:29.438253 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.438000 5605 cpu_manager.go:414] "containerMap: removing container" podUID="aeb1d4c3-d1f5-49f4-9e3c-47ff2b640ac9" containerName="cleanup" Nov 03 02:45:29.438253 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.438005 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aeb1d4c3-d1f5-49f4-9e3c-47ff2b640ac9" containerName="cleanup" Nov 03 02:45:29.438253 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.438014 5605 cpu_manager.go:414] "containerMap: removing container" podUID="757daf90-e81c-4b1d-86ff-c6a6afb9d4e4" containerName="main" Nov 03 02:45:29.438253 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.438019 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="757daf90-e81c-4b1d-86ff-c6a6afb9d4e4" containerName="main" Nov 03 02:45:29.438253 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.438025 5605 cpu_manager.go:414] "containerMap: removing container" podUID="6a07adb9-2325-4532-97d0-00bdd2c8d6f0" containerName="revoke-keys" Nov 03 02:45:29.438253 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.438029 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a07adb9-2325-4532-97d0-00bdd2c8d6f0" containerName="revoke-keys" Nov 03 02:45:29.438253 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.438039 5605 cpu_manager.go:414] "containerMap: removing container" podUID="ed96e13a-3942-46e6-846f-027f5d1718cc" containerName="cleanup" Nov 03 02:45:29.438253 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.438043 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed96e13a-3942-46e6-846f-027f5d1718cc" containerName="cleanup" Nov 03 02:45:29.438253 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.438050 5605 cpu_manager.go:414] "containerMap: removing container" podUID="305b0cb8-2348-4517-86a5-f77bfe6a3814" containerName="cleanup" Nov 03 02:45:29.438253 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.438055 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="305b0cb8-2348-4517-86a5-f77bfe6a3814" containerName="cleanup" Nov 03 02:45:29.438253 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.438063 5605 cpu_manager.go:414] "containerMap: removing container" podUID="ed96e13a-3942-46e6-846f-027f5d1718cc" containerName="cleanup" Nov 03 02:45:29.438253 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.438067 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed96e13a-3942-46e6-846f-027f5d1718cc" containerName="cleanup" Nov 03 02:45:29.438253 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.438079 5605 cpu_manager.go:414] "containerMap: removing container" podUID="e94bb7db-1559-4c16-857f-3d20e0a7a008" containerName="cleanup" Nov 03 02:45:29.438253 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.438083 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e94bb7db-1559-4c16-857f-3d20e0a7a008" containerName="cleanup" Nov 03 02:45:29.438253 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.438088 5605 cpu_manager.go:414] "containerMap: removing container" podUID="660b74cc-c09a-48c2-b447-d869e6f3efa2" containerName="cleanup" Nov 03 02:45:29.438253 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.438110 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="660b74cc-c09a-48c2-b447-d869e6f3efa2" containerName="cleanup" Nov 03 02:45:29.438253 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.438120 5605 cpu_manager.go:414] "containerMap: removing container" podUID="0dc6be8c-60cf-4429-9ced-90ca7ed49656" containerName="cleanup" Nov 03 02:45:29.438253 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.438124 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dc6be8c-60cf-4429-9ced-90ca7ed49656" containerName="cleanup" Nov 03 02:45:29.438253 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.438133 5605 cpu_manager.go:414] "containerMap: removing container" podUID="ed96e13a-3942-46e6-846f-027f5d1718cc" containerName="cleanup" Nov 03 02:45:29.438253 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.438138 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed96e13a-3942-46e6-846f-027f5d1718cc" containerName="cleanup" Nov 03 02:45:29.438253 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.438203 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe40cb78-4bfe-4975-acbd-867c3366f3ae" containerName="cleanup" Nov 03 02:45:29.438253 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.438211 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="305b0cb8-2348-4517-86a5-f77bfe6a3814" containerName="cleanup" Nov 03 02:45:29.438253 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.438216 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="660b74cc-c09a-48c2-b447-d869e6f3efa2" containerName="cleanup" Nov 03 02:45:29.438253 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.438224 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed96e13a-3942-46e6-846f-027f5d1718cc" containerName="cleanup" Nov 03 02:45:29.438253 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.438230 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="660b74cc-c09a-48c2-b447-d869e6f3efa2" containerName="cleanup" Nov 03 02:45:29.438253 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.438236 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b8b50df-b64b-4249-864d-841839156ae5" containerName="cleanup" Nov 03 02:45:29.438253 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.438242 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b8b50df-b64b-4249-864d-841839156ae5" containerName="cleanup" Nov 03 02:45:29.438253 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.438247 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="305b0cb8-2348-4517-86a5-f77bfe6a3814" containerName="cleanup" Nov 03 02:45:29.438253 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.438253 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="660b74cc-c09a-48c2-b447-d869e6f3efa2" containerName="cleanup" Nov 03 02:45:29.438253 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.438259 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b8b50df-b64b-4249-864d-841839156ae5" containerName="cleanup" Nov 03 02:45:29.438253 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.438265 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="aeb1d4c3-d1f5-49f4-9e3c-47ff2b640ac9" containerName="cleanup" Nov 03 02:45:29.438253 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.438270 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="aeb1d4c3-d1f5-49f4-9e3c-47ff2b640ac9" containerName="cleanup" Nov 03 02:45:29.439463 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.438276 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="aeb1d4c3-d1f5-49f4-9e3c-47ff2b640ac9" containerName="cleanup" Nov 03 02:45:29.439463 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.438283 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dc6be8c-60cf-4429-9ced-90ca7ed49656" containerName="cleanup" Nov 03 02:45:29.439463 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.438288 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="757daf90-e81c-4b1d-86ff-c6a6afb9d4e4" containerName="main" Nov 03 02:45:29.439463 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.438294 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a07adb9-2325-4532-97d0-00bdd2c8d6f0" containerName="revoke-keys" Nov 03 02:45:29.439463 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.438301 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed96e13a-3942-46e6-846f-027f5d1718cc" containerName="cleanup" Nov 03 02:45:29.442993 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.442973 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d2c4664-tjjzg" Nov 03 02:45:29.497046 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.497013 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d2c4664-tjjzg"] Nov 03 02:45:29.540620 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.540586 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:45:29.540825 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.540702 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:45:29.540825 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.540737 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:45:29.540825 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.540791 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:45:29.613456 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.613416 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfa077e8-2701-4b9c-a30a-cdc528a66a11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d2c4664-tjjzg\" (UID: \"dfa077e8-2701-4b9c-a30a-cdc528a66a11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d2c4664-tjjzg" Nov 03 02:45:29.613878 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.613636 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfa077e8-2701-4b9c-a30a-cdc528a66a11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d2c4664-tjjzg\" (UID: \"dfa077e8-2701-4b9c-a30a-cdc528a66a11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d2c4664-tjjzg" Nov 03 02:45:29.613878 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.613696 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrd89\" (UniqueName: \"kubernetes.io/projected/dfa077e8-2701-4b9c-a30a-cdc528a66a11-kube-api-access-vrd89\") pod \"maas-api-revoke-sub-1d2c4664-tjjzg\" (UID: \"dfa077e8-2701-4b9c-a30a-cdc528a66a11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d2c4664-tjjzg" Nov 03 02:45:29.715276 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.715172 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfa077e8-2701-4b9c-a30a-cdc528a66a11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d2c4664-tjjzg\" (UID: \"dfa077e8-2701-4b9c-a30a-cdc528a66a11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d2c4664-tjjzg" Nov 03 02:45:29.715276 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.715220 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrd89\" (UniqueName: \"kubernetes.io/projected/dfa077e8-2701-4b9c-a30a-cdc528a66a11-kube-api-access-vrd89\") pod \"maas-api-revoke-sub-1d2c4664-tjjzg\" (UID: \"dfa077e8-2701-4b9c-a30a-cdc528a66a11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d2c4664-tjjzg" Nov 03 02:45:29.715518 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.715319 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfa077e8-2701-4b9c-a30a-cdc528a66a11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d2c4664-tjjzg\" (UID: \"dfa077e8-2701-4b9c-a30a-cdc528a66a11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d2c4664-tjjzg" Nov 03 02:45:29.715958 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.715927 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfa077e8-2701-4b9c-a30a-cdc528a66a11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d2c4664-tjjzg\" (UID: \"dfa077e8-2701-4b9c-a30a-cdc528a66a11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d2c4664-tjjzg" Nov 03 02:45:29.717705 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.717682 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfa077e8-2701-4b9c-a30a-cdc528a66a11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d2c4664-tjjzg\" (UID: \"dfa077e8-2701-4b9c-a30a-cdc528a66a11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d2c4664-tjjzg" Nov 03 02:45:29.738620 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.738581 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx"] Nov 03 02:45:29.739386 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.739340 5605 cpu_manager.go:414] "containerMap: removing container" podUID="660b74cc-c09a-48c2-b447-d869e6f3efa2" containerName="cleanup" Nov 03 02:45:29.739386 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.739384 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="660b74cc-c09a-48c2-b447-d869e6f3efa2" containerName="cleanup" Nov 03 02:45:29.739595 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.739409 5605 cpu_manager.go:414] "containerMap: removing container" podUID="305b0cb8-2348-4517-86a5-f77bfe6a3814" containerName="cleanup" Nov 03 02:45:29.739595 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.739419 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="305b0cb8-2348-4517-86a5-f77bfe6a3814" containerName="cleanup" Nov 03 02:45:29.739595 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.739569 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="305b0cb8-2348-4517-86a5-f77bfe6a3814" containerName="cleanup" Nov 03 02:45:29.739595 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.739588 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed96e13a-3942-46e6-846f-027f5d1718cc" containerName="cleanup" Nov 03 02:45:29.745420 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.743626 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx" Nov 03 02:45:29.771395 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.768993 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrd89\" (UniqueName: \"kubernetes.io/projected/dfa077e8-2701-4b9c-a30a-cdc528a66a11-kube-api-access-vrd89\") pod \"maas-api-revoke-sub-1d2c4664-tjjzg\" (UID: \"dfa077e8-2701-4b9c-a30a-cdc528a66a11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d2c4664-tjjzg" Nov 03 02:45:29.851381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.850122 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-c8ee6b-openshift-default-dockercfg-w7b9g\"" Nov 03 02:45:29.851381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.850404 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bf95767-b937-4d51-86ff-a41db2ffc3a4-workload-certs\") pod \"e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx\" (UID: \"3bf95767-b937-4d51-86ff-a41db2ffc3a4\") " pod="openshift-ingress/e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx" Nov 03 02:45:29.851381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.850442 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bf95767-b937-4d51-86ff-a41db2ffc3a4-istio-podinfo\") pod \"e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx\" (UID: \"3bf95767-b937-4d51-86ff-a41db2ffc3a4\") " pod="openshift-ingress/e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx" Nov 03 02:45:29.851381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.850476 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bf95767-b937-4d51-86ff-a41db2ffc3a4-istiod-ca-cert\") pod \"e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx\" (UID: \"3bf95767-b937-4d51-86ff-a41db2ffc3a4\") " pod="openshift-ingress/e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx" Nov 03 02:45:29.851381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.850549 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bf95767-b937-4d51-86ff-a41db2ffc3a4-istio-token\") pod \"e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx\" (UID: \"3bf95767-b937-4d51-86ff-a41db2ffc3a4\") " pod="openshift-ingress/e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx" Nov 03 02:45:29.851381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.850591 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf95767-b937-4d51-86ff-a41db2ffc3a4-workload-socket\") pod \"e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx\" (UID: \"3bf95767-b937-4d51-86ff-a41db2ffc3a4\") " pod="openshift-ingress/e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx" Nov 03 02:45:29.851381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.850616 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bf95767-b937-4d51-86ff-a41db2ffc3a4-istio-envoy\") pod \"e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx\" (UID: \"3bf95767-b937-4d51-86ff-a41db2ffc3a4\") " pod="openshift-ingress/e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx" Nov 03 02:45:29.851381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.850662 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqmpv\" (UniqueName: \"kubernetes.io/projected/3bf95767-b937-4d51-86ff-a41db2ffc3a4-kube-api-access-sqmpv\") pod \"e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx\" (UID: \"3bf95767-b937-4d51-86ff-a41db2ffc3a4\") " pod="openshift-ingress/e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx" Nov 03 02:45:29.851381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.850688 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bf95767-b937-4d51-86ff-a41db2ffc3a4-istio-data\") pod \"e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx\" (UID: \"3bf95767-b937-4d51-86ff-a41db2ffc3a4\") " pod="openshift-ingress/e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx" Nov 03 02:45:29.851381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.850719 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf95767-b937-4d51-86ff-a41db2ffc3a4-credential-socket\") pod \"e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx\" (UID: \"3bf95767-b937-4d51-86ff-a41db2ffc3a4\") " pod="openshift-ingress/e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx" Nov 03 02:45:29.859257 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.859225 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx"] Nov 03 02:45:29.951430 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.951395 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bf95767-b937-4d51-86ff-a41db2ffc3a4-istio-token\") pod \"e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx\" (UID: \"3bf95767-b937-4d51-86ff-a41db2ffc3a4\") " pod="openshift-ingress/e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx" Nov 03 02:45:29.951621 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.951452 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf95767-b937-4d51-86ff-a41db2ffc3a4-workload-socket\") pod \"e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx\" (UID: \"3bf95767-b937-4d51-86ff-a41db2ffc3a4\") " pod="openshift-ingress/e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx" Nov 03 02:45:29.951621 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.951481 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bf95767-b937-4d51-86ff-a41db2ffc3a4-istio-envoy\") pod \"e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx\" (UID: \"3bf95767-b937-4d51-86ff-a41db2ffc3a4\") " pod="openshift-ingress/e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx" Nov 03 02:45:29.951621 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.951545 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqmpv\" (UniqueName: \"kubernetes.io/projected/3bf95767-b937-4d51-86ff-a41db2ffc3a4-kube-api-access-sqmpv\") pod \"e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx\" (UID: \"3bf95767-b937-4d51-86ff-a41db2ffc3a4\") " pod="openshift-ingress/e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx" Nov 03 02:45:29.951621 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.951573 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bf95767-b937-4d51-86ff-a41db2ffc3a4-istio-data\") pod \"e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx\" (UID: \"3bf95767-b937-4d51-86ff-a41db2ffc3a4\") " pod="openshift-ingress/e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx" Nov 03 02:45:29.951840 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.951704 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf95767-b937-4d51-86ff-a41db2ffc3a4-credential-socket\") pod \"e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx\" (UID: \"3bf95767-b937-4d51-86ff-a41db2ffc3a4\") " pod="openshift-ingress/e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx" Nov 03 02:45:29.951889 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.951859 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bf95767-b937-4d51-86ff-a41db2ffc3a4-workload-certs\") pod \"e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx\" (UID: \"3bf95767-b937-4d51-86ff-a41db2ffc3a4\") " pod="openshift-ingress/e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx" Nov 03 02:45:29.951937 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.951885 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bf95767-b937-4d51-86ff-a41db2ffc3a4-istio-podinfo\") pod \"e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx\" (UID: \"3bf95767-b937-4d51-86ff-a41db2ffc3a4\") " pod="openshift-ingress/e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx" Nov 03 02:45:29.951937 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.951925 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bf95767-b937-4d51-86ff-a41db2ffc3a4-istiod-ca-cert\") pod \"e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx\" (UID: \"3bf95767-b937-4d51-86ff-a41db2ffc3a4\") " pod="openshift-ingress/e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx" Nov 03 02:45:29.952172 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.952148 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf95767-b937-4d51-86ff-a41db2ffc3a4-workload-socket\") pod \"e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx\" (UID: \"3bf95767-b937-4d51-86ff-a41db2ffc3a4\") " pod="openshift-ingress/e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx" Nov 03 02:45:29.952266 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.952176 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bf95767-b937-4d51-86ff-a41db2ffc3a4-istio-data\") pod \"e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx\" (UID: \"3bf95767-b937-4d51-86ff-a41db2ffc3a4\") " pod="openshift-ingress/e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx" Nov 03 02:45:29.952326 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.952305 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf95767-b937-4d51-86ff-a41db2ffc3a4-credential-socket\") pod \"e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx\" (UID: \"3bf95767-b937-4d51-86ff-a41db2ffc3a4\") " pod="openshift-ingress/e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx" Nov 03 02:45:29.952395 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.952309 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bf95767-b937-4d51-86ff-a41db2ffc3a4-workload-certs\") pod \"e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx\" (UID: \"3bf95767-b937-4d51-86ff-a41db2ffc3a4\") " pod="openshift-ingress/e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx" Nov 03 02:45:29.952543 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.952521 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bf95767-b937-4d51-86ff-a41db2ffc3a4-istiod-ca-cert\") pod \"e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx\" (UID: \"3bf95767-b937-4d51-86ff-a41db2ffc3a4\") " pod="openshift-ingress/e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx" Nov 03 02:45:29.954208 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.954182 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bf95767-b937-4d51-86ff-a41db2ffc3a4-istio-envoy\") pod \"e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx\" (UID: \"3bf95767-b937-4d51-86ff-a41db2ffc3a4\") " pod="openshift-ingress/e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx" Nov 03 02:45:29.954737 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:29.954710 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bf95767-b937-4d51-86ff-a41db2ffc3a4-istio-podinfo\") pod \"e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx\" (UID: \"3bf95767-b937-4d51-86ff-a41db2ffc3a4\") " pod="openshift-ingress/e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx" Nov 03 02:45:30.008467 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:30.008223 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bf95767-b937-4d51-86ff-a41db2ffc3a4-istio-token\") pod \"e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx\" (UID: \"3bf95767-b937-4d51-86ff-a41db2ffc3a4\") " pod="openshift-ingress/e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx" Nov 03 02:45:30.019045 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:30.008957 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqmpv\" (UniqueName: \"kubernetes.io/projected/3bf95767-b937-4d51-86ff-a41db2ffc3a4-kube-api-access-sqmpv\") pod \"e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx\" (UID: \"3bf95767-b937-4d51-86ff-a41db2ffc3a4\") " pod="openshift-ingress/e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx" Nov 03 02:45:30.056794 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:30.056757 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d2c4664-tjjzg" Nov 03 02:45:30.060819 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:30.060790 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx" Nov 03 02:45:30.440440 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:30.440397 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d2c4664-tjjzg"] Nov 03 02:45:30.486864 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:45:30.486832 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfa077e8_2701_4b9c_a30a_cdc528a66a11.slice/crio-a7299eca8a564e9cd3068025df116acc5aec688d5fa872afc16109253be9d65f WatchSource:0}: Error finding container a7299eca8a564e9cd3068025df116acc5aec688d5fa872afc16109253be9d65f: Status 404 returned error can't find the container with id a7299eca8a564e9cd3068025df116acc5aec688d5fa872afc16109253be9d65f Nov 03 02:45:30.493909 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:30.493886 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx"] Nov 03 02:45:30.499056 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:45:30.499026 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bf95767_b937_4d51_86ff_a41db2ffc3a4.slice/crio-80cf23efc0c8e687ce8fafeb4cf5ff98b202788e09371e63c421f13729185c58 WatchSource:0}: Error finding container 80cf23efc0c8e687ce8fafeb4cf5ff98b202788e09371e63c421f13729185c58: Status 404 returned error can't find the container with id 80cf23efc0c8e687ce8fafeb4cf5ff98b202788e09371e63c421f13729185c58 Nov 03 02:45:30.501895 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:30.501859 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:45:30.501976 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:30.501945 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:45:30.502025 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:30.501990 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:45:31.065792 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:31.065752 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d2c4664-tjjzg" event={"ID":"dfa077e8-2701-4b9c-a30a-cdc528a66a11","Type":"ContainerStarted","Data":"30bf88dda1735ec7588e266213fcd43ac4b1fcb8a9d974b3a4477a071bc0c4c7"} Nov 03 02:45:31.066238 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:31.065822 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d2c4664-tjjzg" event={"ID":"dfa077e8-2701-4b9c-a30a-cdc528a66a11","Type":"ContainerStarted","Data":"a7299eca8a564e9cd3068025df116acc5aec688d5fa872afc16109253be9d65f"} Nov 03 02:45:31.067663 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:31.067636 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx" event={"ID":"3bf95767-b937-4d51-86ff-a41db2ffc3a4","Type":"ContainerStarted","Data":"158e577028ff4dce01c75829bfb06516735f50c8ef5e1ba438d0c6020dabe8b2"} Nov 03 02:45:31.067809 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:31.067683 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx" event={"ID":"3bf95767-b937-4d51-86ff-a41db2ffc3a4","Type":"ContainerStarted","Data":"80cf23efc0c8e687ce8fafeb4cf5ff98b202788e09371e63c421f13729185c58"} Nov 03 02:45:31.171632 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:31.171526 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d2c4664-tjjzg" podStartSLOduration=2.17151216 podStartE2EDuration="2.17151216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:45:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:45:31.170616827 +0000 UTC m=+3208.429814980" watchObservedRunningTime="2026-11-03 02:45:31.17151216 +0000 UTC m=+3208.430710313" Nov 03 02:45:31.286001 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:31.285942 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx" podStartSLOduration=2.285929245 podStartE2EDuration="2.285929245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:45:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:45:31.283844853 +0000 UTC m=+3208.543043008" watchObservedRunningTime="2026-11-03 02:45:31.285929245 +0000 UTC m=+3208.545127399" Nov 03 02:45:31.429183 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:31.429104 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj"] Nov 03 02:45:31.433093 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:31.433069 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj" Nov 03 02:45:31.469424 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:31.469353 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9aecc649-014f-45ef-849d-27fad4f34933-workload-certs\") pod \"e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj\" (UID: \"9aecc649-014f-45ef-849d-27fad4f34933\") " pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj" Nov 03 02:45:31.469636 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:31.469531 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aecc649-014f-45ef-849d-27fad4f34933-credential-socket\") pod \"e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj\" (UID: \"9aecc649-014f-45ef-849d-27fad4f34933\") " pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj" Nov 03 02:45:31.469636 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:31.469575 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9aecc649-014f-45ef-849d-27fad4f34933-istiod-ca-cert\") pod \"e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj\" (UID: \"9aecc649-014f-45ef-849d-27fad4f34933\") " pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj" Nov 03 02:45:31.469636 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:31.469628 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9aecc649-014f-45ef-849d-27fad4f34933-istio-envoy\") pod \"e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj\" (UID: \"9aecc649-014f-45ef-849d-27fad4f34933\") " pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj" Nov 03 02:45:31.469820 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:31.469656 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92g55\" (UniqueName: \"kubernetes.io/projected/9aecc649-014f-45ef-849d-27fad4f34933-kube-api-access-92g55\") pod \"e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj\" (UID: \"9aecc649-014f-45ef-849d-27fad4f34933\") " pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj" Nov 03 02:45:31.469820 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:31.469718 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aecc649-014f-45ef-849d-27fad4f34933-workload-socket\") pod \"e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj\" (UID: \"9aecc649-014f-45ef-849d-27fad4f34933\") " pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj" Nov 03 02:45:31.469820 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:31.469746 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9aecc649-014f-45ef-849d-27fad4f34933-istio-podinfo\") pod \"e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj\" (UID: \"9aecc649-014f-45ef-849d-27fad4f34933\") " pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj" Nov 03 02:45:31.469989 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:31.469825 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9aecc649-014f-45ef-849d-27fad4f34933-istio-token\") pod \"e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj\" (UID: \"9aecc649-014f-45ef-849d-27fad4f34933\") " pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj" Nov 03 02:45:31.469989 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:31.469915 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9aecc649-014f-45ef-849d-27fad4f34933-istio-data\") pod \"e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj\" (UID: \"9aecc649-014f-45ef-849d-27fad4f34933\") " pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj" Nov 03 02:45:31.482016 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:31.481982 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-ffba43-openshift-default-dockercfg-vshng\"" Nov 03 02:45:31.489943 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:31.489915 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj"] Nov 03 02:45:31.571546 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:31.571486 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9aecc649-014f-45ef-849d-27fad4f34933-istio-data\") pod \"e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj\" (UID: \"9aecc649-014f-45ef-849d-27fad4f34933\") " pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj" Nov 03 02:45:31.571546 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:31.571563 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9aecc649-014f-45ef-849d-27fad4f34933-workload-certs\") pod \"e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj\" (UID: \"9aecc649-014f-45ef-849d-27fad4f34933\") " pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj" Nov 03 02:45:31.571812 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:31.571625 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aecc649-014f-45ef-849d-27fad4f34933-credential-socket\") pod \"e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj\" (UID: \"9aecc649-014f-45ef-849d-27fad4f34933\") " pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj" Nov 03 02:45:31.571812 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:31.571652 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9aecc649-014f-45ef-849d-27fad4f34933-istiod-ca-cert\") pod \"e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj\" (UID: \"9aecc649-014f-45ef-849d-27fad4f34933\") " pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj" Nov 03 02:45:31.571812 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:31.571668 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9aecc649-014f-45ef-849d-27fad4f34933-istio-envoy\") pod \"e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj\" (UID: \"9aecc649-014f-45ef-849d-27fad4f34933\") " pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj" Nov 03 02:45:31.571812 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:31.571688 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92g55\" (UniqueName: \"kubernetes.io/projected/9aecc649-014f-45ef-849d-27fad4f34933-kube-api-access-92g55\") pod \"e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj\" (UID: \"9aecc649-014f-45ef-849d-27fad4f34933\") " pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj" Nov 03 02:45:31.571812 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:31.571720 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aecc649-014f-45ef-849d-27fad4f34933-workload-socket\") pod \"e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj\" (UID: \"9aecc649-014f-45ef-849d-27fad4f34933\") " pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj" Nov 03 02:45:31.571812 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:31.571755 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9aecc649-014f-45ef-849d-27fad4f34933-istio-podinfo\") pod \"e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj\" (UID: \"9aecc649-014f-45ef-849d-27fad4f34933\") " pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj" Nov 03 02:45:31.571812 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:31.571780 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9aecc649-014f-45ef-849d-27fad4f34933-istio-token\") pod \"e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj\" (UID: \"9aecc649-014f-45ef-849d-27fad4f34933\") " pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj" Nov 03 02:45:31.572177 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:31.572023 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9aecc649-014f-45ef-849d-27fad4f34933-istio-data\") pod \"e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj\" (UID: \"9aecc649-014f-45ef-849d-27fad4f34933\") " pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj" Nov 03 02:45:31.572177 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:31.572083 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9aecc649-014f-45ef-849d-27fad4f34933-workload-certs\") pod \"e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj\" (UID: \"9aecc649-014f-45ef-849d-27fad4f34933\") " pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj" Nov 03 02:45:31.572546 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:31.572510 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aecc649-014f-45ef-849d-27fad4f34933-workload-socket\") pod \"e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj\" (UID: \"9aecc649-014f-45ef-849d-27fad4f34933\") " pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj" Nov 03 02:45:31.572546 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:31.572529 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9aecc649-014f-45ef-849d-27fad4f34933-istiod-ca-cert\") pod \"e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj\" (UID: \"9aecc649-014f-45ef-849d-27fad4f34933\") " pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj" Nov 03 02:45:31.572700 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:31.572529 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aecc649-014f-45ef-849d-27fad4f34933-credential-socket\") pod \"e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj\" (UID: \"9aecc649-014f-45ef-849d-27fad4f34933\") " pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj" Nov 03 02:45:31.574283 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:31.574255 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9aecc649-014f-45ef-849d-27fad4f34933-istio-envoy\") pod \"e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj\" (UID: \"9aecc649-014f-45ef-849d-27fad4f34933\") " pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj" Nov 03 02:45:31.574471 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:31.574454 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9aecc649-014f-45ef-849d-27fad4f34933-istio-podinfo\") pod \"e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj\" (UID: \"9aecc649-014f-45ef-849d-27fad4f34933\") " pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj" Nov 03 02:45:31.625098 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:31.625067 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9aecc649-014f-45ef-849d-27fad4f34933-istio-token\") pod \"e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj\" (UID: \"9aecc649-014f-45ef-849d-27fad4f34933\") " pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj" Nov 03 02:45:31.625294 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:31.625269 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92g55\" (UniqueName: \"kubernetes.io/projected/9aecc649-014f-45ef-849d-27fad4f34933-kube-api-access-92g55\") pod \"e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj\" (UID: \"9aecc649-014f-45ef-849d-27fad4f34933\") " pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj" Nov 03 02:45:31.748300 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:31.748210 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj" Nov 03 02:45:32.061515 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:32.061484 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx" Nov 03 02:45:32.067192 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:32.067167 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx" Nov 03 02:45:32.075595 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:32.075566 5605 generic.go:356] "Generic (PLEG): container finished" podID="dfa077e8-2701-4b9c-a30a-cdc528a66a11" containerID="30bf88dda1735ec7588e266213fcd43ac4b1fcb8a9d974b3a4477a071bc0c4c7" exitCode=0 Nov 03 02:45:32.075722 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:32.075644 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d2c4664-tjjzg" event={"ID":"dfa077e8-2701-4b9c-a30a-cdc528a66a11","Type":"ContainerDied","Data":"30bf88dda1735ec7588e266213fcd43ac4b1fcb8a9d974b3a4477a071bc0c4c7"} Nov 03 02:45:32.075962 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:32.075950 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx" Nov 03 02:45:32.076876 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:32.076856 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx" Nov 03 02:45:32.173133 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:32.173098 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj"] Nov 03 02:45:32.178050 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:45:32.178018 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aecc649_014f_45ef_849d_27fad4f34933.slice/crio-1a1ab3e9a5f0f953e9859c7f9e5a3cad3bdf33af422be9ec27bc8da92a874b26 WatchSource:0}: Error finding container 1a1ab3e9a5f0f953e9859c7f9e5a3cad3bdf33af422be9ec27bc8da92a874b26: Status 404 returned error can't find the container with id 1a1ab3e9a5f0f953e9859c7f9e5a3cad3bdf33af422be9ec27bc8da92a874b26 Nov 03 02:45:32.180716 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:32.180682 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:45:32.180802 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:32.180747 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:45:32.180802 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:32.180776 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:45:32.533463 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:32.533377 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:45:33.128504 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:33.128468 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj" event={"ID":"9aecc649-014f-45ef-849d-27fad4f34933","Type":"ContainerStarted","Data":"602dcaadbe89b58dd913ddc8e28386a8308228aba5b09e942a429efb0cfa15cb"} Nov 03 02:45:33.129060 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:33.129035 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj" event={"ID":"9aecc649-014f-45ef-849d-27fad4f34933","Type":"ContainerStarted","Data":"1a1ab3e9a5f0f953e9859c7f9e5a3cad3bdf33af422be9ec27bc8da92a874b26"} Nov 03 02:45:33.241614 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:33.241557 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj" podStartSLOduration=2.241538563 podStartE2EDuration="2.241538563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:45:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:45:33.241059904 +0000 UTC m=+3210.500258060" watchObservedRunningTime="2026-11-03 02:45:33.241538563 +0000 UTC m=+3210.500736716" Nov 03 02:45:33.331167 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:33.331142 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d2c4664-tjjzg" Nov 03 02:45:33.405193 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:33.405120 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfa077e8-2701-4b9c-a30a-cdc528a66a11-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfa077e8-2701-4b9c-a30a-cdc528a66a11-maas-api-openshift-ca\") pod \"dfa077e8-2701-4b9c-a30a-cdc528a66a11\" (UID: \"dfa077e8-2701-4b9c-a30a-cdc528a66a11\") " Nov 03 02:45:33.405193 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:33.405151 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfa077e8-2701-4b9c-a30a-cdc528a66a11-kube-api-access-vrd89\" (UniqueName: \"kubernetes.io/projected/dfa077e8-2701-4b9c-a30a-cdc528a66a11-kube-api-access-vrd89\") pod \"dfa077e8-2701-4b9c-a30a-cdc528a66a11\" (UID: \"dfa077e8-2701-4b9c-a30a-cdc528a66a11\") " Nov 03 02:45:33.405393 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:33.405290 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfa077e8-2701-4b9c-a30a-cdc528a66a11-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfa077e8-2701-4b9c-a30a-cdc528a66a11-maas-api-xks-ca\") pod \"dfa077e8-2701-4b9c-a30a-cdc528a66a11\" (UID: \"dfa077e8-2701-4b9c-a30a-cdc528a66a11\") " Nov 03 02:45:33.405551 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:33.405529 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfa077e8-2701-4b9c-a30a-cdc528a66a11-maas-api-openshift-ca" pod "dfa077e8-2701-4b9c-a30a-cdc528a66a11" (UID: "dfa077e8-2701-4b9c-a30a-cdc528a66a11"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:45:33.405826 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:33.405801 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfa077e8-2701-4b9c-a30a-cdc528a66a11-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:33.407406 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:33.407376 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfa077e8-2701-4b9c-a30a-cdc528a66a11-kube-api-access-vrd89" pod "dfa077e8-2701-4b9c-a30a-cdc528a66a11" (UID: "dfa077e8-2701-4b9c-a30a-cdc528a66a11"). InnerVolumeSpecName "kube-api-access-vrd89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:45:33.407519 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:33.407403 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfa077e8-2701-4b9c-a30a-cdc528a66a11-maas-api-xks-ca" pod "dfa077e8-2701-4b9c-a30a-cdc528a66a11" (UID: "dfa077e8-2701-4b9c-a30a-cdc528a66a11"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:45:33.507302 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:33.507266 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrd89\" (UniqueName: \"kubernetes.io/projected/dfa077e8-2701-4b9c-a30a-cdc528a66a11-kube-api-access-vrd89\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:33.507302 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:33.507297 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfa077e8-2701-4b9c-a30a-cdc528a66a11-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:33.748887 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:33.748799 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj" Nov 03 02:45:33.750349 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:33.750312 5605 patch_prober.go:29] interesting pod/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Nov 03 02:45:33.750474 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:33.750390 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj" podUID="9aecc649-014f-45ef-849d-27fad4f34933" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Nov 03 02:45:34.136502 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:34.136463 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d2c4664-tjjzg" Nov 03 02:45:34.136502 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:34.136480 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d2c4664-tjjzg" event={"ID":"dfa077e8-2701-4b9c-a30a-cdc528a66a11","Type":"ContainerDied","Data":"a7299eca8a564e9cd3068025df116acc5aec688d5fa872afc16109253be9d65f"} Nov 03 02:45:34.136937 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:34.136539 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7299eca8a564e9cd3068025df116acc5aec688d5fa872afc16109253be9d65f" Nov 03 02:45:34.406767 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:34.406681 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-63a61e/noten-model-63a61e-kserve-854bff945-8l56c"] Nov 03 02:45:34.407401 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:34.407352 5605 cpu_manager.go:414] "containerMap: removing container" podUID="dfa077e8-2701-4b9c-a30a-cdc528a66a11" containerName="revoke-keys" Nov 03 02:45:34.407401 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:34.407387 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfa077e8-2701-4b9c-a30a-cdc528a66a11" containerName="revoke-keys" Nov 03 02:45:34.407617 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:34.407531 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfa077e8-2701-4b9c-a30a-cdc528a66a11" containerName="revoke-keys" Nov 03 02:45:34.411297 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:34.411276 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-63a61e/noten-model-63a61e-kserve-854bff945-8l56c" Nov 03 02:45:34.490207 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:34.490170 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-63a61e/noten-model-63a61e-kserve-854bff945-8l56c"] Nov 03 02:45:34.491510 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:34.491492 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-63a61e\"/\"kube-root-ca.crt\"" Nov 03 02:45:34.520002 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:34.519971 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c56126f-02de-4af6-9f2e-c7ba74fc4a5f-tls-certs\") pod \"noten-model-63a61e-kserve-854bff945-8l56c\" (UID: \"9c56126f-02de-4af6-9f2e-c7ba74fc4a5f\") " pod="ai-tenant-e2e-noten-63a61e/noten-model-63a61e-kserve-854bff945-8l56c" Nov 03 02:45:34.520178 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:34.520038 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c56126f-02de-4af6-9f2e-c7ba74fc4a5f-model-cache\") pod \"noten-model-63a61e-kserve-854bff945-8l56c\" (UID: \"9c56126f-02de-4af6-9f2e-c7ba74fc4a5f\") " pod="ai-tenant-e2e-noten-63a61e/noten-model-63a61e-kserve-854bff945-8l56c" Nov 03 02:45:34.520178 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:34.520160 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c56126f-02de-4af6-9f2e-c7ba74fc4a5f-home\") pod \"noten-model-63a61e-kserve-854bff945-8l56c\" (UID: \"9c56126f-02de-4af6-9f2e-c7ba74fc4a5f\") " pod="ai-tenant-e2e-noten-63a61e/noten-model-63a61e-kserve-854bff945-8l56c" Nov 03 02:45:34.520248 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:34.520188 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c56126f-02de-4af6-9f2e-c7ba74fc4a5f-tmp-dir\") pod \"noten-model-63a61e-kserve-854bff945-8l56c\" (UID: \"9c56126f-02de-4af6-9f2e-c7ba74fc4a5f\") " pod="ai-tenant-e2e-noten-63a61e/noten-model-63a61e-kserve-854bff945-8l56c" Nov 03 02:45:34.520248 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:34.520218 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl52t\" (UniqueName: \"kubernetes.io/projected/9c56126f-02de-4af6-9f2e-c7ba74fc4a5f-kube-api-access-xl52t\") pod \"noten-model-63a61e-kserve-854bff945-8l56c\" (UID: \"9c56126f-02de-4af6-9f2e-c7ba74fc4a5f\") " pod="ai-tenant-e2e-noten-63a61e/noten-model-63a61e-kserve-854bff945-8l56c" Nov 03 02:45:34.520420 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:34.520352 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c56126f-02de-4af6-9f2e-c7ba74fc4a5f-dshm\") pod \"noten-model-63a61e-kserve-854bff945-8l56c\" (UID: \"9c56126f-02de-4af6-9f2e-c7ba74fc4a5f\") " pod="ai-tenant-e2e-noten-63a61e/noten-model-63a61e-kserve-854bff945-8l56c" Nov 03 02:45:34.532553 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:34.532523 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-63a61e\"/\"noten-model-63a61e-kserve-self-signed-certs\"" Nov 03 02:45:34.532553 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:34.532525 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-63a61e\"/\"openshift-service-ca.crt\"" Nov 03 02:45:34.532769 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:34.532585 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-63a61e\"/\"default-dockercfg-pc74g\"" Nov 03 02:45:34.621385 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:34.621312 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c56126f-02de-4af6-9f2e-c7ba74fc4a5f-dshm\") pod \"noten-model-63a61e-kserve-854bff945-8l56c\" (UID: \"9c56126f-02de-4af6-9f2e-c7ba74fc4a5f\") " pod="ai-tenant-e2e-noten-63a61e/noten-model-63a61e-kserve-854bff945-8l56c" Nov 03 02:45:34.621602 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:34.621419 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c56126f-02de-4af6-9f2e-c7ba74fc4a5f-tls-certs\") pod \"noten-model-63a61e-kserve-854bff945-8l56c\" (UID: \"9c56126f-02de-4af6-9f2e-c7ba74fc4a5f\") " pod="ai-tenant-e2e-noten-63a61e/noten-model-63a61e-kserve-854bff945-8l56c" Nov 03 02:45:34.621602 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:34.621470 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c56126f-02de-4af6-9f2e-c7ba74fc4a5f-model-cache\") pod \"noten-model-63a61e-kserve-854bff945-8l56c\" (UID: \"9c56126f-02de-4af6-9f2e-c7ba74fc4a5f\") " pod="ai-tenant-e2e-noten-63a61e/noten-model-63a61e-kserve-854bff945-8l56c" Nov 03 02:45:34.621602 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:34.621521 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c56126f-02de-4af6-9f2e-c7ba74fc4a5f-home\") pod \"noten-model-63a61e-kserve-854bff945-8l56c\" (UID: \"9c56126f-02de-4af6-9f2e-c7ba74fc4a5f\") " pod="ai-tenant-e2e-noten-63a61e/noten-model-63a61e-kserve-854bff945-8l56c" Nov 03 02:45:34.621602 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:34.621543 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c56126f-02de-4af6-9f2e-c7ba74fc4a5f-tmp-dir\") pod \"noten-model-63a61e-kserve-854bff945-8l56c\" (UID: \"9c56126f-02de-4af6-9f2e-c7ba74fc4a5f\") " pod="ai-tenant-e2e-noten-63a61e/noten-model-63a61e-kserve-854bff945-8l56c" Nov 03 02:45:34.621602 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:34.621587 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl52t\" (UniqueName: \"kubernetes.io/projected/9c56126f-02de-4af6-9f2e-c7ba74fc4a5f-kube-api-access-xl52t\") pod \"noten-model-63a61e-kserve-854bff945-8l56c\" (UID: \"9c56126f-02de-4af6-9f2e-c7ba74fc4a5f\") " pod="ai-tenant-e2e-noten-63a61e/noten-model-63a61e-kserve-854bff945-8l56c" Nov 03 02:45:34.621943 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:34.621918 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c56126f-02de-4af6-9f2e-c7ba74fc4a5f-model-cache\") pod \"noten-model-63a61e-kserve-854bff945-8l56c\" (UID: \"9c56126f-02de-4af6-9f2e-c7ba74fc4a5f\") " pod="ai-tenant-e2e-noten-63a61e/noten-model-63a61e-kserve-854bff945-8l56c" Nov 03 02:45:34.622020 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:34.621972 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c56126f-02de-4af6-9f2e-c7ba74fc4a5f-home\") pod \"noten-model-63a61e-kserve-854bff945-8l56c\" (UID: \"9c56126f-02de-4af6-9f2e-c7ba74fc4a5f\") " pod="ai-tenant-e2e-noten-63a61e/noten-model-63a61e-kserve-854bff945-8l56c" Nov 03 02:45:34.622076 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:34.622024 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c56126f-02de-4af6-9f2e-c7ba74fc4a5f-tmp-dir\") pod \"noten-model-63a61e-kserve-854bff945-8l56c\" (UID: \"9c56126f-02de-4af6-9f2e-c7ba74fc4a5f\") " pod="ai-tenant-e2e-noten-63a61e/noten-model-63a61e-kserve-854bff945-8l56c" Nov 03 02:45:34.623848 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:34.623818 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c56126f-02de-4af6-9f2e-c7ba74fc4a5f-dshm\") pod \"noten-model-63a61e-kserve-854bff945-8l56c\" (UID: \"9c56126f-02de-4af6-9f2e-c7ba74fc4a5f\") " pod="ai-tenant-e2e-noten-63a61e/noten-model-63a61e-kserve-854bff945-8l56c" Nov 03 02:45:34.624103 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:34.624085 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c56126f-02de-4af6-9f2e-c7ba74fc4a5f-tls-certs\") pod \"noten-model-63a61e-kserve-854bff945-8l56c\" (UID: \"9c56126f-02de-4af6-9f2e-c7ba74fc4a5f\") " pod="ai-tenant-e2e-noten-63a61e/noten-model-63a61e-kserve-854bff945-8l56c" Nov 03 02:45:34.681916 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:34.681787 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl52t\" (UniqueName: \"kubernetes.io/projected/9c56126f-02de-4af6-9f2e-c7ba74fc4a5f-kube-api-access-xl52t\") pod \"noten-model-63a61e-kserve-854bff945-8l56c\" (UID: \"9c56126f-02de-4af6-9f2e-c7ba74fc4a5f\") " pod="ai-tenant-e2e-noten-63a61e/noten-model-63a61e-kserve-854bff945-8l56c" Nov 03 02:45:34.725344 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:34.725301 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-63a61e/noten-model-63a61e-kserve-854bff945-8l56c" Nov 03 02:45:34.767382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:34.767003 5605 patch_prober.go:29] interesting pod/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Nov 03 02:45:34.767382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:34.767063 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj" podUID="9aecc649-014f-45ef-849d-27fad4f34933" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Nov 03 02:45:35.147119 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:35.147086 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-63a61e/noten-model-63a61e-kserve-854bff945-8l56c"] Nov 03 02:45:35.151547 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:45:35.151520 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c56126f_02de_4af6_9f2e_c7ba74fc4a5f.slice/crio-d97a82402c81dcb8cb6dc190b6e0b0a3b32a2ee734e028e3c11a81d7bbadff5b WatchSource:0}: Error finding container d97a82402c81dcb8cb6dc190b6e0b0a3b32a2ee734e028e3c11a81d7bbadff5b: Status 404 returned error can't find the container with id d97a82402c81dcb8cb6dc190b6e0b0a3b32a2ee734e028e3c11a81d7bbadff5b Nov 03 02:45:35.749380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:35.749318 5605 patch_prober.go:29] interesting pod/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Nov 03 02:45:35.749562 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:35.749404 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj" podUID="9aecc649-014f-45ef-849d-27fad4f34933" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Nov 03 02:45:36.151555 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:36.151516 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-63a61e/noten-model-63a61e-kserve-854bff945-8l56c" event={"ID":"9c56126f-02de-4af6-9f2e-c7ba74fc4a5f","Type":"ContainerStarted","Data":"556f5dc078ba121bba06e4a888abef9e18f98275fd6866f2a632038d98c7696c"} Nov 03 02:45:36.151555 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:36.151560 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-63a61e/noten-model-63a61e-kserve-854bff945-8l56c" event={"ID":"9c56126f-02de-4af6-9f2e-c7ba74fc4a5f","Type":"ContainerStarted","Data":"d97a82402c81dcb8cb6dc190b6e0b0a3b32a2ee734e028e3c11a81d7bbadff5b"} Nov 03 02:45:36.277382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:36.269706 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-63a61e/noten-model-63a61e-kserve-854bff945-8l56c" podStartSLOduration=2.269686264 podStartE2EDuration="2.269686264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:45:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:45:36.26787898 +0000 UTC m=+3213.527077136" watchObservedRunningTime="2026-11-03 02:45:36.269686264 +0000 UTC m=+3213.528884419" Nov 03 02:45:36.725611 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:36.725568 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-63a61e/noten-model-63a61e-kserve-854bff945-8l56c" Nov 03 02:45:36.748716 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:36.748680 5605 patch_prober.go:29] interesting pod/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Nov 03 02:45:36.748866 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:36.748740 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj" podUID="9aecc649-014f-45ef-849d-27fad4f34933" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Nov 03 02:45:37.749421 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:37.749386 5605 patch_prober.go:29] interesting pod/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Nov 03 02:45:37.749821 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:37.749441 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj" podUID="9aecc649-014f-45ef-849d-27fad4f34933" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Nov 03 02:45:38.760707 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:38.760668 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj" Nov 03 02:45:38.761540 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:38.761521 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj" Nov 03 02:45:38.762872 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:38.762852 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj" Nov 03 02:45:43.614900 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:43.614856 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-957b33/auth-test-model-957b33-kserve-c8778cdbf-dlqkf" podUID="eddf0af3-8acf-4e49-bd31-a8235ab2e03b" containerName="main" containerID="cri-o://5b950fc72dcce6d2ae8980468caeadcc47e16957c80abbe1d2adc296455ff298" gracePeriod=45 Nov 03 02:45:43.874143 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:43.874088 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-957b33/auth-test-model-957b33-kserve-c8778cdbf-dlqkf" Nov 03 02:45:43.957649 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:43.957622 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eddf0af3-8acf-4e49-bd31-a8235ab2e03b-kube-api-access-n4hr9\" (UniqueName: \"kubernetes.io/projected/eddf0af3-8acf-4e49-bd31-a8235ab2e03b-kube-api-access-n4hr9\") pod \"eddf0af3-8acf-4e49-bd31-a8235ab2e03b\" (UID: \"eddf0af3-8acf-4e49-bd31-a8235ab2e03b\") " Nov 03 02:45:43.957816 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:43.957660 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eddf0af3-8acf-4e49-bd31-a8235ab2e03b-tls-certs\" (UniqueName: \"kubernetes.io/secret/eddf0af3-8acf-4e49-bd31-a8235ab2e03b-tls-certs\") pod \"eddf0af3-8acf-4e49-bd31-a8235ab2e03b\" (UID: \"eddf0af3-8acf-4e49-bd31-a8235ab2e03b\") " Nov 03 02:45:43.957816 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:43.957740 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eddf0af3-8acf-4e49-bd31-a8235ab2e03b-home\" (UniqueName: \"kubernetes.io/empty-dir/eddf0af3-8acf-4e49-bd31-a8235ab2e03b-home\") pod \"eddf0af3-8acf-4e49-bd31-a8235ab2e03b\" (UID: \"eddf0af3-8acf-4e49-bd31-a8235ab2e03b\") " Nov 03 02:45:43.957816 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:43.957769 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eddf0af3-8acf-4e49-bd31-a8235ab2e03b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eddf0af3-8acf-4e49-bd31-a8235ab2e03b-tmp-dir\") pod \"eddf0af3-8acf-4e49-bd31-a8235ab2e03b\" (UID: \"eddf0af3-8acf-4e49-bd31-a8235ab2e03b\") " Nov 03 02:45:43.957997 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:43.957851 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eddf0af3-8acf-4e49-bd31-a8235ab2e03b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/eddf0af3-8acf-4e49-bd31-a8235ab2e03b-dshm\") pod \"eddf0af3-8acf-4e49-bd31-a8235ab2e03b\" (UID: \"eddf0af3-8acf-4e49-bd31-a8235ab2e03b\") " Nov 03 02:45:43.957997 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:43.957923 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eddf0af3-8acf-4e49-bd31-a8235ab2e03b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eddf0af3-8acf-4e49-bd31-a8235ab2e03b-model-cache\") pod \"eddf0af3-8acf-4e49-bd31-a8235ab2e03b\" (UID: \"eddf0af3-8acf-4e49-bd31-a8235ab2e03b\") " Nov 03 02:45:43.958108 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:43.958038 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eddf0af3-8acf-4e49-bd31-a8235ab2e03b-home" pod "eddf0af3-8acf-4e49-bd31-a8235ab2e03b" (UID: "eddf0af3-8acf-4e49-bd31-a8235ab2e03b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:45:43.958108 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:43.958052 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eddf0af3-8acf-4e49-bd31-a8235ab2e03b-tmp-dir" pod "eddf0af3-8acf-4e49-bd31-a8235ab2e03b" (UID: "eddf0af3-8acf-4e49-bd31-a8235ab2e03b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:45:43.958262 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:43.958236 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eddf0af3-8acf-4e49-bd31-a8235ab2e03b-model-cache" pod "eddf0af3-8acf-4e49-bd31-a8235ab2e03b" (UID: "eddf0af3-8acf-4e49-bd31-a8235ab2e03b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:45:43.958400 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:43.958381 5605 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eddf0af3-8acf-4e49-bd31-a8235ab2e03b-model-cache\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:43.958482 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:43.958402 5605 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eddf0af3-8acf-4e49-bd31-a8235ab2e03b-home\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:43.958482 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:43.958429 5605 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eddf0af3-8acf-4e49-bd31-a8235ab2e03b-tmp-dir\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:43.959977 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:43.959954 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eddf0af3-8acf-4e49-bd31-a8235ab2e03b-dshm" pod "eddf0af3-8acf-4e49-bd31-a8235ab2e03b" (UID: "eddf0af3-8acf-4e49-bd31-a8235ab2e03b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:45:43.960253 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:43.959958 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eddf0af3-8acf-4e49-bd31-a8235ab2e03b-tls-certs" pod "eddf0af3-8acf-4e49-bd31-a8235ab2e03b" (UID: "eddf0af3-8acf-4e49-bd31-a8235ab2e03b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:45:43.960253 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:43.960087 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eddf0af3-8acf-4e49-bd31-a8235ab2e03b-kube-api-access-n4hr9" pod "eddf0af3-8acf-4e49-bd31-a8235ab2e03b" (UID: "eddf0af3-8acf-4e49-bd31-a8235ab2e03b"). InnerVolumeSpecName "kube-api-access-n4hr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:45:44.059197 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:44.059154 5605 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eddf0af3-8acf-4e49-bd31-a8235ab2e03b-dshm\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:44.059197 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:44.059194 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4hr9\" (UniqueName: \"kubernetes.io/projected/eddf0af3-8acf-4e49-bd31-a8235ab2e03b-kube-api-access-n4hr9\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:44.059411 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:44.059211 5605 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eddf0af3-8acf-4e49-bd31-a8235ab2e03b-tls-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:44.235770 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:44.235680 5605 generic.go:356] "Generic (PLEG): container finished" podID="eddf0af3-8acf-4e49-bd31-a8235ab2e03b" containerID="5b950fc72dcce6d2ae8980468caeadcc47e16957c80abbe1d2adc296455ff298" exitCode=0 Nov 03 02:45:44.235770 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:44.235724 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-957b33/auth-test-model-957b33-kserve-c8778cdbf-dlqkf" Nov 03 02:45:44.235983 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:44.235768 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-957b33/auth-test-model-957b33-kserve-c8778cdbf-dlqkf" event={"ID":"eddf0af3-8acf-4e49-bd31-a8235ab2e03b","Type":"ContainerDied","Data":"5b950fc72dcce6d2ae8980468caeadcc47e16957c80abbe1d2adc296455ff298"} Nov 03 02:45:44.235983 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:44.235809 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-957b33/auth-test-model-957b33-kserve-c8778cdbf-dlqkf" event={"ID":"eddf0af3-8acf-4e49-bd31-a8235ab2e03b","Type":"ContainerDied","Data":"4d02ffc5f80f36fdb513a3a2095913086e3b753390b33dbdf6214bd4923d9177"} Nov 03 02:45:44.235983 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:44.235826 5605 scope.go:122] "RemoveContainer" containerID="5b950fc72dcce6d2ae8980468caeadcc47e16957c80abbe1d2adc296455ff298" Nov 03 02:45:44.246730 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:44.246714 5605 scope.go:122] "RemoveContainer" containerID="5b950fc72dcce6d2ae8980468caeadcc47e16957c80abbe1d2adc296455ff298" Nov 03 02:45:44.246974 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:45:44.246956 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b950fc72dcce6d2ae8980468caeadcc47e16957c80abbe1d2adc296455ff298\": container with ID starting with 5b950fc72dcce6d2ae8980468caeadcc47e16957c80abbe1d2adc296455ff298 not found: ID does not exist" containerID="5b950fc72dcce6d2ae8980468caeadcc47e16957c80abbe1d2adc296455ff298" Nov 03 02:45:44.247020 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:44.246985 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b950fc72dcce6d2ae8980468caeadcc47e16957c80abbe1d2adc296455ff298"} err="failed to get container status \"5b950fc72dcce6d2ae8980468caeadcc47e16957c80abbe1d2adc296455ff298\": rpc error: code = NotFound desc = could not find container \"5b950fc72dcce6d2ae8980468caeadcc47e16957c80abbe1d2adc296455ff298\": container with ID starting with 5b950fc72dcce6d2ae8980468caeadcc47e16957c80abbe1d2adc296455ff298 not found: ID does not exist" Nov 03 02:45:44.391208 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:44.391145 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-957b33/auth-test-model-957b33-kserve-c8778cdbf-dlqkf"] Nov 03 02:45:44.394425 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:44.394395 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-957b33/auth-test-model-957b33-kserve-c8778cdbf-dlqkf"] Nov 03 02:45:44.725704 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:44.725664 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-63a61e/noten-model-63a61e-kserve-854bff945-8l56c" Nov 03 02:45:44.738020 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:44.737995 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-63a61e/noten-model-63a61e-kserve-854bff945-8l56c" Nov 03 02:45:44.849898 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:44.849868 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eddf0af3-8acf-4e49-bd31-a8235ab2e03b" path="/var/lib/kubelet/pods/eddf0af3-8acf-4e49-bd31-a8235ab2e03b/volumes" Nov 03 02:45:45.254502 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:45.254473 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-63a61e/noten-model-63a61e-kserve-854bff945-8l56c" Nov 03 02:45:45.578810 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:45:45.578777 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-9aebe8/auth-test-model-9aebe8-kserve-949d578b5-l8chz"] Nov 03 02:45:53.914220 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:45:53.914176 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:46:00.600655 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:00.600607 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-9aebe8/auth-test-model-9aebe8-kserve-949d578b5-l8chz" podUID="d715d6df-5d86-45da-80bd-bcdd72a73770" containerName="main" containerID="cri-o://0fdc89f54fd70c209c6d1c18c1d520a7900fa0e3d73654294ede449d764192f6" gracePeriod=45 Nov 03 02:46:00.865558 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:00.865492 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-9aebe8/auth-test-model-9aebe8-kserve-949d578b5-l8chz" Nov 03 02:46:00.951957 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:00.951918 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d715d6df-5d86-45da-80bd-bcdd72a73770-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d715d6df-5d86-45da-80bd-bcdd72a73770-model-cache\") pod \"d715d6df-5d86-45da-80bd-bcdd72a73770\" (UID: \"d715d6df-5d86-45da-80bd-bcdd72a73770\") " Nov 03 02:46:00.952166 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:00.951982 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d715d6df-5d86-45da-80bd-bcdd72a73770-kube-api-access-cgjdw\" (UniqueName: \"kubernetes.io/projected/d715d6df-5d86-45da-80bd-bcdd72a73770-kube-api-access-cgjdw\") pod \"d715d6df-5d86-45da-80bd-bcdd72a73770\" (UID: \"d715d6df-5d86-45da-80bd-bcdd72a73770\") " Nov 03 02:46:00.952166 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:00.952096 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d715d6df-5d86-45da-80bd-bcdd72a73770-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d715d6df-5d86-45da-80bd-bcdd72a73770-tmp-dir\") pod \"d715d6df-5d86-45da-80bd-bcdd72a73770\" (UID: \"d715d6df-5d86-45da-80bd-bcdd72a73770\") " Nov 03 02:46:00.952295 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:00.952197 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d715d6df-5d86-45da-80bd-bcdd72a73770-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d715d6df-5d86-45da-80bd-bcdd72a73770-dshm\") pod \"d715d6df-5d86-45da-80bd-bcdd72a73770\" (UID: \"d715d6df-5d86-45da-80bd-bcdd72a73770\") " Nov 03 02:46:00.952295 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:00.952216 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d715d6df-5d86-45da-80bd-bcdd72a73770-model-cache" pod "d715d6df-5d86-45da-80bd-bcdd72a73770" (UID: "d715d6df-5d86-45da-80bd-bcdd72a73770"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:46:00.952295 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:00.952263 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d715d6df-5d86-45da-80bd-bcdd72a73770-home\" (UniqueName: \"kubernetes.io/empty-dir/d715d6df-5d86-45da-80bd-bcdd72a73770-home\") pod \"d715d6df-5d86-45da-80bd-bcdd72a73770\" (UID: \"d715d6df-5d86-45da-80bd-bcdd72a73770\") " Nov 03 02:46:00.952295 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:00.952284 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d715d6df-5d86-45da-80bd-bcdd72a73770-tls-certs\" (UniqueName: \"kubernetes.io/secret/d715d6df-5d86-45da-80bd-bcdd72a73770-tls-certs\") pod \"d715d6df-5d86-45da-80bd-bcdd72a73770\" (UID: \"d715d6df-5d86-45da-80bd-bcdd72a73770\") " Nov 03 02:46:00.952541 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:00.952389 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d715d6df-5d86-45da-80bd-bcdd72a73770-tmp-dir" pod "d715d6df-5d86-45da-80bd-bcdd72a73770" (UID: "d715d6df-5d86-45da-80bd-bcdd72a73770"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:46:00.952541 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:00.952524 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d715d6df-5d86-45da-80bd-bcdd72a73770-home" pod "d715d6df-5d86-45da-80bd-bcdd72a73770" (UID: "d715d6df-5d86-45da-80bd-bcdd72a73770"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:46:00.952745 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:00.952729 5605 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d715d6df-5d86-45da-80bd-bcdd72a73770-home\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:00.952785 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:00.952750 5605 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d715d6df-5d86-45da-80bd-bcdd72a73770-model-cache\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:00.952785 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:00.952760 5605 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d715d6df-5d86-45da-80bd-bcdd72a73770-tmp-dir\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:00.954478 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:00.954447 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d715d6df-5d86-45da-80bd-bcdd72a73770-kube-api-access-cgjdw" pod "d715d6df-5d86-45da-80bd-bcdd72a73770" (UID: "d715d6df-5d86-45da-80bd-bcdd72a73770"). InnerVolumeSpecName "kube-api-access-cgjdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:46:00.954478 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:00.954471 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d715d6df-5d86-45da-80bd-bcdd72a73770-tls-certs" pod "d715d6df-5d86-45da-80bd-bcdd72a73770" (UID: "d715d6df-5d86-45da-80bd-bcdd72a73770"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:46:00.954628 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:00.954531 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d715d6df-5d86-45da-80bd-bcdd72a73770-dshm" pod "d715d6df-5d86-45da-80bd-bcdd72a73770" (UID: "d715d6df-5d86-45da-80bd-bcdd72a73770"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:46:01.054455 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:01.054419 5605 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d715d6df-5d86-45da-80bd-bcdd72a73770-tls-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:01.054455 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:01.054452 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgjdw\" (UniqueName: \"kubernetes.io/projected/d715d6df-5d86-45da-80bd-bcdd72a73770-kube-api-access-cgjdw\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:01.054455 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:01.054465 5605 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d715d6df-5d86-45da-80bd-bcdd72a73770-dshm\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:01.346995 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:01.346961 5605 generic.go:356] "Generic (PLEG): container finished" podID="d715d6df-5d86-45da-80bd-bcdd72a73770" containerID="0fdc89f54fd70c209c6d1c18c1d520a7900fa0e3d73654294ede449d764192f6" exitCode=0 Nov 03 02:46:01.347190 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:01.347004 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-9aebe8/auth-test-model-9aebe8-kserve-949d578b5-l8chz" Nov 03 02:46:01.347190 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:01.347026 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-9aebe8/auth-test-model-9aebe8-kserve-949d578b5-l8chz" event={"ID":"d715d6df-5d86-45da-80bd-bcdd72a73770","Type":"ContainerDied","Data":"0fdc89f54fd70c209c6d1c18c1d520a7900fa0e3d73654294ede449d764192f6"} Nov 03 02:46:01.347190 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:01.347071 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-9aebe8/auth-test-model-9aebe8-kserve-949d578b5-l8chz" event={"ID":"d715d6df-5d86-45da-80bd-bcdd72a73770","Type":"ContainerDied","Data":"a5831fec3572096f8d31b357152497a401e9c3d87ef710176dd2e4808e4be48d"} Nov 03 02:46:01.347190 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:01.347088 5605 scope.go:122] "RemoveContainer" containerID="0fdc89f54fd70c209c6d1c18c1d520a7900fa0e3d73654294ede449d764192f6" Nov 03 02:46:01.357610 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:01.357587 5605 scope.go:122] "RemoveContainer" containerID="0fdc89f54fd70c209c6d1c18c1d520a7900fa0e3d73654294ede449d764192f6" Nov 03 02:46:01.357958 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:46:01.357933 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fdc89f54fd70c209c6d1c18c1d520a7900fa0e3d73654294ede449d764192f6\": container with ID starting with 0fdc89f54fd70c209c6d1c18c1d520a7900fa0e3d73654294ede449d764192f6 not found: ID does not exist" containerID="0fdc89f54fd70c209c6d1c18c1d520a7900fa0e3d73654294ede449d764192f6" Nov 03 02:46:01.358044 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:01.357965 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fdc89f54fd70c209c6d1c18c1d520a7900fa0e3d73654294ede449d764192f6"} err="failed to get container status \"0fdc89f54fd70c209c6d1c18c1d520a7900fa0e3d73654294ede449d764192f6\": rpc error: code = NotFound desc = could not find container \"0fdc89f54fd70c209c6d1c18c1d520a7900fa0e3d73654294ede449d764192f6\": container with ID starting with 0fdc89f54fd70c209c6d1c18c1d520a7900fa0e3d73654294ede449d764192f6 not found: ID does not exist" Nov 03 02:46:01.500907 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:01.500867 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-9aebe8/auth-test-model-9aebe8-kserve-949d578b5-l8chz"] Nov 03 02:46:01.547745 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:01.547708 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-9aebe8/auth-test-model-9aebe8-kserve-949d578b5-l8chz"] Nov 03 02:46:01.681898 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:01.681794 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-78b5c5/auth-test-model-78b5c5-kserve-5bc8bf4cf7-gptc4"] Nov 03 02:46:01.967025 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:01.966917 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7232-0e9ec9"] Nov 03 02:46:01.967689 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:01.967663 5605 cpu_manager.go:414] "containerMap: removing container" podUID="d715d6df-5d86-45da-80bd-bcdd72a73770" containerName="main" Nov 03 02:46:01.967689 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:01.967687 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d715d6df-5d86-45da-80bd-bcdd72a73770" containerName="main" Nov 03 02:46:01.967867 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:01.967720 5605 cpu_manager.go:414] "containerMap: removing container" podUID="eddf0af3-8acf-4e49-bd31-a8235ab2e03b" containerName="main" Nov 03 02:46:01.967867 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:01.967731 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eddf0af3-8acf-4e49-bd31-a8235ab2e03b" containerName="main" Nov 03 02:46:01.967933 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:01.967884 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="d715d6df-5d86-45da-80bd-bcdd72a73770" containerName="main" Nov 03 02:46:01.967933 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:01.967902 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="eddf0af3-8acf-4e49-bd31-a8235ab2e03b" containerName="main" Nov 03 02:46:01.971791 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:01.971773 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7232-0e9ec9" Nov 03 02:46:02.020047 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:02.020009 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7232-0e9ec9"] Nov 03 02:46:02.020530 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:02.020506 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-579jc\"" Nov 03 02:46:02.068470 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:02.068435 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv2s7\" (UniqueName: \"kubernetes.io/projected/93559923-4b62-4fa7-a7aa-9bbc8dbff381-kube-api-access-gv2s7\") pod \"test-curl-7232-0e9ec9\" (UID: \"93559923-4b62-4fa7-a7aa-9bbc8dbff381\") " pod="opendatahub/test-curl-7232-0e9ec9" Nov 03 02:46:02.170288 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:02.170239 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv2s7\" (UniqueName: \"kubernetes.io/projected/93559923-4b62-4fa7-a7aa-9bbc8dbff381-kube-api-access-gv2s7\") pod \"test-curl-7232-0e9ec9\" (UID: \"93559923-4b62-4fa7-a7aa-9bbc8dbff381\") " pod="opendatahub/test-curl-7232-0e9ec9" Nov 03 02:46:02.223560 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:02.223484 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv2s7\" (UniqueName: \"kubernetes.io/projected/93559923-4b62-4fa7-a7aa-9bbc8dbff381-kube-api-access-gv2s7\") pod \"test-curl-7232-0e9ec9\" (UID: \"93559923-4b62-4fa7-a7aa-9bbc8dbff381\") " pod="opendatahub/test-curl-7232-0e9ec9" Nov 03 02:46:02.284530 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:02.284495 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7232-0e9ec9" Nov 03 02:46:02.658154 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:02.658115 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7232-0e9ec9"] Nov 03 02:46:02.664435 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:46:02.664394 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod93559923_4b62_4fa7_a7aa_9bbc8dbff381.slice/crio-2ae2ce175b35fa4f82a11c60429c327216196fed6f35eb7c42e9b4d071254ac5 WatchSource:0}: Error finding container 2ae2ce175b35fa4f82a11c60429c327216196fed6f35eb7c42e9b4d071254ac5: Status 404 returned error can't find the container with id 2ae2ce175b35fa4f82a11c60429c327216196fed6f35eb7c42e9b4d071254ac5 Nov 03 02:46:02.851907 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:02.851872 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d715d6df-5d86-45da-80bd-bcdd72a73770" path="/var/lib/kubelet/pods/d715d6df-5d86-45da-80bd-bcdd72a73770/volumes" Nov 03 02:46:03.373095 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:03.373056 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7232-0e9ec9" event={"ID":"93559923-4b62-4fa7-a7aa-9bbc8dbff381","Type":"ContainerStarted","Data":"6dd6da982f023169af6aa9a4569b6293aeb4158e919c68d261a22d06669aead3"} Nov 03 02:46:03.373279 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:03.373105 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7232-0e9ec9" event={"ID":"93559923-4b62-4fa7-a7aa-9bbc8dbff381","Type":"ContainerStarted","Data":"2ae2ce175b35fa4f82a11c60429c327216196fed6f35eb7c42e9b4d071254ac5"} Nov 03 02:46:03.479096 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:03.479036 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7232-0e9ec9" podStartSLOduration=2.479021883 podStartE2EDuration="2.479021883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:46:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:46:03.478008897 +0000 UTC m=+3240.737207050" watchObservedRunningTime="2026-11-03 02:46:03.479021883 +0000 UTC m=+3240.738220037" Nov 03 02:46:04.937785 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:04.937735 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7232-0e9ec9"] Nov 03 02:46:04.940694 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:04.940665 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7232-0e9ec9"] Nov 03 02:46:05.119049 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:05.119010 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7232-b72706"] Nov 03 02:46:05.119552 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:05.119537 5605 cpu_manager.go:414] "containerMap: removing container" podUID="93559923-4b62-4fa7-a7aa-9bbc8dbff381" containerName="test-curl-7232-0e9ec9" Nov 03 02:46:05.119606 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:05.119555 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93559923-4b62-4fa7-a7aa-9bbc8dbff381" containerName="test-curl-7232-0e9ec9" Nov 03 02:46:05.119663 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:05.119652 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="93559923-4b62-4fa7-a7aa-9bbc8dbff381" containerName="test-curl-7232-0e9ec9" Nov 03 02:46:05.124541 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:05.124518 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7232-b72706" Nov 03 02:46:05.174729 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:05.174698 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7232-b72706"] Nov 03 02:46:05.206268 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:05.206164 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgcpd\" (UniqueName: \"kubernetes.io/projected/02e6ee73-d18c-4804-b6c1-6198b08b9a82-kube-api-access-xgcpd\") pod \"test-curl-7232-b72706\" (UID: \"02e6ee73-d18c-4804-b6c1-6198b08b9a82\") " pod="opendatahub/test-curl-7232-b72706" Nov 03 02:46:05.307120 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:05.307078 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgcpd\" (UniqueName: \"kubernetes.io/projected/02e6ee73-d18c-4804-b6c1-6198b08b9a82-kube-api-access-xgcpd\") pod \"test-curl-7232-b72706\" (UID: \"02e6ee73-d18c-4804-b6c1-6198b08b9a82\") " pod="opendatahub/test-curl-7232-b72706" Nov 03 02:46:05.360828 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:05.360781 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgcpd\" (UniqueName: \"kubernetes.io/projected/02e6ee73-d18c-4804-b6c1-6198b08b9a82-kube-api-access-xgcpd\") pod \"test-curl-7232-b72706\" (UID: \"02e6ee73-d18c-4804-b6c1-6198b08b9a82\") " pod="opendatahub/test-curl-7232-b72706" Nov 03 02:46:05.385190 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:05.385147 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7232-0e9ec9" podUID="93559923-4b62-4fa7-a7aa-9bbc8dbff381" containerName="test-curl-7232-0e9ec9" containerID="cri-o://6dd6da982f023169af6aa9a4569b6293aeb4158e919c68d261a22d06669aead3" gracePeriod=2 Nov 03 02:46:05.433020 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:46:05.432974 5605 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7232-0e9ec9\" is forbidden: User \"system:node:ip-10-0-140-140.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-140.us-west-2.compute.internal' and this object" podUID="93559923-4b62-4fa7-a7aa-9bbc8dbff381" pod="opendatahub/test-curl-7232-0e9ec9" Nov 03 02:46:05.436485 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:05.436459 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7232-b72706" Nov 03 02:46:05.805368 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:05.805326 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7232-b72706"] Nov 03 02:46:05.811985 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:46:05.811950 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02e6ee73_d18c_4804_b6c1_6198b08b9a82.slice/crio-568c6bf272062a08b954f05711942b9ea289bae75c1cb7a1845d00cd27244124 WatchSource:0}: Error finding container 568c6bf272062a08b954f05711942b9ea289bae75c1cb7a1845d00cd27244124: Status 404 returned error can't find the container with id 568c6bf272062a08b954f05711942b9ea289bae75c1cb7a1845d00cd27244124 Nov 03 02:46:06.393326 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:06.393284 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7232-b72706" event={"ID":"02e6ee73-d18c-4804-b6c1-6198b08b9a82","Type":"ContainerStarted","Data":"f8123243ae78cc6b1f54f972e3a3f22e28d27ead8ea5feaf5d56a967bef18599"} Nov 03 02:46:06.393734 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:06.393335 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7232-b72706" event={"ID":"02e6ee73-d18c-4804-b6c1-6198b08b9a82","Type":"ContainerStarted","Data":"568c6bf272062a08b954f05711942b9ea289bae75c1cb7a1845d00cd27244124"} Nov 03 02:46:06.441091 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:46:06.441053 5605 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7232-0e9ec9\" is forbidden: User \"system:node:ip-10-0-140-140.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-140.us-west-2.compute.internal' and this object" podUID="93559923-4b62-4fa7-a7aa-9bbc8dbff381" pod="opendatahub/test-curl-7232-0e9ec9" Nov 03 02:46:06.546013 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:06.545957 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7232-b72706" podStartSLOduration=1.545942207 podStartE2EDuration="1.545942207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:46:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:46:06.544779597 +0000 UTC m=+3243.803977751" watchObservedRunningTime="2026-11-03 02:46:06.545942207 +0000 UTC m=+3243.805140360" Nov 03 02:46:07.607849 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:07.607819 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7232-0e9ec9" Nov 03 02:46:07.737824 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:07.737743 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93559923-4b62-4fa7-a7aa-9bbc8dbff381-kube-api-access-gv2s7\" (UniqueName: \"kubernetes.io/projected/93559923-4b62-4fa7-a7aa-9bbc8dbff381-kube-api-access-gv2s7\") pod \"93559923-4b62-4fa7-a7aa-9bbc8dbff381\" (UID: \"93559923-4b62-4fa7-a7aa-9bbc8dbff381\") " Nov 03 02:46:07.739707 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:07.739677 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93559923-4b62-4fa7-a7aa-9bbc8dbff381-kube-api-access-gv2s7" pod "93559923-4b62-4fa7-a7aa-9bbc8dbff381" (UID: "93559923-4b62-4fa7-a7aa-9bbc8dbff381"). InnerVolumeSpecName "kube-api-access-gv2s7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:46:07.838899 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:07.838865 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv2s7\" (UniqueName: \"kubernetes.io/projected/93559923-4b62-4fa7-a7aa-9bbc8dbff381-kube-api-access-gv2s7\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:07.905491 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:07.905462 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7232-b72706"] Nov 03 02:46:07.910497 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:07.910467 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7232-b72706"] Nov 03 02:46:08.062044 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:08.062015 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7232-3a363f"] Nov 03 02:46:08.062494 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:08.062481 5605 cpu_manager.go:414] "containerMap: removing container" podUID="02e6ee73-d18c-4804-b6c1-6198b08b9a82" containerName="test-curl-7232-b72706" Nov 03 02:46:08.062553 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:08.062495 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02e6ee73-d18c-4804-b6c1-6198b08b9a82" containerName="test-curl-7232-b72706" Nov 03 02:46:08.062627 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:08.062618 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="02e6ee73-d18c-4804-b6c1-6198b08b9a82" containerName="test-curl-7232-b72706" Nov 03 02:46:08.065799 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:08.065779 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7232-3a363f" Nov 03 02:46:08.116740 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:08.116716 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7232-3a363f"] Nov 03 02:46:08.142322 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:08.142298 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b77ds\" (UniqueName: \"kubernetes.io/projected/7085212b-ea42-42ec-94f1-17d236621431-kube-api-access-b77ds\") pod \"test-curl-7232-3a363f\" (UID: \"7085212b-ea42-42ec-94f1-17d236621431\") " pod="opendatahub/test-curl-7232-3a363f" Nov 03 02:46:08.243557 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:08.243525 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b77ds\" (UniqueName: \"kubernetes.io/projected/7085212b-ea42-42ec-94f1-17d236621431-kube-api-access-b77ds\") pod \"test-curl-7232-3a363f\" (UID: \"7085212b-ea42-42ec-94f1-17d236621431\") " pod="opendatahub/test-curl-7232-3a363f" Nov 03 02:46:08.296761 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:08.296725 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b77ds\" (UniqueName: \"kubernetes.io/projected/7085212b-ea42-42ec-94f1-17d236621431-kube-api-access-b77ds\") pod \"test-curl-7232-3a363f\" (UID: \"7085212b-ea42-42ec-94f1-17d236621431\") " pod="opendatahub/test-curl-7232-3a363f" Nov 03 02:46:08.377764 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:08.377685 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7232-3a363f" Nov 03 02:46:08.407568 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:08.407537 5605 generic.go:356] "Generic (PLEG): container finished" podID="93559923-4b62-4fa7-a7aa-9bbc8dbff381" containerID="6dd6da982f023169af6aa9a4569b6293aeb4158e919c68d261a22d06669aead3" exitCode=137 Nov 03 02:46:08.407710 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:08.407606 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7232-0e9ec9" Nov 03 02:46:08.407835 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:08.407807 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7232-b72706" podUID="02e6ee73-d18c-4804-b6c1-6198b08b9a82" containerName="test-curl-7232-b72706" containerID="cri-o://f8123243ae78cc6b1f54f972e3a3f22e28d27ead8ea5feaf5d56a967bef18599" gracePeriod=2 Nov 03 02:46:08.408100 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:08.407919 5605 scope.go:122] "RemoveContainer" containerID="6dd6da982f023169af6aa9a4569b6293aeb4158e919c68d261a22d06669aead3" Nov 03 02:46:08.421235 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:08.421039 5605 scope.go:122] "RemoveContainer" containerID="6dd6da982f023169af6aa9a4569b6293aeb4158e919c68d261a22d06669aead3" Nov 03 02:46:08.421471 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:46:08.421439 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dd6da982f023169af6aa9a4569b6293aeb4158e919c68d261a22d06669aead3\": container with ID starting with 6dd6da982f023169af6aa9a4569b6293aeb4158e919c68d261a22d06669aead3 not found: ID does not exist" containerID="6dd6da982f023169af6aa9a4569b6293aeb4158e919c68d261a22d06669aead3" Nov 03 02:46:08.421586 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:08.421479 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dd6da982f023169af6aa9a4569b6293aeb4158e919c68d261a22d06669aead3"} err="failed to get container status \"6dd6da982f023169af6aa9a4569b6293aeb4158e919c68d261a22d06669aead3\": rpc error: code = NotFound desc = could not find container \"6dd6da982f023169af6aa9a4569b6293aeb4158e919c68d261a22d06669aead3\": container with ID starting with 6dd6da982f023169af6aa9a4569b6293aeb4158e919c68d261a22d06669aead3 not found: ID does not exist" Nov 03 02:46:08.455667 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:46:08.455630 5605 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7232-b72706\" is forbidden: User \"system:node:ip-10-0-140-140.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-140.us-west-2.compute.internal' and this object" podUID="02e6ee73-d18c-4804-b6c1-6198b08b9a82" pod="opendatahub/test-curl-7232-b72706" Nov 03 02:46:08.503320 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:46:08.503288 5605 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7232-0e9ec9\" is forbidden: User \"system:node:ip-10-0-140-140.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-140.us-west-2.compute.internal' and this object" podUID="93559923-4b62-4fa7-a7aa-9bbc8dbff381" pod="opendatahub/test-curl-7232-0e9ec9" Nov 03 02:46:08.550721 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:46:08.550683 5605 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7232-b72706\" is forbidden: User \"system:node:ip-10-0-140-140.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-140.us-west-2.compute.internal' and this object" podUID="02e6ee73-d18c-4804-b6c1-6198b08b9a82" pod="opendatahub/test-curl-7232-b72706" Nov 03 02:46:08.748907 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:08.748819 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7232-3a363f"] Nov 03 02:46:08.754317 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:46:08.754285 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7085212b_ea42_42ec_94f1_17d236621431.slice/crio-5a53424ad252463461a117c8339a2131f352184e43df9fbab22c47a80b5b5062 WatchSource:0}: Error finding container 5a53424ad252463461a117c8339a2131f352184e43df9fbab22c47a80b5b5062: Status 404 returned error can't find the container with id 5a53424ad252463461a117c8339a2131f352184e43df9fbab22c47a80b5b5062 Nov 03 02:46:08.850254 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:08.850223 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93559923-4b62-4fa7-a7aa-9bbc8dbff381" path="/var/lib/kubelet/pods/93559923-4b62-4fa7-a7aa-9bbc8dbff381/volumes" Nov 03 02:46:09.414707 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:09.414669 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7232-3a363f" event={"ID":"7085212b-ea42-42ec-94f1-17d236621431","Type":"ContainerStarted","Data":"88d37e23e60fb2e50f81d9256dcf20f22891ed9d688caaf676672a2262778e2e"} Nov 03 02:46:09.414707 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:09.414718 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7232-3a363f" event={"ID":"7085212b-ea42-42ec-94f1-17d236621431","Type":"ContainerStarted","Data":"5a53424ad252463461a117c8339a2131f352184e43df9fbab22c47a80b5b5062"} Nov 03 02:46:09.521346 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:09.521293 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7232-3a363f" podStartSLOduration=1.521279917 podStartE2EDuration="1.521279917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:46:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:46:09.518651713 +0000 UTC m=+3246.777849866" watchObservedRunningTime="2026-11-03 02:46:09.521279917 +0000 UTC m=+3246.780478071" Nov 03 02:46:10.265883 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:10.265849 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-ffba43/auth-test-model-ffba43-kserve-5bddd45877-ln2tc"] Nov 03 02:46:10.670105 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:10.670078 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7232-b72706" Nov 03 02:46:10.772484 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:10.772449 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02e6ee73-d18c-4804-b6c1-6198b08b9a82-kube-api-access-xgcpd\" (UniqueName: \"kubernetes.io/projected/02e6ee73-d18c-4804-b6c1-6198b08b9a82-kube-api-access-xgcpd\") pod \"02e6ee73-d18c-4804-b6c1-6198b08b9a82\" (UID: \"02e6ee73-d18c-4804-b6c1-6198b08b9a82\") " Nov 03 02:46:10.774539 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:10.774509 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02e6ee73-d18c-4804-b6c1-6198b08b9a82-kube-api-access-xgcpd" pod "02e6ee73-d18c-4804-b6c1-6198b08b9a82" (UID: "02e6ee73-d18c-4804-b6c1-6198b08b9a82"). InnerVolumeSpecName "kube-api-access-xgcpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:46:10.850431 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:10.850335 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02e6ee73-d18c-4804-b6c1-6198b08b9a82" path="/var/lib/kubelet/pods/02e6ee73-d18c-4804-b6c1-6198b08b9a82/volumes" Nov 03 02:46:10.874208 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:10.874174 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgcpd\" (UniqueName: \"kubernetes.io/projected/02e6ee73-d18c-4804-b6c1-6198b08b9a82-kube-api-access-xgcpd\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:10.963462 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:10.963433 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7232-3a363f"] Nov 03 02:46:10.967395 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:10.967375 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7232-3a363f"] Nov 03 02:46:11.127160 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:11.127089 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7232-26e0d0"] Nov 03 02:46:11.127587 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:11.127573 5605 cpu_manager.go:414] "containerMap: removing container" podUID="7085212b-ea42-42ec-94f1-17d236621431" containerName="test-curl-7232-3a363f" Nov 03 02:46:11.127634 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:11.127588 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7085212b-ea42-42ec-94f1-17d236621431" containerName="test-curl-7232-3a363f" Nov 03 02:46:11.127687 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:11.127678 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="7085212b-ea42-42ec-94f1-17d236621431" containerName="test-curl-7232-3a363f" Nov 03 02:46:11.131064 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:11.131045 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7232-26e0d0" Nov 03 02:46:11.198276 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:11.198232 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7232-26e0d0"] Nov 03 02:46:11.301002 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:11.300970 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntv9c\" (UniqueName: \"kubernetes.io/projected/15fd430f-bf80-40da-b9b9-9f1ef4f040e0-kube-api-access-ntv9c\") pod \"test-curl-7232-26e0d0\" (UID: \"15fd430f-bf80-40da-b9b9-9f1ef4f040e0\") " pod="opendatahub/test-curl-7232-26e0d0" Nov 03 02:46:11.402341 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:11.402248 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntv9c\" (UniqueName: \"kubernetes.io/projected/15fd430f-bf80-40da-b9b9-9f1ef4f040e0-kube-api-access-ntv9c\") pod \"test-curl-7232-26e0d0\" (UID: \"15fd430f-bf80-40da-b9b9-9f1ef4f040e0\") " pod="opendatahub/test-curl-7232-26e0d0" Nov 03 02:46:11.434302 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:11.434267 5605 generic.go:356] "Generic (PLEG): container finished" podID="02e6ee73-d18c-4804-b6c1-6198b08b9a82" containerID="f8123243ae78cc6b1f54f972e3a3f22e28d27ead8ea5feaf5d56a967bef18599" exitCode=137 Nov 03 02:46:11.434509 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:11.434310 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7232-b72706" Nov 03 02:46:11.434509 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:11.434419 5605 scope.go:122] "RemoveContainer" containerID="f8123243ae78cc6b1f54f972e3a3f22e28d27ead8ea5feaf5d56a967bef18599" Nov 03 02:46:11.434626 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:11.434503 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7232-3a363f" podUID="7085212b-ea42-42ec-94f1-17d236621431" containerName="test-curl-7232-3a363f" containerID="cri-o://88d37e23e60fb2e50f81d9256dcf20f22891ed9d688caaf676672a2262778e2e" gracePeriod=2 Nov 03 02:46:11.444548 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:11.444524 5605 scope.go:122] "RemoveContainer" containerID="f8123243ae78cc6b1f54f972e3a3f22e28d27ead8ea5feaf5d56a967bef18599" Nov 03 02:46:11.444833 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:46:11.444814 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8123243ae78cc6b1f54f972e3a3f22e28d27ead8ea5feaf5d56a967bef18599\": container with ID starting with f8123243ae78cc6b1f54f972e3a3f22e28d27ead8ea5feaf5d56a967bef18599 not found: ID does not exist" containerID="f8123243ae78cc6b1f54f972e3a3f22e28d27ead8ea5feaf5d56a967bef18599" Nov 03 02:46:11.444905 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:11.444840 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8123243ae78cc6b1f54f972e3a3f22e28d27ead8ea5feaf5d56a967bef18599"} err="failed to get container status \"f8123243ae78cc6b1f54f972e3a3f22e28d27ead8ea5feaf5d56a967bef18599\": rpc error: code = NotFound desc = could not find container \"f8123243ae78cc6b1f54f972e3a3f22e28d27ead8ea5feaf5d56a967bef18599\": container with ID starting with f8123243ae78cc6b1f54f972e3a3f22e28d27ead8ea5feaf5d56a967bef18599 not found: ID does not exist" Nov 03 02:46:11.461918 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:11.461892 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntv9c\" (UniqueName: \"kubernetes.io/projected/15fd430f-bf80-40da-b9b9-9f1ef4f040e0-kube-api-access-ntv9c\") pod \"test-curl-7232-26e0d0\" (UID: \"15fd430f-bf80-40da-b9b9-9f1ef4f040e0\") " pod="opendatahub/test-curl-7232-26e0d0" Nov 03 02:46:11.482231 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:46:11.482205 5605 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7232-3a363f\" is forbidden: User \"system:node:ip-10-0-140-140.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-140.us-west-2.compute.internal' and this object" podUID="7085212b-ea42-42ec-94f1-17d236621431" pod="opendatahub/test-curl-7232-3a363f" Nov 03 02:46:11.529766 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:46:11.529734 5605 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7232-3a363f\" is forbidden: User \"system:node:ip-10-0-140-140.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-140.us-west-2.compute.internal' and this object" podUID="7085212b-ea42-42ec-94f1-17d236621431" pod="opendatahub/test-curl-7232-3a363f" Nov 03 02:46:11.742833 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:11.742748 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7232-26e0d0" Nov 03 02:46:12.111593 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:12.111559 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7232-26e0d0"] Nov 03 02:46:12.116951 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:46:12.116924 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod15fd430f_bf80_40da_b9b9_9f1ef4f040e0.slice/crio-2b15d36561b777c635be05521cc3ff9bc28940e16b7405e23ba8d153a07f5f6b WatchSource:0}: Error finding container 2b15d36561b777c635be05521cc3ff9bc28940e16b7405e23ba8d153a07f5f6b: Status 404 returned error can't find the container with id 2b15d36561b777c635be05521cc3ff9bc28940e16b7405e23ba8d153a07f5f6b Nov 03 02:46:12.453283 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:12.453177 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7232-26e0d0" event={"ID":"15fd430f-bf80-40da-b9b9-9f1ef4f040e0","Type":"ContainerStarted","Data":"91627a4e87cd6b6e0f901a6d4505ce3d46c490c6623f7ec81fc188d5d0223af3"} Nov 03 02:46:12.453283 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:12.453236 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7232-26e0d0" event={"ID":"15fd430f-bf80-40da-b9b9-9f1ef4f040e0","Type":"ContainerStarted","Data":"2b15d36561b777c635be05521cc3ff9bc28940e16b7405e23ba8d153a07f5f6b"} Nov 03 02:46:12.565577 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:12.565508 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7232-26e0d0" podStartSLOduration=1.565486932 podStartE2EDuration="1.565486932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:46:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:46:12.564335961 +0000 UTC m=+3249.823534116" watchObservedRunningTime="2026-11-03 02:46:12.565486932 +0000 UTC m=+3249.824685086" Nov 03 02:46:12.615067 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:46:12.615015 5605 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7232-3a363f\" is forbidden: User \"system:node:ip-10-0-140-140.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-140.us-west-2.compute.internal' and this object" podUID="7085212b-ea42-42ec-94f1-17d236621431" pod="opendatahub/test-curl-7232-3a363f" Nov 03 02:46:12.619899 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:12.619866 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2"] Nov 03 02:46:12.625208 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:12.625173 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2" Nov 03 02:46:12.700118 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:12.686128 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2"] Nov 03 02:46:12.700118 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:46:12.686287 5605 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7232-3a363f\" is forbidden: User \"system:node:ip-10-0-140-140.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-140.us-west-2.compute.internal' and this object" podUID="7085212b-ea42-42ec-94f1-17d236621431" pod="opendatahub/test-curl-7232-3a363f" Nov 03 02:46:12.727224 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:12.727143 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-ca6ef4-openshift-default-dockercfg-bt9m5\"" Nov 03 02:46:12.731696 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:12.731669 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/955ee3cb-72bd-4dab-91ce-faffcbbe130d-istio-envoy\") pod \"e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2\" (UID: \"955ee3cb-72bd-4dab-91ce-faffcbbe130d\") " pod="openshift-ingress/e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2" Nov 03 02:46:12.731747 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:12.731727 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/955ee3cb-72bd-4dab-91ce-faffcbbe130d-istio-podinfo\") pod \"e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2\" (UID: \"955ee3cb-72bd-4dab-91ce-faffcbbe130d\") " pod="openshift-ingress/e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2" Nov 03 02:46:12.731879 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:12.731857 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/955ee3cb-72bd-4dab-91ce-faffcbbe130d-credential-socket\") pod \"e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2\" (UID: \"955ee3cb-72bd-4dab-91ce-faffcbbe130d\") " pod="openshift-ingress/e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2" Nov 03 02:46:12.732024 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:12.732001 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/955ee3cb-72bd-4dab-91ce-faffcbbe130d-workload-socket\") pod \"e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2\" (UID: \"955ee3cb-72bd-4dab-91ce-faffcbbe130d\") " pod="openshift-ingress/e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2" Nov 03 02:46:12.732075 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:12.732063 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/955ee3cb-72bd-4dab-91ce-faffcbbe130d-istiod-ca-cert\") pod \"e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2\" (UID: \"955ee3cb-72bd-4dab-91ce-faffcbbe130d\") " pod="openshift-ingress/e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2" Nov 03 02:46:12.732123 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:12.732107 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/955ee3cb-72bd-4dab-91ce-faffcbbe130d-workload-certs\") pod \"e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2\" (UID: \"955ee3cb-72bd-4dab-91ce-faffcbbe130d\") " pod="openshift-ingress/e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2" Nov 03 02:46:12.732194 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:12.732180 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/955ee3cb-72bd-4dab-91ce-faffcbbe130d-istio-data\") pod \"e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2\" (UID: \"955ee3cb-72bd-4dab-91ce-faffcbbe130d\") " pod="openshift-ingress/e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2" Nov 03 02:46:12.732234 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:12.732208 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/955ee3cb-72bd-4dab-91ce-faffcbbe130d-istio-token\") pod \"e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2\" (UID: \"955ee3cb-72bd-4dab-91ce-faffcbbe130d\") " pod="openshift-ingress/e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2" Nov 03 02:46:12.732320 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:12.732307 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k6p7\" (UniqueName: \"kubernetes.io/projected/955ee3cb-72bd-4dab-91ce-faffcbbe130d-kube-api-access-6k6p7\") pod \"e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2\" (UID: \"955ee3cb-72bd-4dab-91ce-faffcbbe130d\") " pod="openshift-ingress/e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2" Nov 03 02:46:12.847021 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:12.846971 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/955ee3cb-72bd-4dab-91ce-faffcbbe130d-istio-podinfo\") pod \"e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2\" (UID: \"955ee3cb-72bd-4dab-91ce-faffcbbe130d\") " pod="openshift-ingress/e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2" Nov 03 02:46:12.847394 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:12.847350 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/955ee3cb-72bd-4dab-91ce-faffcbbe130d-credential-socket\") pod \"e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2\" (UID: \"955ee3cb-72bd-4dab-91ce-faffcbbe130d\") " pod="openshift-ingress/e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2" Nov 03 02:46:12.847652 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:12.847634 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/955ee3cb-72bd-4dab-91ce-faffcbbe130d-workload-socket\") pod \"e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2\" (UID: \"955ee3cb-72bd-4dab-91ce-faffcbbe130d\") " pod="openshift-ingress/e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2" Nov 03 02:46:12.847802 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:12.847785 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/955ee3cb-72bd-4dab-91ce-faffcbbe130d-istiod-ca-cert\") pod \"e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2\" (UID: \"955ee3cb-72bd-4dab-91ce-faffcbbe130d\") " pod="openshift-ingress/e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2" Nov 03 02:46:12.847974 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:12.847957 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/955ee3cb-72bd-4dab-91ce-faffcbbe130d-workload-certs\") pod \"e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2\" (UID: \"955ee3cb-72bd-4dab-91ce-faffcbbe130d\") " pod="openshift-ingress/e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2" Nov 03 02:46:12.848123 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:12.848108 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/955ee3cb-72bd-4dab-91ce-faffcbbe130d-istio-data\") pod \"e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2\" (UID: \"955ee3cb-72bd-4dab-91ce-faffcbbe130d\") " pod="openshift-ingress/e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2" Nov 03 02:46:12.848260 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:12.848245 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/955ee3cb-72bd-4dab-91ce-faffcbbe130d-istio-token\") pod \"e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2\" (UID: \"955ee3cb-72bd-4dab-91ce-faffcbbe130d\") " pod="openshift-ingress/e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2" Nov 03 02:46:12.848484 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:12.848437 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k6p7\" (UniqueName: \"kubernetes.io/projected/955ee3cb-72bd-4dab-91ce-faffcbbe130d-kube-api-access-6k6p7\") pod \"e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2\" (UID: \"955ee3cb-72bd-4dab-91ce-faffcbbe130d\") " pod="openshift-ingress/e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2" Nov 03 02:46:12.848576 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:12.848526 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/955ee3cb-72bd-4dab-91ce-faffcbbe130d-istio-envoy\") pod \"e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2\" (UID: \"955ee3cb-72bd-4dab-91ce-faffcbbe130d\") " pod="openshift-ingress/e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2" Nov 03 02:46:12.848914 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:12.848887 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/955ee3cb-72bd-4dab-91ce-faffcbbe130d-workload-socket\") pod \"e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2\" (UID: \"955ee3cb-72bd-4dab-91ce-faffcbbe130d\") " pod="openshift-ingress/e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2" Nov 03 02:46:12.849128 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:12.848935 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/955ee3cb-72bd-4dab-91ce-faffcbbe130d-istio-data\") pod \"e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2\" (UID: \"955ee3cb-72bd-4dab-91ce-faffcbbe130d\") " pod="openshift-ingress/e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2" Nov 03 02:46:12.850585 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:12.849580 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/955ee3cb-72bd-4dab-91ce-faffcbbe130d-istiod-ca-cert\") pod \"e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2\" (UID: \"955ee3cb-72bd-4dab-91ce-faffcbbe130d\") " pod="openshift-ingress/e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2" Nov 03 02:46:12.850585 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:12.850173 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/955ee3cb-72bd-4dab-91ce-faffcbbe130d-credential-socket\") pod \"e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2\" (UID: \"955ee3cb-72bd-4dab-91ce-faffcbbe130d\") " pod="openshift-ingress/e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2" Nov 03 02:46:12.850917 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:12.850549 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/955ee3cb-72bd-4dab-91ce-faffcbbe130d-workload-certs\") pod \"e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2\" (UID: \"955ee3cb-72bd-4dab-91ce-faffcbbe130d\") " pod="openshift-ingress/e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2" Nov 03 02:46:12.853499 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:12.853447 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/955ee3cb-72bd-4dab-91ce-faffcbbe130d-istio-envoy\") pod \"e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2\" (UID: \"955ee3cb-72bd-4dab-91ce-faffcbbe130d\") " pod="openshift-ingress/e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2" Nov 03 02:46:12.854494 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:12.854435 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/955ee3cb-72bd-4dab-91ce-faffcbbe130d-istio-podinfo\") pod \"e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2\" (UID: \"955ee3cb-72bd-4dab-91ce-faffcbbe130d\") " pod="openshift-ingress/e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2" Nov 03 02:46:12.898579 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:46:12.898541 5605 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7232-3a363f\" is forbidden: User \"system:node:ip-10-0-140-140.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-140.us-west-2.compute.internal' and this object" podUID="7085212b-ea42-42ec-94f1-17d236621431" pod="opendatahub/test-curl-7232-3a363f" Nov 03 02:46:12.903003 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:12.902975 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/955ee3cb-72bd-4dab-91ce-faffcbbe130d-istio-token\") pod \"e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2\" (UID: \"955ee3cb-72bd-4dab-91ce-faffcbbe130d\") " pod="openshift-ingress/e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2" Nov 03 02:46:12.903329 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:12.903309 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k6p7\" (UniqueName: \"kubernetes.io/projected/955ee3cb-72bd-4dab-91ce-faffcbbe130d-kube-api-access-6k6p7\") pod \"e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2\" (UID: \"955ee3cb-72bd-4dab-91ce-faffcbbe130d\") " pod="openshift-ingress/e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2" Nov 03 02:46:12.952379 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:12.949559 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2" Nov 03 02:46:13.433348 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:13.433313 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2"] Nov 03 02:46:13.438883 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:46:13.438849 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod955ee3cb_72bd_4dab_91ce_faffcbbe130d.slice/crio-856fcde7324f4527597b1334ff70403342fb1109367f06f03e67bc1436b18fb1 WatchSource:0}: Error finding container 856fcde7324f4527597b1334ff70403342fb1109367f06f03e67bc1436b18fb1: Status 404 returned error can't find the container with id 856fcde7324f4527597b1334ff70403342fb1109367f06f03e67bc1436b18fb1 Nov 03 02:46:13.442131 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:13.442094 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:46:13.442197 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:13.442163 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:46:13.442231 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:13.442194 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:46:13.485052 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:13.485017 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2" event={"ID":"955ee3cb-72bd-4dab-91ce-faffcbbe130d","Type":"ContainerStarted","Data":"856fcde7324f4527597b1334ff70403342fb1109367f06f03e67bc1436b18fb1"} Nov 03 02:46:13.701203 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:13.701121 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7232-3a363f" Nov 03 02:46:13.891380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:13.883851 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7085212b-ea42-42ec-94f1-17d236621431-kube-api-access-b77ds\" (UniqueName: \"kubernetes.io/projected/7085212b-ea42-42ec-94f1-17d236621431-kube-api-access-b77ds\") pod \"7085212b-ea42-42ec-94f1-17d236621431\" (UID: \"7085212b-ea42-42ec-94f1-17d236621431\") " Nov 03 02:46:13.891985 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:13.891947 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7085212b-ea42-42ec-94f1-17d236621431-kube-api-access-b77ds" pod "7085212b-ea42-42ec-94f1-17d236621431" (UID: "7085212b-ea42-42ec-94f1-17d236621431"). InnerVolumeSpecName "kube-api-access-b77ds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:46:13.924399 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:13.917288 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7232-26e0d0"] Nov 03 02:46:13.943614 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:13.943527 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7232-26e0d0"] Nov 03 02:46:13.991386 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:13.990958 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b77ds\" (UniqueName: \"kubernetes.io/projected/7085212b-ea42-42ec-94f1-17d236621431-kube-api-access-b77ds\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:14.084383 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:14.083541 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7232-bff49a"] Nov 03 02:46:14.084383 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:14.084323 5605 cpu_manager.go:414] "containerMap: removing container" podUID="15fd430f-bf80-40da-b9b9-9f1ef4f040e0" containerName="test-curl-7232-26e0d0" Nov 03 02:46:14.084383 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:14.084344 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15fd430f-bf80-40da-b9b9-9f1ef4f040e0" containerName="test-curl-7232-26e0d0" Nov 03 02:46:14.084781 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:14.084547 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="15fd430f-bf80-40da-b9b9-9f1ef4f040e0" containerName="test-curl-7232-26e0d0" Nov 03 02:46:14.088378 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:14.088135 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7232-bff49a" Nov 03 02:46:14.117255 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:14.117229 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7232-bff49a"] Nov 03 02:46:14.192978 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:14.192936 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktws5\" (UniqueName: \"kubernetes.io/projected/aaa737a3-6588-44bd-a759-d082cdf196e1-kube-api-access-ktws5\") pod \"test-curl-7232-bff49a\" (UID: \"aaa737a3-6588-44bd-a759-d082cdf196e1\") " pod="opendatahub/test-curl-7232-bff49a" Nov 03 02:46:14.294708 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:14.294626 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktws5\" (UniqueName: \"kubernetes.io/projected/aaa737a3-6588-44bd-a759-d082cdf196e1-kube-api-access-ktws5\") pod \"test-curl-7232-bff49a\" (UID: \"aaa737a3-6588-44bd-a759-d082cdf196e1\") " pod="opendatahub/test-curl-7232-bff49a" Nov 03 02:46:14.348083 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:14.348051 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktws5\" (UniqueName: \"kubernetes.io/projected/aaa737a3-6588-44bd-a759-d082cdf196e1-kube-api-access-ktws5\") pod \"test-curl-7232-bff49a\" (UID: \"aaa737a3-6588-44bd-a759-d082cdf196e1\") " pod="opendatahub/test-curl-7232-bff49a" Nov 03 02:46:14.407748 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:14.407721 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7232-bff49a" Nov 03 02:46:14.490382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:14.489938 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4"] Nov 03 02:46:14.490382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:14.490186 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4" podUID="33cae564-16a9-42bb-a1a5-dc5c3257f56b" containerName="istio-proxy" containerID="cri-o://eea83ba4c269a0fb4f25276b7d3c175be50b5f2d8f90be9f333bcf1dc73f65e3" gracePeriod=30 Nov 03 02:46:14.517410 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:14.510554 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2" event={"ID":"955ee3cb-72bd-4dab-91ce-faffcbbe130d","Type":"ContainerStarted","Data":"7584787c15af69972b9bda79ce98e2f7221feeb167c253d85c111ee287ddfcd9"} Nov 03 02:46:14.517410 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:14.512385 5605 generic.go:356] "Generic (PLEG): container finished" podID="7085212b-ea42-42ec-94f1-17d236621431" containerID="88d37e23e60fb2e50f81d9256dcf20f22891ed9d688caaf676672a2262778e2e" exitCode=137 Nov 03 02:46:14.517410 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:14.512612 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7232-26e0d0" podUID="15fd430f-bf80-40da-b9b9-9f1ef4f040e0" containerName="test-curl-7232-26e0d0" containerID="cri-o://91627a4e87cd6b6e0f901a6d4505ce3d46c490c6623f7ec81fc188d5d0223af3" gracePeriod=2 Nov 03 02:46:14.517410 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:14.512764 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7232-3a363f" Nov 03 02:46:14.517410 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:14.513423 5605 scope.go:122] "RemoveContainer" containerID="88d37e23e60fb2e50f81d9256dcf20f22891ed9d688caaf676672a2262778e2e" Nov 03 02:46:14.553484 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:14.550189 5605 scope.go:122] "RemoveContainer" containerID="88d37e23e60fb2e50f81d9256dcf20f22891ed9d688caaf676672a2262778e2e" Nov 03 02:46:14.553484 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:46:14.553242 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88d37e23e60fb2e50f81d9256dcf20f22891ed9d688caaf676672a2262778e2e\": container with ID starting with 88d37e23e60fb2e50f81d9256dcf20f22891ed9d688caaf676672a2262778e2e not found: ID does not exist" containerID="88d37e23e60fb2e50f81d9256dcf20f22891ed9d688caaf676672a2262778e2e" Nov 03 02:46:14.553484 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:14.553279 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88d37e23e60fb2e50f81d9256dcf20f22891ed9d688caaf676672a2262778e2e"} err="failed to get container status \"88d37e23e60fb2e50f81d9256dcf20f22891ed9d688caaf676672a2262778e2e\": rpc error: code = NotFound desc = could not find container \"88d37e23e60fb2e50f81d9256dcf20f22891ed9d688caaf676672a2262778e2e\": container with ID starting with 88d37e23e60fb2e50f81d9256dcf20f22891ed9d688caaf676672a2262778e2e not found: ID does not exist" Nov 03 02:46:14.711153 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:46:14.711112 5605 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7232-26e0d0\" is forbidden: User \"system:node:ip-10-0-140-140.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-140.us-west-2.compute.internal' and this object" podUID="15fd430f-bf80-40da-b9b9-9f1ef4f040e0" pod="opendatahub/test-curl-7232-26e0d0" Nov 03 02:46:14.760393 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:46:14.759564 5605 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7232-3a363f\" is forbidden: User \"system:node:ip-10-0-140-140.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-140.us-west-2.compute.internal' and this object" podUID="7085212b-ea42-42ec-94f1-17d236621431" pod="opendatahub/test-curl-7232-3a363f" Nov 03 02:46:14.850010 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:14.849939 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7085212b-ea42-42ec-94f1-17d236621431" path="/var/lib/kubelet/pods/7085212b-ea42-42ec-94f1-17d236621431/volumes" Nov 03 02:46:14.877386 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:14.862297 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7232-bff49a"] Nov 03 02:46:14.950085 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:14.950065 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2" Nov 03 02:46:14.972389 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:14.962222 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2" Nov 03 02:46:15.085153 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:15.077596 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2" podStartSLOduration=3.077576947 podStartE2EDuration="3.077576947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:46:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:46:14.902623339 +0000 UTC m=+3252.161821492" watchObservedRunningTime="2026-11-03 02:46:15.077576947 +0000 UTC m=+3252.336775105" Nov 03 02:46:15.520903 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:15.520867 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7232-bff49a" event={"ID":"aaa737a3-6588-44bd-a759-d082cdf196e1","Type":"ContainerStarted","Data":"4b206d94984ba09ebb838d6e14a6482f2de713db314d3978fee3e00453e5502c"} Nov 03 02:46:15.520903 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:15.520912 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7232-bff49a" event={"ID":"aaa737a3-6588-44bd-a759-d082cdf196e1","Type":"ContainerStarted","Data":"2ee29761be407d85c5a4a6f9ce99771f4f937c326d697454bb6011f27a1031c6"} Nov 03 02:46:15.521478 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:15.521086 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2" Nov 03 02:46:15.522132 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:15.522116 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2" Nov 03 02:46:15.629590 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:15.629537 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7232-bff49a" podStartSLOduration=1.629522492 podStartE2EDuration="1.629522492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:46:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:46:15.627539331 +0000 UTC m=+3252.886737485" watchObservedRunningTime="2026-11-03 02:46:15.629522492 +0000 UTC m=+3252.888720645" Nov 03 02:46:16.511906 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:16.511874 5605 patch_prober.go:29] interesting pod/e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 02:46:16.512090 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:16.511924 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4" podUID="33cae564-16a9-42bb-a1a5-dc5c3257f56b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 02:46:16.707777 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:16.707730 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-78b5c5/auth-test-model-78b5c5-kserve-5bc8bf4cf7-gptc4" podUID="7e3f152e-f3d1-4e14-85a9-6f456467cf6a" containerName="main" containerID="cri-o://2c805c7a180e859e26dd6526605c5357ade0f14dbac6b817f3f7f8a4e4fb44f0" gracePeriod=45 Nov 03 02:46:16.845621 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:16.845595 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7232-26e0d0" Nov 03 02:46:16.946746 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:16.946714 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15fd430f-bf80-40da-b9b9-9f1ef4f040e0-kube-api-access-ntv9c\" (UniqueName: \"kubernetes.io/projected/15fd430f-bf80-40da-b9b9-9f1ef4f040e0-kube-api-access-ntv9c\") pod \"15fd430f-bf80-40da-b9b9-9f1ef4f040e0\" (UID: \"15fd430f-bf80-40da-b9b9-9f1ef4f040e0\") " Nov 03 02:46:16.949092 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:16.949063 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15fd430f-bf80-40da-b9b9-9f1ef4f040e0-kube-api-access-ntv9c" pod "15fd430f-bf80-40da-b9b9-9f1ef4f040e0" (UID: "15fd430f-bf80-40da-b9b9-9f1ef4f040e0"). InnerVolumeSpecName "kube-api-access-ntv9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:46:16.993145 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:16.993122 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-78b5c5/auth-test-model-78b5c5-kserve-5bc8bf4cf7-gptc4" Nov 03 02:46:17.048694 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:17.048669 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntv9c\" (UniqueName: \"kubernetes.io/projected/15fd430f-bf80-40da-b9b9-9f1ef4f040e0-kube-api-access-ntv9c\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:17.091726 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:17.091654 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7232-bff49a"] Nov 03 02:46:17.094445 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:17.094416 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7232-bff49a"] Nov 03 02:46:17.149832 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:17.149804 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e3f152e-f3d1-4e14-85a9-6f456467cf6a-kube-api-access-qgb4w\" (UniqueName: \"kubernetes.io/projected/7e3f152e-f3d1-4e14-85a9-6f456467cf6a-kube-api-access-qgb4w\") pod \"7e3f152e-f3d1-4e14-85a9-6f456467cf6a\" (UID: \"7e3f152e-f3d1-4e14-85a9-6f456467cf6a\") " Nov 03 02:46:17.149997 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:17.149876 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e3f152e-f3d1-4e14-85a9-6f456467cf6a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e3f152e-f3d1-4e14-85a9-6f456467cf6a-model-cache\") pod \"7e3f152e-f3d1-4e14-85a9-6f456467cf6a\" (UID: \"7e3f152e-f3d1-4e14-85a9-6f456467cf6a\") " Nov 03 02:46:17.149997 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:17.149972 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e3f152e-f3d1-4e14-85a9-6f456467cf6a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e3f152e-f3d1-4e14-85a9-6f456467cf6a-dshm\") pod \"7e3f152e-f3d1-4e14-85a9-6f456467cf6a\" (UID: \"7e3f152e-f3d1-4e14-85a9-6f456467cf6a\") " Nov 03 02:46:17.150131 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:17.150021 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e3f152e-f3d1-4e14-85a9-6f456467cf6a-tls-certs\" (UniqueName: \"kubernetes.io/secret/7e3f152e-f3d1-4e14-85a9-6f456467cf6a-tls-certs\") pod \"7e3f152e-f3d1-4e14-85a9-6f456467cf6a\" (UID: \"7e3f152e-f3d1-4e14-85a9-6f456467cf6a\") " Nov 03 02:46:17.150131 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:17.150109 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e3f152e-f3d1-4e14-85a9-6f456467cf6a-home\" (UniqueName: \"kubernetes.io/empty-dir/7e3f152e-f3d1-4e14-85a9-6f456467cf6a-home\") pod \"7e3f152e-f3d1-4e14-85a9-6f456467cf6a\" (UID: \"7e3f152e-f3d1-4e14-85a9-6f456467cf6a\") " Nov 03 02:46:17.150248 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:17.150206 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e3f152e-f3d1-4e14-85a9-6f456467cf6a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e3f152e-f3d1-4e14-85a9-6f456467cf6a-tmp-dir\") pod \"7e3f152e-f3d1-4e14-85a9-6f456467cf6a\" (UID: \"7e3f152e-f3d1-4e14-85a9-6f456467cf6a\") " Nov 03 02:46:17.150577 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:17.150435 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e3f152e-f3d1-4e14-85a9-6f456467cf6a-home" pod "7e3f152e-f3d1-4e14-85a9-6f456467cf6a" (UID: "7e3f152e-f3d1-4e14-85a9-6f456467cf6a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:46:17.150577 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:17.150540 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e3f152e-f3d1-4e14-85a9-6f456467cf6a-tmp-dir" pod "7e3f152e-f3d1-4e14-85a9-6f456467cf6a" (UID: "7e3f152e-f3d1-4e14-85a9-6f456467cf6a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:46:17.150775 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:17.150752 5605 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e3f152e-f3d1-4e14-85a9-6f456467cf6a-home\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:17.150832 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:17.150774 5605 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e3f152e-f3d1-4e14-85a9-6f456467cf6a-tmp-dir\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:17.150936 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:17.150184 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e3f152e-f3d1-4e14-85a9-6f456467cf6a-model-cache" pod "7e3f152e-f3d1-4e14-85a9-6f456467cf6a" (UID: "7e3f152e-f3d1-4e14-85a9-6f456467cf6a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:46:17.152352 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:17.152314 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e3f152e-f3d1-4e14-85a9-6f456467cf6a-tls-certs" pod "7e3f152e-f3d1-4e14-85a9-6f456467cf6a" (UID: "7e3f152e-f3d1-4e14-85a9-6f456467cf6a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:46:17.152528 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:17.152397 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e3f152e-f3d1-4e14-85a9-6f456467cf6a-kube-api-access-qgb4w" pod "7e3f152e-f3d1-4e14-85a9-6f456467cf6a" (UID: "7e3f152e-f3d1-4e14-85a9-6f456467cf6a"). InnerVolumeSpecName "kube-api-access-qgb4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:46:17.152528 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:17.152513 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e3f152e-f3d1-4e14-85a9-6f456467cf6a-dshm" pod "7e3f152e-f3d1-4e14-85a9-6f456467cf6a" (UID: "7e3f152e-f3d1-4e14-85a9-6f456467cf6a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:46:17.252657 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:17.252620 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgb4w\" (UniqueName: \"kubernetes.io/projected/7e3f152e-f3d1-4e14-85a9-6f456467cf6a-kube-api-access-qgb4w\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:17.252657 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:17.252654 5605 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e3f152e-f3d1-4e14-85a9-6f456467cf6a-model-cache\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:17.252868 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:17.252669 5605 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e3f152e-f3d1-4e14-85a9-6f456467cf6a-dshm\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:17.252868 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:17.252683 5605 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e3f152e-f3d1-4e14-85a9-6f456467cf6a-tls-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:17.544750 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:17.544708 5605 generic.go:356] "Generic (PLEG): container finished" podID="15fd430f-bf80-40da-b9b9-9f1ef4f040e0" containerID="91627a4e87cd6b6e0f901a6d4505ce3d46c490c6623f7ec81fc188d5d0223af3" exitCode=137 Nov 03 02:46:17.544911 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:17.544865 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7232-26e0d0" Nov 03 02:46:17.544911 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:17.544883 5605 scope.go:122] "RemoveContainer" containerID="91627a4e87cd6b6e0f901a6d4505ce3d46c490c6623f7ec81fc188d5d0223af3" Nov 03 02:46:17.546747 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:17.546722 5605 generic.go:356] "Generic (PLEG): container finished" podID="7e3f152e-f3d1-4e14-85a9-6f456467cf6a" containerID="2c805c7a180e859e26dd6526605c5357ade0f14dbac6b817f3f7f8a4e4fb44f0" exitCode=0 Nov 03 02:46:17.546950 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:17.546917 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-78b5c5/auth-test-model-78b5c5-kserve-5bc8bf4cf7-gptc4" event={"ID":"7e3f152e-f3d1-4e14-85a9-6f456467cf6a","Type":"ContainerDied","Data":"2c805c7a180e859e26dd6526605c5357ade0f14dbac6b817f3f7f8a4e4fb44f0"} Nov 03 02:46:17.547069 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:17.546971 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-78b5c5/auth-test-model-78b5c5-kserve-5bc8bf4cf7-gptc4" event={"ID":"7e3f152e-f3d1-4e14-85a9-6f456467cf6a","Type":"ContainerDied","Data":"87e54581ae566e3c8ba6152b070686c426ef471d66c8c4b855141f54aee4e49f"} Nov 03 02:46:17.547069 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:17.546999 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-78b5c5/auth-test-model-78b5c5-kserve-5bc8bf4cf7-gptc4" Nov 03 02:46:17.547702 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:17.547675 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7232-bff49a" podUID="aaa737a3-6588-44bd-a759-d082cdf196e1" containerName="test-curl-7232-bff49a" containerID="cri-o://4b206d94984ba09ebb838d6e14a6482f2de713db314d3978fee3e00453e5502c" gracePeriod=2 Nov 03 02:46:17.581737 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:17.581713 5605 scope.go:122] "RemoveContainer" containerID="91627a4e87cd6b6e0f901a6d4505ce3d46c490c6623f7ec81fc188d5d0223af3" Nov 03 02:46:17.582083 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:46:17.582061 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91627a4e87cd6b6e0f901a6d4505ce3d46c490c6623f7ec81fc188d5d0223af3\": container with ID starting with 91627a4e87cd6b6e0f901a6d4505ce3d46c490c6623f7ec81fc188d5d0223af3 not found: ID does not exist" containerID="91627a4e87cd6b6e0f901a6d4505ce3d46c490c6623f7ec81fc188d5d0223af3" Nov 03 02:46:17.582181 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:17.582089 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91627a4e87cd6b6e0f901a6d4505ce3d46c490c6623f7ec81fc188d5d0223af3"} err="failed to get container status \"91627a4e87cd6b6e0f901a6d4505ce3d46c490c6623f7ec81fc188d5d0223af3\": rpc error: code = NotFound desc = could not find container \"91627a4e87cd6b6e0f901a6d4505ce3d46c490c6623f7ec81fc188d5d0223af3\": container with ID starting with 91627a4e87cd6b6e0f901a6d4505ce3d46c490c6623f7ec81fc188d5d0223af3 not found: ID does not exist" Nov 03 02:46:17.582181 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:17.582108 5605 scope.go:122] "RemoveContainer" containerID="2c805c7a180e859e26dd6526605c5357ade0f14dbac6b817f3f7f8a4e4fb44f0" Nov 03 02:46:17.590760 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:17.590744 5605 scope.go:122] "RemoveContainer" containerID="2c805c7a180e859e26dd6526605c5357ade0f14dbac6b817f3f7f8a4e4fb44f0" Nov 03 02:46:17.591028 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:46:17.591008 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c805c7a180e859e26dd6526605c5357ade0f14dbac6b817f3f7f8a4e4fb44f0\": container with ID starting with 2c805c7a180e859e26dd6526605c5357ade0f14dbac6b817f3f7f8a4e4fb44f0 not found: ID does not exist" containerID="2c805c7a180e859e26dd6526605c5357ade0f14dbac6b817f3f7f8a4e4fb44f0" Nov 03 02:46:17.591071 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:17.591034 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c805c7a180e859e26dd6526605c5357ade0f14dbac6b817f3f7f8a4e4fb44f0"} err="failed to get container status \"2c805c7a180e859e26dd6526605c5357ade0f14dbac6b817f3f7f8a4e4fb44f0\": rpc error: code = NotFound desc = could not find container \"2c805c7a180e859e26dd6526605c5357ade0f14dbac6b817f3f7f8a4e4fb44f0\": container with ID starting with 2c805c7a180e859e26dd6526605c5357ade0f14dbac6b817f3f7f8a4e4fb44f0 not found: ID does not exist" Nov 03 02:46:17.644492 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:46:17.644466 5605 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7232-bff49a\" is forbidden: User \"system:node:ip-10-0-140-140.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-140.us-west-2.compute.internal' and this object" podUID="aaa737a3-6588-44bd-a759-d082cdf196e1" pod="opendatahub/test-curl-7232-bff49a" Nov 03 02:46:17.750436 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:17.750406 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-78b5c5/auth-test-model-78b5c5-kserve-5bc8bf4cf7-gptc4"] Nov 03 02:46:17.778379 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:17.768149 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-78b5c5/auth-test-model-78b5c5-kserve-5bc8bf4cf7-gptc4"] Nov 03 02:46:17.800825 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:46:17.800788 5605 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7232-bff49a\" is forbidden: User \"system:node:ip-10-0-140-140.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-140.us-west-2.compute.internal' and this object" podUID="aaa737a3-6588-44bd-a759-d082cdf196e1" pod="opendatahub/test-curl-7232-bff49a" Nov 03 02:46:18.850656 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:18.850612 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15fd430f-bf80-40da-b9b9-9f1ef4f040e0" path="/var/lib/kubelet/pods/15fd430f-bf80-40da-b9b9-9f1ef4f040e0/volumes" Nov 03 02:46:18.851271 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:18.851250 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e3f152e-f3d1-4e14-85a9-6f456467cf6a" path="/var/lib/kubelet/pods/7e3f152e-f3d1-4e14-85a9-6f456467cf6a/volumes" Nov 03 02:46:19.235966 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.235881 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp"] Nov 03 02:46:19.236424 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.236401 5605 cpu_manager.go:414] "containerMap: removing container" podUID="aaa737a3-6588-44bd-a759-d082cdf196e1" containerName="test-curl-7232-bff49a" Nov 03 02:46:19.236497 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.236426 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaa737a3-6588-44bd-a759-d082cdf196e1" containerName="test-curl-7232-bff49a" Nov 03 02:46:19.236497 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.236481 5605 cpu_manager.go:414] "containerMap: removing container" podUID="7e3f152e-f3d1-4e14-85a9-6f456467cf6a" containerName="main" Nov 03 02:46:19.236497 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.236486 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e3f152e-f3d1-4e14-85a9-6f456467cf6a" containerName="main" Nov 03 02:46:19.236608 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.236576 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e3f152e-f3d1-4e14-85a9-6f456467cf6a" containerName="main" Nov 03 02:46:19.236608 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.236588 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaa737a3-6588-44bd-a759-d082cdf196e1" containerName="test-curl-7232-bff49a" Nov 03 02:46:19.241475 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.241450 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp" Nov 03 02:46:19.320612 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.318685 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-604da0-openshift-default-dockercfg-d2sd7\"" Nov 03 02:46:19.320612 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.319948 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d98a3d42-36f0-48d4-b6dc-7dd978b953ae-istio-podinfo\") pod \"e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp\" (UID: \"d98a3d42-36f0-48d4-b6dc-7dd978b953ae\") " pod="openshift-ingress/e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp" Nov 03 02:46:19.320612 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.320010 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d98a3d42-36f0-48d4-b6dc-7dd978b953ae-istio-data\") pod \"e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp\" (UID: \"d98a3d42-36f0-48d4-b6dc-7dd978b953ae\") " pod="openshift-ingress/e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp" Nov 03 02:46:19.320612 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.320040 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqx6t\" (UniqueName: \"kubernetes.io/projected/d98a3d42-36f0-48d4-b6dc-7dd978b953ae-kube-api-access-zqx6t\") pod \"e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp\" (UID: \"d98a3d42-36f0-48d4-b6dc-7dd978b953ae\") " pod="openshift-ingress/e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp" Nov 03 02:46:19.320612 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.320129 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d98a3d42-36f0-48d4-b6dc-7dd978b953ae-istio-envoy\") pod \"e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp\" (UID: \"d98a3d42-36f0-48d4-b6dc-7dd978b953ae\") " pod="openshift-ingress/e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp" Nov 03 02:46:19.320612 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.320224 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98a3d42-36f0-48d4-b6dc-7dd978b953ae-credential-socket\") pod \"e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp\" (UID: \"d98a3d42-36f0-48d4-b6dc-7dd978b953ae\") " pod="openshift-ingress/e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp" Nov 03 02:46:19.320612 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.320266 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98a3d42-36f0-48d4-b6dc-7dd978b953ae-workload-socket\") pod \"e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp\" (UID: \"d98a3d42-36f0-48d4-b6dc-7dd978b953ae\") " pod="openshift-ingress/e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp" Nov 03 02:46:19.320612 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.320295 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d98a3d42-36f0-48d4-b6dc-7dd978b953ae-workload-certs\") pod \"e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp\" (UID: \"d98a3d42-36f0-48d4-b6dc-7dd978b953ae\") " pod="openshift-ingress/e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp" Nov 03 02:46:19.320612 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.320335 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d98a3d42-36f0-48d4-b6dc-7dd978b953ae-istio-token\") pod \"e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp\" (UID: \"d98a3d42-36f0-48d4-b6dc-7dd978b953ae\") " pod="openshift-ingress/e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp" Nov 03 02:46:19.320612 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.320377 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d98a3d42-36f0-48d4-b6dc-7dd978b953ae-istiod-ca-cert\") pod \"e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp\" (UID: \"d98a3d42-36f0-48d4-b6dc-7dd978b953ae\") " pod="openshift-ingress/e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp" Nov 03 02:46:19.346093 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.346046 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp"] Nov 03 02:46:19.421592 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.421547 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d98a3d42-36f0-48d4-b6dc-7dd978b953ae-istio-podinfo\") pod \"e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp\" (UID: \"d98a3d42-36f0-48d4-b6dc-7dd978b953ae\") " pod="openshift-ingress/e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp" Nov 03 02:46:19.421792 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.421612 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d98a3d42-36f0-48d4-b6dc-7dd978b953ae-istio-data\") pod \"e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp\" (UID: \"d98a3d42-36f0-48d4-b6dc-7dd978b953ae\") " pod="openshift-ingress/e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp" Nov 03 02:46:19.421792 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.421642 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqx6t\" (UniqueName: \"kubernetes.io/projected/d98a3d42-36f0-48d4-b6dc-7dd978b953ae-kube-api-access-zqx6t\") pod \"e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp\" (UID: \"d98a3d42-36f0-48d4-b6dc-7dd978b953ae\") " pod="openshift-ingress/e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp" Nov 03 02:46:19.421792 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.421732 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d98a3d42-36f0-48d4-b6dc-7dd978b953ae-istio-envoy\") pod \"e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp\" (UID: \"d98a3d42-36f0-48d4-b6dc-7dd978b953ae\") " pod="openshift-ingress/e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp" Nov 03 02:46:19.421991 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.421823 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98a3d42-36f0-48d4-b6dc-7dd978b953ae-credential-socket\") pod \"e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp\" (UID: \"d98a3d42-36f0-48d4-b6dc-7dd978b953ae\") " pod="openshift-ingress/e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp" Nov 03 02:46:19.421991 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.421862 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98a3d42-36f0-48d4-b6dc-7dd978b953ae-workload-socket\") pod \"e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp\" (UID: \"d98a3d42-36f0-48d4-b6dc-7dd978b953ae\") " pod="openshift-ingress/e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp" Nov 03 02:46:19.421991 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.421891 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d98a3d42-36f0-48d4-b6dc-7dd978b953ae-workload-certs\") pod \"e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp\" (UID: \"d98a3d42-36f0-48d4-b6dc-7dd978b953ae\") " pod="openshift-ingress/e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp" Nov 03 02:46:19.421991 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.421925 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d98a3d42-36f0-48d4-b6dc-7dd978b953ae-istio-token\") pod \"e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp\" (UID: \"d98a3d42-36f0-48d4-b6dc-7dd978b953ae\") " pod="openshift-ingress/e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp" Nov 03 02:46:19.421991 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.421951 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d98a3d42-36f0-48d4-b6dc-7dd978b953ae-istiod-ca-cert\") pod \"e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp\" (UID: \"d98a3d42-36f0-48d4-b6dc-7dd978b953ae\") " pod="openshift-ingress/e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp" Nov 03 02:46:19.423503 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.422763 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d98a3d42-36f0-48d4-b6dc-7dd978b953ae-istiod-ca-cert\") pod \"e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp\" (UID: \"d98a3d42-36f0-48d4-b6dc-7dd978b953ae\") " pod="openshift-ingress/e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp" Nov 03 02:46:19.423503 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.423048 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98a3d42-36f0-48d4-b6dc-7dd978b953ae-credential-socket\") pod \"e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp\" (UID: \"d98a3d42-36f0-48d4-b6dc-7dd978b953ae\") " pod="openshift-ingress/e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp" Nov 03 02:46:19.423723 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.423678 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d98a3d42-36f0-48d4-b6dc-7dd978b953ae-istio-data\") pod \"e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp\" (UID: \"d98a3d42-36f0-48d4-b6dc-7dd978b953ae\") " pod="openshift-ingress/e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp" Nov 03 02:46:19.424150 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.424102 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d98a3d42-36f0-48d4-b6dc-7dd978b953ae-workload-certs\") pod \"e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp\" (UID: \"d98a3d42-36f0-48d4-b6dc-7dd978b953ae\") " pod="openshift-ingress/e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp" Nov 03 02:46:19.424277 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.424255 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98a3d42-36f0-48d4-b6dc-7dd978b953ae-workload-socket\") pod \"e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp\" (UID: \"d98a3d42-36f0-48d4-b6dc-7dd978b953ae\") " pod="openshift-ingress/e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp" Nov 03 02:46:19.425278 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.425249 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d98a3d42-36f0-48d4-b6dc-7dd978b953ae-istio-envoy\") pod \"e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp\" (UID: \"d98a3d42-36f0-48d4-b6dc-7dd978b953ae\") " pod="openshift-ingress/e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp" Nov 03 02:46:19.427244 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.427165 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d98a3d42-36f0-48d4-b6dc-7dd978b953ae-istio-podinfo\") pod \"e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp\" (UID: \"d98a3d42-36f0-48d4-b6dc-7dd978b953ae\") " pod="openshift-ingress/e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp" Nov 03 02:46:19.477432 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.477388 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d98a3d42-36f0-48d4-b6dc-7dd978b953ae-istio-token\") pod \"e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp\" (UID: \"d98a3d42-36f0-48d4-b6dc-7dd978b953ae\") " pod="openshift-ingress/e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp" Nov 03 02:46:19.477630 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.477523 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqx6t\" (UniqueName: \"kubernetes.io/projected/d98a3d42-36f0-48d4-b6dc-7dd978b953ae-kube-api-access-zqx6t\") pod \"e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp\" (UID: \"d98a3d42-36f0-48d4-b6dc-7dd978b953ae\") " pod="openshift-ingress/e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp" Nov 03 02:46:19.584246 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.584199 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp" Nov 03 02:46:19.821399 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.814952 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4" Nov 03 02:46:19.829442 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.829393 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33cae564-16a9-42bb-a1a5-dc5c3257f56b-istio-token\" (UniqueName: \"kubernetes.io/projected/33cae564-16a9-42bb-a1a5-dc5c3257f56b-istio-token\") pod \"33cae564-16a9-42bb-a1a5-dc5c3257f56b\" (UID: \"33cae564-16a9-42bb-a1a5-dc5c3257f56b\") " Nov 03 02:46:19.829625 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.829471 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/33cae564-16a9-42bb-a1a5-dc5c3257f56b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33cae564-16a9-42bb-a1a5-dc5c3257f56b-istio-podinfo\") pod \"33cae564-16a9-42bb-a1a5-dc5c3257f56b\" (UID: \"33cae564-16a9-42bb-a1a5-dc5c3257f56b\") " Nov 03 02:46:19.829625 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.829529 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33cae564-16a9-42bb-a1a5-dc5c3257f56b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33cae564-16a9-42bb-a1a5-dc5c3257f56b-istiod-ca-cert\") pod \"33cae564-16a9-42bb-a1a5-dc5c3257f56b\" (UID: \"33cae564-16a9-42bb-a1a5-dc5c3257f56b\") " Nov 03 02:46:19.829625 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.829572 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33cae564-16a9-42bb-a1a5-dc5c3257f56b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33cae564-16a9-42bb-a1a5-dc5c3257f56b-workload-socket\") pod \"33cae564-16a9-42bb-a1a5-dc5c3257f56b\" (UID: \"33cae564-16a9-42bb-a1a5-dc5c3257f56b\") " Nov 03 02:46:19.829812 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.829691 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33cae564-16a9-42bb-a1a5-dc5c3257f56b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33cae564-16a9-42bb-a1a5-dc5c3257f56b-istio-data\") pod \"33cae564-16a9-42bb-a1a5-dc5c3257f56b\" (UID: \"33cae564-16a9-42bb-a1a5-dc5c3257f56b\") " Nov 03 02:46:19.829812 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.829797 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33cae564-16a9-42bb-a1a5-dc5c3257f56b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33cae564-16a9-42bb-a1a5-dc5c3257f56b-istio-envoy\") pod \"33cae564-16a9-42bb-a1a5-dc5c3257f56b\" (UID: \"33cae564-16a9-42bb-a1a5-dc5c3257f56b\") " Nov 03 02:46:19.829925 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.829850 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33cae564-16a9-42bb-a1a5-dc5c3257f56b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33cae564-16a9-42bb-a1a5-dc5c3257f56b-workload-certs\") pod \"33cae564-16a9-42bb-a1a5-dc5c3257f56b\" (UID: \"33cae564-16a9-42bb-a1a5-dc5c3257f56b\") " Nov 03 02:46:19.829925 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.829885 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33cae564-16a9-42bb-a1a5-dc5c3257f56b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33cae564-16a9-42bb-a1a5-dc5c3257f56b-credential-socket\") pod \"33cae564-16a9-42bb-a1a5-dc5c3257f56b\" (UID: \"33cae564-16a9-42bb-a1a5-dc5c3257f56b\") " Nov 03 02:46:19.830341 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.830085 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33cae564-16a9-42bb-a1a5-dc5c3257f56b-workload-certs" pod "33cae564-16a9-42bb-a1a5-dc5c3257f56b" (UID: "33cae564-16a9-42bb-a1a5-dc5c3257f56b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:46:19.830341 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.830233 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33cae564-16a9-42bb-a1a5-dc5c3257f56b-workload-socket" pod "33cae564-16a9-42bb-a1a5-dc5c3257f56b" (UID: "33cae564-16a9-42bb-a1a5-dc5c3257f56b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:46:19.830341 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.830272 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33cae564-16a9-42bb-a1a5-dc5c3257f56b-istio-data" pod "33cae564-16a9-42bb-a1a5-dc5c3257f56b" (UID: "33cae564-16a9-42bb-a1a5-dc5c3257f56b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:46:19.830341 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.830305 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33cae564-16a9-42bb-a1a5-dc5c3257f56b-credential-socket" pod "33cae564-16a9-42bb-a1a5-dc5c3257f56b" (UID: "33cae564-16a9-42bb-a1a5-dc5c3257f56b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:46:19.830341 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.830338 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33cae564-16a9-42bb-a1a5-dc5c3257f56b-istiod-ca-cert" pod "33cae564-16a9-42bb-a1a5-dc5c3257f56b" (UID: "33cae564-16a9-42bb-a1a5-dc5c3257f56b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:46:19.830706 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.830562 5605 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33cae564-16a9-42bb-a1a5-dc5c3257f56b-istiod-ca-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:19.830706 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.830583 5605 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33cae564-16a9-42bb-a1a5-dc5c3257f56b-workload-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:19.830706 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.830597 5605 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33cae564-16a9-42bb-a1a5-dc5c3257f56b-istio-data\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:19.830706 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.830613 5605 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33cae564-16a9-42bb-a1a5-dc5c3257f56b-workload-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:19.830706 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.830626 5605 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33cae564-16a9-42bb-a1a5-dc5c3257f56b-credential-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:19.832210 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.832177 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33cae564-16a9-42bb-a1a5-dc5c3257f56b-istio-token" pod "33cae564-16a9-42bb-a1a5-dc5c3257f56b" (UID: "33cae564-16a9-42bb-a1a5-dc5c3257f56b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:46:19.832701 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.832680 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/33cae564-16a9-42bb-a1a5-dc5c3257f56b-istio-podinfo" pod "33cae564-16a9-42bb-a1a5-dc5c3257f56b" (UID: "33cae564-16a9-42bb-a1a5-dc5c3257f56b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:46:19.832900 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.832877 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33cae564-16a9-42bb-a1a5-dc5c3257f56b-istio-envoy" pod "33cae564-16a9-42bb-a1a5-dc5c3257f56b" (UID: "33cae564-16a9-42bb-a1a5-dc5c3257f56b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:46:19.852434 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.852396 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7232-bff49a" Nov 03 02:46:19.931152 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.931121 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33cae564-16a9-42bb-a1a5-dc5c3257f56b-kube-api-access-bk8tn\" (UniqueName: \"kubernetes.io/projected/33cae564-16a9-42bb-a1a5-dc5c3257f56b-kube-api-access-bk8tn\") pod \"33cae564-16a9-42bb-a1a5-dc5c3257f56b\" (UID: \"33cae564-16a9-42bb-a1a5-dc5c3257f56b\") " Nov 03 02:46:19.931377 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.931191 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaa737a3-6588-44bd-a759-d082cdf196e1-kube-api-access-ktws5\" (UniqueName: \"kubernetes.io/projected/aaa737a3-6588-44bd-a759-d082cdf196e1-kube-api-access-ktws5\") pod \"aaa737a3-6588-44bd-a759-d082cdf196e1\" (UID: \"aaa737a3-6588-44bd-a759-d082cdf196e1\") " Nov 03 02:46:19.931589 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.931562 5605 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33cae564-16a9-42bb-a1a5-dc5c3257f56b-istio-envoy\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:19.931654 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.931595 5605 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33cae564-16a9-42bb-a1a5-dc5c3257f56b-istio-token\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:19.931654 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.931610 5605 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33cae564-16a9-42bb-a1a5-dc5c3257f56b-istio-podinfo\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:19.933890 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.933861 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33cae564-16a9-42bb-a1a5-dc5c3257f56b-kube-api-access-bk8tn" pod "33cae564-16a9-42bb-a1a5-dc5c3257f56b" (UID: "33cae564-16a9-42bb-a1a5-dc5c3257f56b"). InnerVolumeSpecName "kube-api-access-bk8tn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:46:19.934071 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.934052 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaa737a3-6588-44bd-a759-d082cdf196e1-kube-api-access-ktws5" pod "aaa737a3-6588-44bd-a759-d082cdf196e1" (UID: "aaa737a3-6588-44bd-a759-d082cdf196e1"). InnerVolumeSpecName "kube-api-access-ktws5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:46:19.940983 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:19.940954 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-63a61e/noten-model-63a61e-kserve-854bff945-8l56c"] Nov 03 02:46:20.032923 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:20.032884 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bk8tn\" (UniqueName: \"kubernetes.io/projected/33cae564-16a9-42bb-a1a5-dc5c3257f56b-kube-api-access-bk8tn\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:20.032923 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:20.032919 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktws5\" (UniqueName: \"kubernetes.io/projected/aaa737a3-6588-44bd-a759-d082cdf196e1-kube-api-access-ktws5\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:20.078583 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:20.078488 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp"] Nov 03 02:46:20.082951 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:46:20.082910 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd98a3d42_36f0_48d4_b6dc_7dd978b953ae.slice/crio-458f4a7ff90155a3aff7f45b0202042d3743dfa5fd7d23db2c301a0e7dd2c5b8 WatchSource:0}: Error finding container 458f4a7ff90155a3aff7f45b0202042d3743dfa5fd7d23db2c301a0e7dd2c5b8: Status 404 returned error can't find the container with id 458f4a7ff90155a3aff7f45b0202042d3743dfa5fd7d23db2c301a0e7dd2c5b8 Nov 03 02:46:20.086065 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:20.086022 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:46:20.086171 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:20.086121 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:46:20.086209 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:20.086167 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:46:20.576349 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:20.576314 5605 generic.go:356] "Generic (PLEG): container finished" podID="aaa737a3-6588-44bd-a759-d082cdf196e1" containerID="4b206d94984ba09ebb838d6e14a6482f2de713db314d3978fee3e00453e5502c" exitCode=137 Nov 03 02:46:20.576576 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:20.576385 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7232-bff49a" Nov 03 02:46:20.576576 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:20.576429 5605 scope.go:122] "RemoveContainer" containerID="4b206d94984ba09ebb838d6e14a6482f2de713db314d3978fee3e00453e5502c" Nov 03 02:46:20.606577 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:20.606544 5605 scope.go:122] "RemoveContainer" containerID="4b206d94984ba09ebb838d6e14a6482f2de713db314d3978fee3e00453e5502c" Nov 03 02:46:20.606979 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:46:20.606943 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b206d94984ba09ebb838d6e14a6482f2de713db314d3978fee3e00453e5502c\": container with ID starting with 4b206d94984ba09ebb838d6e14a6482f2de713db314d3978fee3e00453e5502c not found: ID does not exist" containerID="4b206d94984ba09ebb838d6e14a6482f2de713db314d3978fee3e00453e5502c" Nov 03 02:46:20.607113 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:20.606991 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b206d94984ba09ebb838d6e14a6482f2de713db314d3978fee3e00453e5502c"} err="failed to get container status \"4b206d94984ba09ebb838d6e14a6482f2de713db314d3978fee3e00453e5502c\": rpc error: code = NotFound desc = could not find container \"4b206d94984ba09ebb838d6e14a6482f2de713db314d3978fee3e00453e5502c\": container with ID starting with 4b206d94984ba09ebb838d6e14a6482f2de713db314d3978fee3e00453e5502c not found: ID does not exist" Nov 03 02:46:20.607370 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:20.607324 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp" event={"ID":"d98a3d42-36f0-48d4-b6dc-7dd978b953ae","Type":"ContainerStarted","Data":"be592416eb06a16a33616e403d22f5e788f0a6eb30c86edc20fe36fddb62bd1a"} Nov 03 02:46:20.607568 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:20.607548 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp" event={"ID":"d98a3d42-36f0-48d4-b6dc-7dd978b953ae","Type":"ContainerStarted","Data":"458f4a7ff90155a3aff7f45b0202042d3743dfa5fd7d23db2c301a0e7dd2c5b8"} Nov 03 02:46:20.609161 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:20.609137 5605 generic.go:356] "Generic (PLEG): container finished" podID="33cae564-16a9-42bb-a1a5-dc5c3257f56b" containerID="eea83ba4c269a0fb4f25276b7d3c175be50b5f2d8f90be9f333bcf1dc73f65e3" exitCode=0 Nov 03 02:46:20.609261 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:20.609207 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4" Nov 03 02:46:20.609261 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:20.609223 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4" event={"ID":"33cae564-16a9-42bb-a1a5-dc5c3257f56b","Type":"ContainerDied","Data":"eea83ba4c269a0fb4f25276b7d3c175be50b5f2d8f90be9f333bcf1dc73f65e3"} Nov 03 02:46:20.609350 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:20.609279 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4" event={"ID":"33cae564-16a9-42bb-a1a5-dc5c3257f56b","Type":"ContainerDied","Data":"568a4212a803d029fece1e38c59ebfaa780aadf6752bc8c3f77fc3edd88f4b3d"} Nov 03 02:46:20.609350 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:20.609308 5605 scope.go:122] "RemoveContainer" containerID="eea83ba4c269a0fb4f25276b7d3c175be50b5f2d8f90be9f333bcf1dc73f65e3" Nov 03 02:46:20.622005 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:20.621982 5605 scope.go:122] "RemoveContainer" containerID="eea83ba4c269a0fb4f25276b7d3c175be50b5f2d8f90be9f333bcf1dc73f65e3" Nov 03 02:46:20.622417 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:46:20.622394 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eea83ba4c269a0fb4f25276b7d3c175be50b5f2d8f90be9f333bcf1dc73f65e3\": container with ID starting with eea83ba4c269a0fb4f25276b7d3c175be50b5f2d8f90be9f333bcf1dc73f65e3 not found: ID does not exist" containerID="eea83ba4c269a0fb4f25276b7d3c175be50b5f2d8f90be9f333bcf1dc73f65e3" Nov 03 02:46:20.622503 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:20.622433 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eea83ba4c269a0fb4f25276b7d3c175be50b5f2d8f90be9f333bcf1dc73f65e3"} err="failed to get container status \"eea83ba4c269a0fb4f25276b7d3c175be50b5f2d8f90be9f333bcf1dc73f65e3\": rpc error: code = NotFound desc = could not find container \"eea83ba4c269a0fb4f25276b7d3c175be50b5f2d8f90be9f333bcf1dc73f65e3\": container with ID starting with eea83ba4c269a0fb4f25276b7d3c175be50b5f2d8f90be9f333bcf1dc73f65e3 not found: ID does not exist" Nov 03 02:46:20.649659 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:46:20.649619 5605 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7232-bff49a\" is forbidden: User \"system:node:ip-10-0-140-140.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-140.us-west-2.compute.internal' and this object" podUID="aaa737a3-6588-44bd-a759-d082cdf196e1" pod="opendatahub/test-curl-7232-bff49a" Nov 03 02:46:20.698313 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:46:20.698216 5605 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7232-bff49a\" is forbidden: User \"system:node:ip-10-0-140-140.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-140.us-west-2.compute.internal' and this object" podUID="aaa737a3-6588-44bd-a759-d082cdf196e1" pod="opendatahub/test-curl-7232-bff49a" Nov 03 02:46:20.810167 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:20.810122 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4"] Nov 03 02:46:20.854025 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:20.853985 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaa737a3-6588-44bd-a759-d082cdf196e1" path="/var/lib/kubelet/pods/aaa737a3-6588-44bd-a759-d082cdf196e1/volumes" Nov 03 02:46:20.855794 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:20.855766 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-63a61e-openshift-default-d7b64cf89-5gmf4"] Nov 03 02:46:20.919629 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:20.919567 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp" podStartSLOduration=1.919548402 podStartE2EDuration="1.919548402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:46:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:46:20.918662935 +0000 UTC m=+3258.177861083" watchObservedRunningTime="2026-11-03 02:46:20.919548402 +0000 UTC m=+3258.178746559" Nov 03 02:46:21.367550 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:21.367514 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ca6ef4-74f5d66687-d4vr2"] Nov 03 02:46:21.368011 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:21.367998 5605 cpu_manager.go:414] "containerMap: removing container" podUID="33cae564-16a9-42bb-a1a5-dc5c3257f56b" containerName="istio-proxy" Nov 03 02:46:21.368118 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:21.368011 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33cae564-16a9-42bb-a1a5-dc5c3257f56b" containerName="istio-proxy" Nov 03 02:46:21.368179 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:21.368137 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="33cae564-16a9-42bb-a1a5-dc5c3257f56b" containerName="istio-proxy" Nov 03 02:46:21.373048 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:21.373027 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ca6ef4-74f5d66687-d4vr2" Nov 03 02:46:21.422797 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:21.422762 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-ca6ef4\"" Nov 03 02:46:21.423061 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:21.422905 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-ca6ef4-dockercfg-hrgr2\"" Nov 03 02:46:21.424782 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:21.424748 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ca6ef4-74f5d66687-d4vr2"] Nov 03 02:46:21.482386 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:21.466307 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd54f6c1-849d-49e1-9d0c-ce8f8e7ccc96-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-ca6ef4-74f5d66687-d4vr2\" (UID: \"dd54f6c1-849d-49e1-9d0c-ce8f8e7ccc96\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ca6ef4-74f5d66687-d4vr2" Nov 03 02:46:21.482386 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:21.466574 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vblnv\" (UniqueName: \"kubernetes.io/projected/dd54f6c1-849d-49e1-9d0c-ce8f8e7ccc96-kube-api-access-vblnv\") pod \"payload-pre-processing-e2e-extmod-a-ca6ef4-74f5d66687-d4vr2\" (UID: \"dd54f6c1-849d-49e1-9d0c-ce8f8e7ccc96\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ca6ef4-74f5d66687-d4vr2" Nov 03 02:46:21.521244 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:21.521203 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ca6ef4-7f7486c797-6kt8x"] Nov 03 02:46:21.527103 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:21.527072 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ca6ef4-7f7486c797-6kt8x" Nov 03 02:46:21.567763 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:21.567726 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frts2\" (UniqueName: \"kubernetes.io/projected/b22351dc-095e-4437-a2d7-23a1addf255b-kube-api-access-frts2\") pod \"payload-processing-e2e-extmod-a-ca6ef4-7f7486c797-6kt8x\" (UID: \"b22351dc-095e-4437-a2d7-23a1addf255b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ca6ef4-7f7486c797-6kt8x" Nov 03 02:46:21.573066 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:21.567865 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vblnv\" (UniqueName: \"kubernetes.io/projected/dd54f6c1-849d-49e1-9d0c-ce8f8e7ccc96-kube-api-access-vblnv\") pod \"payload-pre-processing-e2e-extmod-a-ca6ef4-74f5d66687-d4vr2\" (UID: \"dd54f6c1-849d-49e1-9d0c-ce8f8e7ccc96\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ca6ef4-74f5d66687-d4vr2" Nov 03 02:46:21.573066 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:21.567939 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b22351dc-095e-4437-a2d7-23a1addf255b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-ca6ef4-7f7486c797-6kt8x\" (UID: \"b22351dc-095e-4437-a2d7-23a1addf255b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ca6ef4-7f7486c797-6kt8x" Nov 03 02:46:21.573066 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:21.567962 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd54f6c1-849d-49e1-9d0c-ce8f8e7ccc96-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-ca6ef4-74f5d66687-d4vr2\" (UID: \"dd54f6c1-849d-49e1-9d0c-ce8f8e7ccc96\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ca6ef4-74f5d66687-d4vr2" Nov 03 02:46:21.573066 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:21.568699 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd54f6c1-849d-49e1-9d0c-ce8f8e7ccc96-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-ca6ef4-74f5d66687-d4vr2\" (UID: \"dd54f6c1-849d-49e1-9d0c-ce8f8e7ccc96\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ca6ef4-74f5d66687-d4vr2" Nov 03 02:46:21.582846 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:21.582815 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ca6ef4-7f7486c797-6kt8x"] Nov 03 02:46:21.588491 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:21.588452 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp" Nov 03 02:46:21.602105 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:21.602073 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp" Nov 03 02:46:21.622958 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:21.622698 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp" Nov 03 02:46:21.623915 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:21.623890 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp" Nov 03 02:46:21.624062 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:21.624038 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vblnv\" (UniqueName: \"kubernetes.io/projected/dd54f6c1-849d-49e1-9d0c-ce8f8e7ccc96-kube-api-access-vblnv\") pod \"payload-pre-processing-e2e-extmod-a-ca6ef4-74f5d66687-d4vr2\" (UID: \"dd54f6c1-849d-49e1-9d0c-ce8f8e7ccc96\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ca6ef4-74f5d66687-d4vr2" Nov 03 02:46:21.669055 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:21.669013 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b22351dc-095e-4437-a2d7-23a1addf255b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-ca6ef4-7f7486c797-6kt8x\" (UID: \"b22351dc-095e-4437-a2d7-23a1addf255b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ca6ef4-7f7486c797-6kt8x" Nov 03 02:46:21.669279 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:21.669164 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frts2\" (UniqueName: \"kubernetes.io/projected/b22351dc-095e-4437-a2d7-23a1addf255b-kube-api-access-frts2\") pod \"payload-processing-e2e-extmod-a-ca6ef4-7f7486c797-6kt8x\" (UID: \"b22351dc-095e-4437-a2d7-23a1addf255b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ca6ef4-7f7486c797-6kt8x" Nov 03 02:46:21.669844 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:21.669823 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b22351dc-095e-4437-a2d7-23a1addf255b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-ca6ef4-7f7486c797-6kt8x\" (UID: \"b22351dc-095e-4437-a2d7-23a1addf255b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ca6ef4-7f7486c797-6kt8x" Nov 03 02:46:21.686782 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:21.686671 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ca6ef4-74f5d66687-d4vr2" Nov 03 02:46:21.722766 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:21.722735 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frts2\" (UniqueName: \"kubernetes.io/projected/b22351dc-095e-4437-a2d7-23a1addf255b-kube-api-access-frts2\") pod \"payload-processing-e2e-extmod-a-ca6ef4-7f7486c797-6kt8x\" (UID: \"b22351dc-095e-4437-a2d7-23a1addf255b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ca6ef4-7f7486c797-6kt8x" Nov 03 02:46:21.843142 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:21.843109 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ca6ef4-7f7486c797-6kt8x" Nov 03 02:46:22.059196 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:22.059157 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ca6ef4-74f5d66687-d4vr2"] Nov 03 02:46:22.064891 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:46:22.064860 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd54f6c1_849d_49e1_9d0c_ce8f8e7ccc96.slice/crio-31fae1962c9104a7174228e0877d5b61b78d91bfd9aee84ca12738b5a9aee92d WatchSource:0}: Error finding container 31fae1962c9104a7174228e0877d5b61b78d91bfd9aee84ca12738b5a9aee92d: Status 404 returned error can't find the container with id 31fae1962c9104a7174228e0877d5b61b78d91bfd9aee84ca12738b5a9aee92d Nov 03 02:46:22.218307 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:22.218217 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ca6ef4-7f7486c797-6kt8x"] Nov 03 02:46:22.223843 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:46:22.223816 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb22351dc_095e_4437_a2d7_23a1addf255b.slice/crio-bfcc308ec0917d0154aac8433ea76548d7669c3b8160a0f40edf2ac391c49b41 WatchSource:0}: Error finding container bfcc308ec0917d0154aac8433ea76548d7669c3b8160a0f40edf2ac391c49b41: Status 404 returned error can't find the container with id bfcc308ec0917d0154aac8433ea76548d7669c3b8160a0f40edf2ac391c49b41 Nov 03 02:46:22.630163 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:22.630132 5605 generic.go:356] "Generic (PLEG): container finished" podID="f7c23f7b-4a52-4a24-8fce-ba087104fca5" containerID="ea5fea9d3bd0b499ada713f2daec3443f2cf147eb3aabdc778cc85c2e515ae44" exitCode=0 Nov 03 02:46:22.630382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:22.630211 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-038c21-openshift-default-5d5c88569c-4r995" event={"ID":"f7c23f7b-4a52-4a24-8fce-ba087104fca5","Type":"ContainerDied","Data":"ea5fea9d3bd0b499ada713f2daec3443f2cf147eb3aabdc778cc85c2e515ae44"} Nov 03 02:46:22.630753 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:22.630729 5605 scope.go:122] "RemoveContainer" containerID="ea5fea9d3bd0b499ada713f2daec3443f2cf147eb3aabdc778cc85c2e515ae44" Nov 03 02:46:22.632004 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:22.631973 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ca6ef4-7f7486c797-6kt8x" event={"ID":"b22351dc-095e-4437-a2d7-23a1addf255b","Type":"ContainerStarted","Data":"d15e5978ef6a7c452d65969e5487411857b579b87bacb4897d83f59687901ba1"} Nov 03 02:46:22.632113 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:22.632028 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ca6ef4-7f7486c797-6kt8x" event={"ID":"b22351dc-095e-4437-a2d7-23a1addf255b","Type":"ContainerStarted","Data":"bfcc308ec0917d0154aac8433ea76548d7669c3b8160a0f40edf2ac391c49b41"} Nov 03 02:46:22.632113 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:22.632076 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-ca6ef4-7f7486c797-6kt8x" Nov 03 02:46:22.632527 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:22.632500 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:46:22.632604 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:22.632569 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:46:22.632643 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:22.632611 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:46:22.633743 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:22.633718 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ca6ef4-74f5d66687-d4vr2" event={"ID":"dd54f6c1-849d-49e1-9d0c-ce8f8e7ccc96","Type":"ContainerStarted","Data":"7a88d9a9c4e94abc57ffe8bad693960985c955457d8c3cf9994194b8d0232cb6"} Nov 03 02:46:22.633830 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:22.633763 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ca6ef4-74f5d66687-d4vr2" event={"ID":"dd54f6c1-849d-49e1-9d0c-ce8f8e7ccc96","Type":"ContainerStarted","Data":"31fae1962c9104a7174228e0877d5b61b78d91bfd9aee84ca12738b5a9aee92d"} Nov 03 02:46:22.850383 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:22.850282 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-ca6ef4-7f7486c797-6kt8x" podStartSLOduration=1.8502617799999999 podStartE2EDuration="1.85026178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:46:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:46:22.849242412 +0000 UTC m=+3260.108440564" watchObservedRunningTime="2026-11-03 02:46:22.85026178 +0000 UTC m=+3260.109459936" Nov 03 02:46:22.851739 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:22.851689 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33cae564-16a9-42bb-a1a5-dc5c3257f56b" path="/var/lib/kubelet/pods/33cae564-16a9-42bb-a1a5-dc5c3257f56b/volumes" Nov 03 02:46:22.962702 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:22.962549 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ca6ef4-74f5d66687-d4vr2" podStartSLOduration=1.962530724 podStartE2EDuration="1.962530724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:46:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:46:22.960145289 +0000 UTC m=+3260.219343443" watchObservedRunningTime="2026-11-03 02:46:22.962530724 +0000 UTC m=+3260.221728877" Nov 03 02:46:23.642225 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:23.642181 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-038c21-openshift-default-5d5c88569c-4r995" event={"ID":"f7c23f7b-4a52-4a24-8fce-ba087104fca5","Type":"ContainerStarted","Data":"cd1a796693e719811c9ea87d3f42db5145e34c4a7353a3a61e0b2c2205843ec8"} Nov 03 02:46:23.642751 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:23.642736 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ca6ef4-74f5d66687-d4vr2" Nov 03 02:46:23.917191 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:46:23.917099 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:46:24.151557 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:24.151520 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-038c21-openshift-default-5d5c88569c-4r995" Nov 03 02:46:24.151746 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:24.151696 5605 patch_prober.go:29] interesting pod/e2e-explicit-038c21-openshift-default-5d5c88569c-4r995 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Nov 03 02:46:24.151796 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:24.151742 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-038c21-openshift-default-5d5c88569c-4r995" podUID="f7c23f7b-4a52-4a24-8fce-ba087104fca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Nov 03 02:46:25.152232 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:25.152188 5605 patch_prober.go:29] interesting pod/e2e-explicit-038c21-openshift-default-5d5c88569c-4r995 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Nov 03 02:46:25.152777 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:25.152264 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-038c21-openshift-default-5d5c88569c-4r995" podUID="f7c23f7b-4a52-4a24-8fce-ba087104fca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Nov 03 02:46:25.286828 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:25.286766 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-ffba43/auth-test-model-ffba43-kserve-5bddd45877-ln2tc" podUID="f0ff182d-490c-47e5-b098-e87bbf5f0e1c" containerName="main" containerID="cri-o://1fb2107621324bc4809f1eec256547c30696316013b5bfd40edab3286cbdbd03" gracePeriod=45 Nov 03 02:46:25.304938 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:25.304896 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-ffba43/auth-test-model-ffba43-kserve-5bddd45877-ln2tc" podUID="f0ff182d-490c-47e5-b098-e87bbf5f0e1c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.157:8000/ready\": dial tcp 10.134.0.157:8000: connect: connection refused" Nov 03 02:46:25.597597 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:25.597568 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-ffba43/auth-test-model-ffba43-kserve-5bddd45877-ln2tc" Nov 03 02:46:25.681068 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:25.681036 5605 generic.go:356] "Generic (PLEG): container finished" podID="f0ff182d-490c-47e5-b098-e87bbf5f0e1c" containerID="1fb2107621324bc4809f1eec256547c30696316013b5bfd40edab3286cbdbd03" exitCode=0 Nov 03 02:46:25.681250 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:25.681114 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-ffba43/auth-test-model-ffba43-kserve-5bddd45877-ln2tc" Nov 03 02:46:25.681250 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:25.681121 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-ffba43/auth-test-model-ffba43-kserve-5bddd45877-ln2tc" event={"ID":"f0ff182d-490c-47e5-b098-e87bbf5f0e1c","Type":"ContainerDied","Data":"1fb2107621324bc4809f1eec256547c30696316013b5bfd40edab3286cbdbd03"} Nov 03 02:46:25.681250 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:25.681171 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-ffba43/auth-test-model-ffba43-kserve-5bddd45877-ln2tc" event={"ID":"f0ff182d-490c-47e5-b098-e87bbf5f0e1c","Type":"ContainerDied","Data":"716017d907a27efd9f4832ab3ff2bd3792237a8e8829eaa912597c847e2e771d"} Nov 03 02:46:25.681250 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:25.681208 5605 scope.go:122] "RemoveContainer" containerID="1fb2107621324bc4809f1eec256547c30696316013b5bfd40edab3286cbdbd03" Nov 03 02:46:25.691127 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:25.691103 5605 scope.go:122] "RemoveContainer" containerID="1fb2107621324bc4809f1eec256547c30696316013b5bfd40edab3286cbdbd03" Nov 03 02:46:25.691423 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:46:25.691401 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fb2107621324bc4809f1eec256547c30696316013b5bfd40edab3286cbdbd03\": container with ID starting with 1fb2107621324bc4809f1eec256547c30696316013b5bfd40edab3286cbdbd03 not found: ID does not exist" containerID="1fb2107621324bc4809f1eec256547c30696316013b5bfd40edab3286cbdbd03" Nov 03 02:46:25.691519 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:25.691435 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fb2107621324bc4809f1eec256547c30696316013b5bfd40edab3286cbdbd03"} err="failed to get container status \"1fb2107621324bc4809f1eec256547c30696316013b5bfd40edab3286cbdbd03\": rpc error: code = NotFound desc = could not find container \"1fb2107621324bc4809f1eec256547c30696316013b5bfd40edab3286cbdbd03\": container with ID starting with 1fb2107621324bc4809f1eec256547c30696316013b5bfd40edab3286cbdbd03 not found: ID does not exist" Nov 03 02:46:25.714525 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:25.714434 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0ff182d-490c-47e5-b098-e87bbf5f0e1c-tls-certs\" (UniqueName: \"kubernetes.io/secret/f0ff182d-490c-47e5-b098-e87bbf5f0e1c-tls-certs\") pod \"f0ff182d-490c-47e5-b098-e87bbf5f0e1c\" (UID: \"f0ff182d-490c-47e5-b098-e87bbf5f0e1c\") " Nov 03 02:46:25.714525 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:25.714490 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0ff182d-490c-47e5-b098-e87bbf5f0e1c-kube-api-access-8wj5p\" (UniqueName: \"kubernetes.io/projected/f0ff182d-490c-47e5-b098-e87bbf5f0e1c-kube-api-access-8wj5p\") pod \"f0ff182d-490c-47e5-b098-e87bbf5f0e1c\" (UID: \"f0ff182d-490c-47e5-b098-e87bbf5f0e1c\") " Nov 03 02:46:25.714760 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:25.714620 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0ff182d-490c-47e5-b098-e87bbf5f0e1c-home\" (UniqueName: \"kubernetes.io/empty-dir/f0ff182d-490c-47e5-b098-e87bbf5f0e1c-home\") pod \"f0ff182d-490c-47e5-b098-e87bbf5f0e1c\" (UID: \"f0ff182d-490c-47e5-b098-e87bbf5f0e1c\") " Nov 03 02:46:25.714760 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:25.714672 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0ff182d-490c-47e5-b098-e87bbf5f0e1c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0ff182d-490c-47e5-b098-e87bbf5f0e1c-tmp-dir\") pod \"f0ff182d-490c-47e5-b098-e87bbf5f0e1c\" (UID: \"f0ff182d-490c-47e5-b098-e87bbf5f0e1c\") " Nov 03 02:46:25.714760 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:25.714698 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0ff182d-490c-47e5-b098-e87bbf5f0e1c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0ff182d-490c-47e5-b098-e87bbf5f0e1c-dshm\") pod \"f0ff182d-490c-47e5-b098-e87bbf5f0e1c\" (UID: \"f0ff182d-490c-47e5-b098-e87bbf5f0e1c\") " Nov 03 02:46:25.714983 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:25.714779 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0ff182d-490c-47e5-b098-e87bbf5f0e1c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0ff182d-490c-47e5-b098-e87bbf5f0e1c-model-cache\") pod \"f0ff182d-490c-47e5-b098-e87bbf5f0e1c\" (UID: \"f0ff182d-490c-47e5-b098-e87bbf5f0e1c\") " Nov 03 02:46:25.715571 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:25.715078 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0ff182d-490c-47e5-b098-e87bbf5f0e1c-tmp-dir" pod "f0ff182d-490c-47e5-b098-e87bbf5f0e1c" (UID: "f0ff182d-490c-47e5-b098-e87bbf5f0e1c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:46:25.715571 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:25.715287 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0ff182d-490c-47e5-b098-e87bbf5f0e1c-home" pod "f0ff182d-490c-47e5-b098-e87bbf5f0e1c" (UID: "f0ff182d-490c-47e5-b098-e87bbf5f0e1c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:46:25.715571 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:25.715298 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0ff182d-490c-47e5-b098-e87bbf5f0e1c-model-cache" pod "f0ff182d-490c-47e5-b098-e87bbf5f0e1c" (UID: "f0ff182d-490c-47e5-b098-e87bbf5f0e1c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:46:25.715571 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:25.715484 5605 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0ff182d-490c-47e5-b098-e87bbf5f0e1c-home\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:25.715571 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:25.715529 5605 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0ff182d-490c-47e5-b098-e87bbf5f0e1c-tmp-dir\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:25.715571 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:25.715545 5605 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0ff182d-490c-47e5-b098-e87bbf5f0e1c-model-cache\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:25.717297 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:25.717271 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0ff182d-490c-47e5-b098-e87bbf5f0e1c-tls-certs" pod "f0ff182d-490c-47e5-b098-e87bbf5f0e1c" (UID: "f0ff182d-490c-47e5-b098-e87bbf5f0e1c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:46:25.717619 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:25.717601 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0ff182d-490c-47e5-b098-e87bbf5f0e1c-dshm" pod "f0ff182d-490c-47e5-b098-e87bbf5f0e1c" (UID: "f0ff182d-490c-47e5-b098-e87bbf5f0e1c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:46:25.718958 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:25.718939 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0ff182d-490c-47e5-b098-e87bbf5f0e1c-kube-api-access-8wj5p" pod "f0ff182d-490c-47e5-b098-e87bbf5f0e1c" (UID: "f0ff182d-490c-47e5-b098-e87bbf5f0e1c"). InnerVolumeSpecName "kube-api-access-8wj5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:46:25.816277 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:25.816242 5605 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0ff182d-490c-47e5-b098-e87bbf5f0e1c-dshm\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:25.816277 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:25.816271 5605 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0ff182d-490c-47e5-b098-e87bbf5f0e1c-tls-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:25.816277 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:25.816283 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wj5p\" (UniqueName: \"kubernetes.io/projected/f0ff182d-490c-47e5-b098-e87bbf5f0e1c-kube-api-access-8wj5p\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:26.094870 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:26.094837 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-ffba43/auth-test-model-ffba43-kserve-5bddd45877-ln2tc"] Nov 03 02:46:26.097505 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:26.097478 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-ffba43/auth-test-model-ffba43-kserve-5bddd45877-ln2tc"] Nov 03 02:46:26.151638 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:26.151604 5605 patch_prober.go:29] interesting pod/e2e-explicit-038c21-openshift-default-5d5c88569c-4r995 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Nov 03 02:46:26.151812 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:26.151661 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-038c21-openshift-default-5d5c88569c-4r995" podUID="f7c23f7b-4a52-4a24-8fce-ba087104fca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Nov 03 02:46:26.850272 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:26.850236 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0ff182d-490c-47e5-b098-e87bbf5f0e1c" path="/var/lib/kubelet/pods/f0ff182d-490c-47e5-b098-e87bbf5f0e1c/volumes" Nov 03 02:46:27.151761 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:27.151671 5605 patch_prober.go:29] interesting pod/e2e-explicit-038c21-openshift-default-5d5c88569c-4r995 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Nov 03 02:46:27.151761 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:27.151735 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-038c21-openshift-default-5d5c88569c-4r995" podUID="f7c23f7b-4a52-4a24-8fce-ba087104fca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Nov 03 02:46:27.390332 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:27.390294 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-604da0-6f5db6dc4c-98d9d"] Nov 03 02:46:27.390839 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:27.390826 5605 cpu_manager.go:414] "containerMap: removing container" podUID="f0ff182d-490c-47e5-b098-e87bbf5f0e1c" containerName="main" Nov 03 02:46:27.390886 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:27.390840 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0ff182d-490c-47e5-b098-e87bbf5f0e1c" containerName="main" Nov 03 02:46:27.390977 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:27.390968 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0ff182d-490c-47e5-b098-e87bbf5f0e1c" containerName="main" Nov 03 02:46:27.395664 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:27.395641 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-604da0-6f5db6dc4c-98d9d" Nov 03 02:46:27.438962 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:27.438934 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd510465-adee-438c-908b-79808324282a-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-604da0-6f5db6dc4c-98d9d\" (UID: \"bd510465-adee-438c-908b-79808324282a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-604da0-6f5db6dc4c-98d9d" Nov 03 02:46:27.439106 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:27.438994 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfpv9\" (UniqueName: \"kubernetes.io/projected/bd510465-adee-438c-908b-79808324282a-kube-api-access-nfpv9\") pod \"payload-pre-processing-e2e-extmod-a-604da0-6f5db6dc4c-98d9d\" (UID: \"bd510465-adee-438c-908b-79808324282a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-604da0-6f5db6dc4c-98d9d" Nov 03 02:46:27.445068 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:27.445035 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-604da0\"" Nov 03 02:46:27.448967 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:27.448945 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-604da0-6f5db6dc4c-98d9d"] Nov 03 02:46:27.548381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:27.546249 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd510465-adee-438c-908b-79808324282a-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-604da0-6f5db6dc4c-98d9d\" (UID: \"bd510465-adee-438c-908b-79808324282a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-604da0-6f5db6dc4c-98d9d" Nov 03 02:46:27.548381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:27.546374 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfpv9\" (UniqueName: \"kubernetes.io/projected/bd510465-adee-438c-908b-79808324282a-kube-api-access-nfpv9\") pod \"payload-pre-processing-e2e-extmod-a-604da0-6f5db6dc4c-98d9d\" (UID: \"bd510465-adee-438c-908b-79808324282a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-604da0-6f5db6dc4c-98d9d" Nov 03 02:46:27.548381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:27.547611 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd510465-adee-438c-908b-79808324282a-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-604da0-6f5db6dc4c-98d9d\" (UID: \"bd510465-adee-438c-908b-79808324282a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-604da0-6f5db6dc4c-98d9d" Nov 03 02:46:27.600994 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:27.599972 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-604da0-77cc94d8c7-bssrs"] Nov 03 02:46:27.601492 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:27.601466 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfpv9\" (UniqueName: \"kubernetes.io/projected/bd510465-adee-438c-908b-79808324282a-kube-api-access-nfpv9\") pod \"payload-pre-processing-e2e-extmod-a-604da0-6f5db6dc4c-98d9d\" (UID: \"bd510465-adee-438c-908b-79808324282a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-604da0-6f5db6dc4c-98d9d" Nov 03 02:46:27.604659 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:27.604636 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-604da0-77cc94d8c7-bssrs" Nov 03 02:46:27.626596 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:27.626572 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-604da0-77cc94d8c7-bssrs"] Nov 03 02:46:27.647455 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:27.647417 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e7953a5-9b97-41de-96af-c3827c82254c-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-604da0-77cc94d8c7-bssrs\" (UID: \"2e7953a5-9b97-41de-96af-c3827c82254c\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-604da0-77cc94d8c7-bssrs" Nov 03 02:46:27.647597 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:27.647577 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pj95\" (UniqueName: \"kubernetes.io/projected/2e7953a5-9b97-41de-96af-c3827c82254c-kube-api-access-6pj95\") pod \"payload-processing-e2e-extmod-a-604da0-77cc94d8c7-bssrs\" (UID: \"2e7953a5-9b97-41de-96af-c3827c82254c\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-604da0-77cc94d8c7-bssrs" Nov 03 02:46:27.653621 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:27.653547 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-604da0-dockercfg-bc6vw\"" Nov 03 02:46:27.709115 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:27.709076 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-604da0-6f5db6dc4c-98d9d" Nov 03 02:46:27.749191 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:27.749135 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pj95\" (UniqueName: \"kubernetes.io/projected/2e7953a5-9b97-41de-96af-c3827c82254c-kube-api-access-6pj95\") pod \"payload-processing-e2e-extmod-a-604da0-77cc94d8c7-bssrs\" (UID: \"2e7953a5-9b97-41de-96af-c3827c82254c\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-604da0-77cc94d8c7-bssrs" Nov 03 02:46:27.749408 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:27.749388 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e7953a5-9b97-41de-96af-c3827c82254c-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-604da0-77cc94d8c7-bssrs\" (UID: \"2e7953a5-9b97-41de-96af-c3827c82254c\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-604da0-77cc94d8c7-bssrs" Nov 03 02:46:27.750748 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:27.750267 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e7953a5-9b97-41de-96af-c3827c82254c-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-604da0-77cc94d8c7-bssrs\" (UID: \"2e7953a5-9b97-41de-96af-c3827c82254c\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-604da0-77cc94d8c7-bssrs" Nov 03 02:46:27.809392 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:27.806180 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pj95\" (UniqueName: \"kubernetes.io/projected/2e7953a5-9b97-41de-96af-c3827c82254c-kube-api-access-6pj95\") pod \"payload-processing-e2e-extmod-a-604da0-77cc94d8c7-bssrs\" (UID: \"2e7953a5-9b97-41de-96af-c3827c82254c\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-604da0-77cc94d8c7-bssrs" Nov 03 02:46:27.928193 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:27.928106 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-604da0-77cc94d8c7-bssrs" Nov 03 02:46:28.102507 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:28.102474 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-604da0-6f5db6dc4c-98d9d"] Nov 03 02:46:28.108257 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:46:28.108229 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd510465_adee_438c_908b_79808324282a.slice/crio-46be6258471609ead0d359296cd03612095e5b8202167616546cc5a17a7fb487 WatchSource:0}: Error finding container 46be6258471609ead0d359296cd03612095e5b8202167616546cc5a17a7fb487: Status 404 returned error can't find the container with id 46be6258471609ead0d359296cd03612095e5b8202167616546cc5a17a7fb487 Nov 03 02:46:28.151571 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:28.151539 5605 patch_prober.go:29] interesting pod/e2e-explicit-038c21-openshift-default-5d5c88569c-4r995 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Nov 03 02:46:28.151679 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:28.151602 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-038c21-openshift-default-5d5c88569c-4r995" podUID="f7c23f7b-4a52-4a24-8fce-ba087104fca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Nov 03 02:46:28.324448 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:28.324410 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-604da0-77cc94d8c7-bssrs"] Nov 03 02:46:28.329287 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:46:28.329260 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e7953a5_9b97_41de_96af_c3827c82254c.slice/crio-729fcb729fb2ef0067feeb155dcfb1408f1ddc2883b33e6305116321d0da65f8 WatchSource:0}: Error finding container 729fcb729fb2ef0067feeb155dcfb1408f1ddc2883b33e6305116321d0da65f8: Status 404 returned error can't find the container with id 729fcb729fb2ef0067feeb155dcfb1408f1ddc2883b33e6305116321d0da65f8 Nov 03 02:46:28.702846 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:28.702793 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-604da0-77cc94d8c7-bssrs" event={"ID":"2e7953a5-9b97-41de-96af-c3827c82254c","Type":"ContainerStarted","Data":"1c430c3f1820aee3a0e99983aba6956ce12d7bc17832cb3a892ecd2e03b0d8ea"} Nov 03 02:46:28.703062 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:28.702865 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-604da0-77cc94d8c7-bssrs" event={"ID":"2e7953a5-9b97-41de-96af-c3827c82254c","Type":"ContainerStarted","Data":"729fcb729fb2ef0067feeb155dcfb1408f1ddc2883b33e6305116321d0da65f8"} Nov 03 02:46:28.703062 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:28.702893 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-604da0-77cc94d8c7-bssrs" Nov 03 02:46:28.704547 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:28.704516 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-604da0-6f5db6dc4c-98d9d" event={"ID":"bd510465-adee-438c-908b-79808324282a","Type":"ContainerStarted","Data":"55d2dd3c161752bbb98fb98aed6722bb94c58c289069b46d2d32fdd759621f6a"} Nov 03 02:46:28.704665 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:28.704563 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-604da0-6f5db6dc4c-98d9d" event={"ID":"bd510465-adee-438c-908b-79808324282a","Type":"ContainerStarted","Data":"46be6258471609ead0d359296cd03612095e5b8202167616546cc5a17a7fb487"} Nov 03 02:46:28.704665 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:28.704656 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-604da0-6f5db6dc4c-98d9d" Nov 03 02:46:28.821266 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:28.821219 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-604da0-77cc94d8c7-bssrs" podStartSLOduration=1.821204947 podStartE2EDuration="1.821204947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:46:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:46:28.819554931 +0000 UTC m=+3266.078753084" watchObservedRunningTime="2026-11-03 02:46:28.821204947 +0000 UTC m=+3266.080403100" Nov 03 02:46:28.927330 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:28.927268 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-604da0-6f5db6dc4c-98d9d" podStartSLOduration=1.92725487 podStartE2EDuration="1.92725487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:46:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:46:28.925515252 +0000 UTC m=+3266.184713409" watchObservedRunningTime="2026-11-03 02:46:28.92725487 +0000 UTC m=+3266.186453023" Nov 03 02:46:29.156580 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:29.156551 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-038c21-openshift-default-5d5c88569c-4r995" Nov 03 02:46:29.157012 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:29.156891 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-038c21-openshift-default-5d5c88569c-4r995" Nov 03 02:46:29.157712 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:29.157690 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-038c21-openshift-default-5d5c88569c-4r995" Nov 03 02:46:33.645453 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:33.645423 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-ca6ef4-7f7486c797-6kt8x" Nov 03 02:46:34.651454 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:34.651427 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ca6ef4-74f5d66687-d4vr2" Nov 03 02:46:34.964221 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:34.964137 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-63a61e/noten-model-63a61e-kserve-854bff945-8l56c" podUID="9c56126f-02de-4af6-9f2e-c7ba74fc4a5f" containerName="main" containerID="cri-o://556f5dc078ba121bba06e4a888abef9e18f98275fd6866f2a632038d98c7696c" gracePeriod=45 Nov 03 02:46:35.206443 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:35.206421 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-63a61e/noten-model-63a61e-kserve-854bff945-8l56c" Nov 03 02:46:35.345811 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:35.345771 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c56126f-02de-4af6-9f2e-c7ba74fc4a5f-home\" (UniqueName: \"kubernetes.io/empty-dir/9c56126f-02de-4af6-9f2e-c7ba74fc4a5f-home\") pod \"9c56126f-02de-4af6-9f2e-c7ba74fc4a5f\" (UID: \"9c56126f-02de-4af6-9f2e-c7ba74fc4a5f\") " Nov 03 02:46:35.346018 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:35.345878 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c56126f-02de-4af6-9f2e-c7ba74fc4a5f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c56126f-02de-4af6-9f2e-c7ba74fc4a5f-dshm\") pod \"9c56126f-02de-4af6-9f2e-c7ba74fc4a5f\" (UID: \"9c56126f-02de-4af6-9f2e-c7ba74fc4a5f\") " Nov 03 02:46:35.346097 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:35.346035 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c56126f-02de-4af6-9f2e-c7ba74fc4a5f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c56126f-02de-4af6-9f2e-c7ba74fc4a5f-tmp-dir\") pod \"9c56126f-02de-4af6-9f2e-c7ba74fc4a5f\" (UID: \"9c56126f-02de-4af6-9f2e-c7ba74fc4a5f\") " Nov 03 02:46:35.346097 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:35.346074 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c56126f-02de-4af6-9f2e-c7ba74fc4a5f-kube-api-access-xl52t\" (UniqueName: \"kubernetes.io/projected/9c56126f-02de-4af6-9f2e-c7ba74fc4a5f-kube-api-access-xl52t\") pod \"9c56126f-02de-4af6-9f2e-c7ba74fc4a5f\" (UID: \"9c56126f-02de-4af6-9f2e-c7ba74fc4a5f\") " Nov 03 02:46:35.346209 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:35.346086 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c56126f-02de-4af6-9f2e-c7ba74fc4a5f-home" pod "9c56126f-02de-4af6-9f2e-c7ba74fc4a5f" (UID: "9c56126f-02de-4af6-9f2e-c7ba74fc4a5f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:46:35.346209 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:35.346110 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c56126f-02de-4af6-9f2e-c7ba74fc4a5f-tls-certs\" (UniqueName: \"kubernetes.io/secret/9c56126f-02de-4af6-9f2e-c7ba74fc4a5f-tls-certs\") pod \"9c56126f-02de-4af6-9f2e-c7ba74fc4a5f\" (UID: \"9c56126f-02de-4af6-9f2e-c7ba74fc4a5f\") " Nov 03 02:46:35.346209 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:35.346144 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c56126f-02de-4af6-9f2e-c7ba74fc4a5f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c56126f-02de-4af6-9f2e-c7ba74fc4a5f-model-cache\") pod \"9c56126f-02de-4af6-9f2e-c7ba74fc4a5f\" (UID: \"9c56126f-02de-4af6-9f2e-c7ba74fc4a5f\") " Nov 03 02:46:35.346379 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:35.346335 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c56126f-02de-4af6-9f2e-c7ba74fc4a5f-tmp-dir" pod "9c56126f-02de-4af6-9f2e-c7ba74fc4a5f" (UID: "9c56126f-02de-4af6-9f2e-c7ba74fc4a5f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:46:35.346482 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:35.346460 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c56126f-02de-4af6-9f2e-c7ba74fc4a5f-model-cache" pod "9c56126f-02de-4af6-9f2e-c7ba74fc4a5f" (UID: "9c56126f-02de-4af6-9f2e-c7ba74fc4a5f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:46:35.346692 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:35.346675 5605 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c56126f-02de-4af6-9f2e-c7ba74fc4a5f-home\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:35.346763 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:35.346695 5605 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c56126f-02de-4af6-9f2e-c7ba74fc4a5f-tmp-dir\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:35.346763 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:35.346708 5605 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c56126f-02de-4af6-9f2e-c7ba74fc4a5f-model-cache\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:35.348216 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:35.348186 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c56126f-02de-4af6-9f2e-c7ba74fc4a5f-dshm" pod "9c56126f-02de-4af6-9f2e-c7ba74fc4a5f" (UID: "9c56126f-02de-4af6-9f2e-c7ba74fc4a5f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:46:35.348430 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:35.348411 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c56126f-02de-4af6-9f2e-c7ba74fc4a5f-kube-api-access-xl52t" pod "9c56126f-02de-4af6-9f2e-c7ba74fc4a5f" (UID: "9c56126f-02de-4af6-9f2e-c7ba74fc4a5f"). InnerVolumeSpecName "kube-api-access-xl52t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:46:35.348524 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:35.348478 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c56126f-02de-4af6-9f2e-c7ba74fc4a5f-tls-certs" pod "9c56126f-02de-4af6-9f2e-c7ba74fc4a5f" (UID: "9c56126f-02de-4af6-9f2e-c7ba74fc4a5f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:46:35.447860 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:35.447828 5605 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c56126f-02de-4af6-9f2e-c7ba74fc4a5f-dshm\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:35.447860 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:35.447858 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl52t\" (UniqueName: \"kubernetes.io/projected/9c56126f-02de-4af6-9f2e-c7ba74fc4a5f-kube-api-access-xl52t\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:35.447860 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:35.447868 5605 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c56126f-02de-4af6-9f2e-c7ba74fc4a5f-tls-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:35.748731 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:35.748653 5605 generic.go:356] "Generic (PLEG): container finished" podID="9c56126f-02de-4af6-9f2e-c7ba74fc4a5f" containerID="556f5dc078ba121bba06e4a888abef9e18f98275fd6866f2a632038d98c7696c" exitCode=0 Nov 03 02:46:35.748731 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:35.748695 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-63a61e/noten-model-63a61e-kserve-854bff945-8l56c" Nov 03 02:46:35.749183 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:35.748727 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-63a61e/noten-model-63a61e-kserve-854bff945-8l56c" event={"ID":"9c56126f-02de-4af6-9f2e-c7ba74fc4a5f","Type":"ContainerDied","Data":"556f5dc078ba121bba06e4a888abef9e18f98275fd6866f2a632038d98c7696c"} Nov 03 02:46:35.749183 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:35.748771 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-63a61e/noten-model-63a61e-kserve-854bff945-8l56c" event={"ID":"9c56126f-02de-4af6-9f2e-c7ba74fc4a5f","Type":"ContainerDied","Data":"d97a82402c81dcb8cb6dc190b6e0b0a3b32a2ee734e028e3c11a81d7bbadff5b"} Nov 03 02:46:35.749183 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:35.748787 5605 scope.go:122] "RemoveContainer" containerID="556f5dc078ba121bba06e4a888abef9e18f98275fd6866f2a632038d98c7696c" Nov 03 02:46:35.758925 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:35.758901 5605 scope.go:122] "RemoveContainer" containerID="556f5dc078ba121bba06e4a888abef9e18f98275fd6866f2a632038d98c7696c" Nov 03 02:46:35.759265 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:46:35.759240 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"556f5dc078ba121bba06e4a888abef9e18f98275fd6866f2a632038d98c7696c\": container with ID starting with 556f5dc078ba121bba06e4a888abef9e18f98275fd6866f2a632038d98c7696c not found: ID does not exist" containerID="556f5dc078ba121bba06e4a888abef9e18f98275fd6866f2a632038d98c7696c" Nov 03 02:46:35.759352 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:35.759276 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"556f5dc078ba121bba06e4a888abef9e18f98275fd6866f2a632038d98c7696c"} err="failed to get container status \"556f5dc078ba121bba06e4a888abef9e18f98275fd6866f2a632038d98c7696c\": rpc error: code = NotFound desc = could not find container \"556f5dc078ba121bba06e4a888abef9e18f98275fd6866f2a632038d98c7696c\": container with ID starting with 556f5dc078ba121bba06e4a888abef9e18f98275fd6866f2a632038d98c7696c not found: ID does not exist" Nov 03 02:46:35.902090 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:35.902066 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-63a61e/noten-model-63a61e-kserve-854bff945-8l56c"] Nov 03 02:46:35.948716 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:35.948691 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-63a61e/noten-model-63a61e-kserve-854bff945-8l56c"] Nov 03 02:46:36.850780 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:36.850749 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c56126f-02de-4af6-9f2e-c7ba74fc4a5f" path="/var/lib/kubelet/pods/9c56126f-02de-4af6-9f2e-c7ba74fc4a5f/volumes" Nov 03 02:46:39.714306 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:39.714277 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-604da0-6f5db6dc4c-98d9d" Nov 03 02:46:39.714719 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:39.714346 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-604da0-77cc94d8c7-bssrs" Nov 03 02:46:40.891938 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:40.891899 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7238-51ddba"] Nov 03 02:46:40.892595 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:40.892573 5605 cpu_manager.go:414] "containerMap: removing container" podUID="9c56126f-02de-4af6-9f2e-c7ba74fc4a5f" containerName="main" Nov 03 02:46:40.892693 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:40.892596 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c56126f-02de-4af6-9f2e-c7ba74fc4a5f" containerName="main" Nov 03 02:46:40.892765 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:40.892732 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c56126f-02de-4af6-9f2e-c7ba74fc4a5f" containerName="main" Nov 03 02:46:40.897283 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:40.897261 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7238-51ddba" Nov 03 02:46:40.947727 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:40.947704 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7238-51ddba"] Nov 03 02:46:40.991353 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:40.991324 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-579jc\"" Nov 03 02:46:40.997294 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:40.997269 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7chf\" (UniqueName: \"kubernetes.io/projected/f429d9da-333f-459d-9aa1-da1533b0082b-kube-api-access-g7chf\") pod \"test-curl-7238-51ddba\" (UID: \"f429d9da-333f-459d-9aa1-da1533b0082b\") " pod="opendatahub/test-curl-7238-51ddba" Nov 03 02:46:41.098474 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:41.098447 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7chf\" (UniqueName: \"kubernetes.io/projected/f429d9da-333f-459d-9aa1-da1533b0082b-kube-api-access-g7chf\") pod \"test-curl-7238-51ddba\" (UID: \"f429d9da-333f-459d-9aa1-da1533b0082b\") " pod="opendatahub/test-curl-7238-51ddba" Nov 03 02:46:41.151746 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:41.151689 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7chf\" (UniqueName: \"kubernetes.io/projected/f429d9da-333f-459d-9aa1-da1533b0082b-kube-api-access-g7chf\") pod \"test-curl-7238-51ddba\" (UID: \"f429d9da-333f-459d-9aa1-da1533b0082b\") " pod="opendatahub/test-curl-7238-51ddba" Nov 03 02:46:41.210283 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:41.210262 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7238-51ddba" Nov 03 02:46:41.577830 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:41.577802 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7238-51ddba"] Nov 03 02:46:41.583264 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:46:41.583237 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf429d9da_333f_459d_9aa1_da1533b0082b.slice/crio-5188450a0584768b8e354e290edc0fbd034697cd28e26f95bb66daa26011ec31 WatchSource:0}: Error finding container 5188450a0584768b8e354e290edc0fbd034697cd28e26f95bb66daa26011ec31: Status 404 returned error can't find the container with id 5188450a0584768b8e354e290edc0fbd034697cd28e26f95bb66daa26011ec31 Nov 03 02:46:41.789857 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:41.789824 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7238-51ddba" event={"ID":"f429d9da-333f-459d-9aa1-da1533b0082b","Type":"ContainerStarted","Data":"fe13ce5040695560d9a8a4cdb4e738ddb69f9951c0031f3c5dbd7cf256a45372"} Nov 03 02:46:41.789961 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:41.789866 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7238-51ddba" event={"ID":"f429d9da-333f-459d-9aa1-da1533b0082b","Type":"ContainerStarted","Data":"5188450a0584768b8e354e290edc0fbd034697cd28e26f95bb66daa26011ec31"} Nov 03 02:46:41.895883 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:41.895806 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7238-51ddba" podStartSLOduration=1.895794042 podStartE2EDuration="1.895794042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:46:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:46:41.894411094 +0000 UTC m=+3279.153609245" watchObservedRunningTime="2026-11-03 02:46:41.895794042 +0000 UTC m=+3279.154992193" Nov 03 02:46:43.292244 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:43.292214 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7238-51ddba"] Nov 03 02:46:43.297645 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:43.297620 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7238-51ddba"] Nov 03 02:46:43.801969 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:43.801930 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7238-51ddba" podUID="f429d9da-333f-459d-9aa1-da1533b0082b" containerName="test-curl-7238-51ddba" containerID="cri-o://fe13ce5040695560d9a8a4cdb4e738ddb69f9951c0031f3c5dbd7cf256a45372" gracePeriod=2 Nov 03 02:46:43.850093 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:46:43.850056 5605 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7238-51ddba\" is forbidden: User \"system:node:ip-10-0-140-140.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-140.us-west-2.compute.internal' and this object" podUID="f429d9da-333f-459d-9aa1-da1533b0082b" pod="opendatahub/test-curl-7238-51ddba" Nov 03 02:46:45.696539 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:45.696505 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9"] Nov 03 02:46:45.697275 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:45.697253 5605 cpu_manager.go:414] "containerMap: removing container" podUID="f429d9da-333f-459d-9aa1-da1533b0082b" containerName="test-curl-7238-51ddba" Nov 03 02:46:45.697372 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:45.697278 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f429d9da-333f-459d-9aa1-da1533b0082b" containerName="test-curl-7238-51ddba" Nov 03 02:46:45.697486 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:45.697473 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="f429d9da-333f-459d-9aa1-da1533b0082b" containerName="test-curl-7238-51ddba" Nov 03 02:46:45.701655 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:45.701632 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9" Nov 03 02:46:45.750523 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:45.750497 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-e3cc05-openshift-default-dockercfg-4t924\"" Nov 03 02:46:45.801594 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:45.801566 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9"] Nov 03 02:46:45.850865 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:45.850257 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75df2f44-a9e5-47df-b676-6c0882b547e2-istiod-ca-cert\") pod \"e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9\" (UID: \"75df2f44-a9e5-47df-b676-6c0882b547e2\") " pod="openshift-ingress/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9" Nov 03 02:46:45.850865 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:45.850348 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75df2f44-a9e5-47df-b676-6c0882b547e2-istio-token\") pod \"e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9\" (UID: \"75df2f44-a9e5-47df-b676-6c0882b547e2\") " pod="openshift-ingress/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9" Nov 03 02:46:45.850865 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:45.850424 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2fpx\" (UniqueName: \"kubernetes.io/projected/75df2f44-a9e5-47df-b676-6c0882b547e2-kube-api-access-g2fpx\") pod \"e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9\" (UID: \"75df2f44-a9e5-47df-b676-6c0882b547e2\") " pod="openshift-ingress/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9" Nov 03 02:46:45.850865 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:45.850530 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75df2f44-a9e5-47df-b676-6c0882b547e2-credential-socket\") pod \"e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9\" (UID: \"75df2f44-a9e5-47df-b676-6c0882b547e2\") " pod="openshift-ingress/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9" Nov 03 02:46:45.850865 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:45.850573 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75df2f44-a9e5-47df-b676-6c0882b547e2-istio-podinfo\") pod \"e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9\" (UID: \"75df2f44-a9e5-47df-b676-6c0882b547e2\") " pod="openshift-ingress/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9" Nov 03 02:46:45.850865 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:45.850604 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75df2f44-a9e5-47df-b676-6c0882b547e2-workload-certs\") pod \"e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9\" (UID: \"75df2f44-a9e5-47df-b676-6c0882b547e2\") " pod="openshift-ingress/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9" Nov 03 02:46:45.850865 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:45.850635 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75df2f44-a9e5-47df-b676-6c0882b547e2-workload-socket\") pod \"e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9\" (UID: \"75df2f44-a9e5-47df-b676-6c0882b547e2\") " pod="openshift-ingress/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9" Nov 03 02:46:45.850865 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:45.850675 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75df2f44-a9e5-47df-b676-6c0882b547e2-istio-data\") pod \"e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9\" (UID: \"75df2f44-a9e5-47df-b676-6c0882b547e2\") " pod="openshift-ingress/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9" Nov 03 02:46:45.850865 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:45.850715 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75df2f44-a9e5-47df-b676-6c0882b547e2-istio-envoy\") pod \"e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9\" (UID: \"75df2f44-a9e5-47df-b676-6c0882b547e2\") " pod="openshift-ingress/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9" Nov 03 02:46:45.951760 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:45.951678 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75df2f44-a9e5-47df-b676-6c0882b547e2-istio-envoy\") pod \"e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9\" (UID: \"75df2f44-a9e5-47df-b676-6c0882b547e2\") " pod="openshift-ingress/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9" Nov 03 02:46:45.951894 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:45.951770 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75df2f44-a9e5-47df-b676-6c0882b547e2-istiod-ca-cert\") pod \"e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9\" (UID: \"75df2f44-a9e5-47df-b676-6c0882b547e2\") " pod="openshift-ingress/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9" Nov 03 02:46:45.951894 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:45.951825 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75df2f44-a9e5-47df-b676-6c0882b547e2-istio-token\") pod \"e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9\" (UID: \"75df2f44-a9e5-47df-b676-6c0882b547e2\") " pod="openshift-ingress/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9" Nov 03 02:46:45.951894 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:45.951878 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2fpx\" (UniqueName: \"kubernetes.io/projected/75df2f44-a9e5-47df-b676-6c0882b547e2-kube-api-access-g2fpx\") pod \"e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9\" (UID: \"75df2f44-a9e5-47df-b676-6c0882b547e2\") " pod="openshift-ingress/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9" Nov 03 02:46:45.952030 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:45.951981 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75df2f44-a9e5-47df-b676-6c0882b547e2-credential-socket\") pod \"e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9\" (UID: \"75df2f44-a9e5-47df-b676-6c0882b547e2\") " pod="openshift-ingress/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9" Nov 03 02:46:45.952030 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:45.952020 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75df2f44-a9e5-47df-b676-6c0882b547e2-istio-podinfo\") pod \"e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9\" (UID: \"75df2f44-a9e5-47df-b676-6c0882b547e2\") " pod="openshift-ingress/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9" Nov 03 02:46:45.952132 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:45.952054 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75df2f44-a9e5-47df-b676-6c0882b547e2-workload-certs\") pod \"e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9\" (UID: \"75df2f44-a9e5-47df-b676-6c0882b547e2\") " pod="openshift-ingress/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9" Nov 03 02:46:45.952132 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:45.952086 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75df2f44-a9e5-47df-b676-6c0882b547e2-workload-socket\") pod \"e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9\" (UID: \"75df2f44-a9e5-47df-b676-6c0882b547e2\") " pod="openshift-ingress/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9" Nov 03 02:46:45.952243 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:45.952127 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75df2f44-a9e5-47df-b676-6c0882b547e2-istio-data\") pod \"e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9\" (UID: \"75df2f44-a9e5-47df-b676-6c0882b547e2\") " pod="openshift-ingress/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9" Nov 03 02:46:45.952627 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:45.952568 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75df2f44-a9e5-47df-b676-6c0882b547e2-workload-certs\") pod \"e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9\" (UID: \"75df2f44-a9e5-47df-b676-6c0882b547e2\") " pod="openshift-ingress/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9" Nov 03 02:46:45.952627 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:45.952601 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75df2f44-a9e5-47df-b676-6c0882b547e2-istio-data\") pod \"e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9\" (UID: \"75df2f44-a9e5-47df-b676-6c0882b547e2\") " pod="openshift-ingress/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9" Nov 03 02:46:45.952803 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:45.952631 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75df2f44-a9e5-47df-b676-6c0882b547e2-istiod-ca-cert\") pod \"e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9\" (UID: \"75df2f44-a9e5-47df-b676-6c0882b547e2\") " pod="openshift-ingress/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9" Nov 03 02:46:45.952803 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:45.952637 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75df2f44-a9e5-47df-b676-6c0882b547e2-workload-socket\") pod \"e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9\" (UID: \"75df2f44-a9e5-47df-b676-6c0882b547e2\") " pod="openshift-ingress/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9" Nov 03 02:46:45.952803 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:45.952660 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75df2f44-a9e5-47df-b676-6c0882b547e2-credential-socket\") pod \"e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9\" (UID: \"75df2f44-a9e5-47df-b676-6c0882b547e2\") " pod="openshift-ingress/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9" Nov 03 02:46:45.954099 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:45.954074 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75df2f44-a9e5-47df-b676-6c0882b547e2-istio-envoy\") pod \"e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9\" (UID: \"75df2f44-a9e5-47df-b676-6c0882b547e2\") " pod="openshift-ingress/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9" Nov 03 02:46:45.954499 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:45.954481 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75df2f44-a9e5-47df-b676-6c0882b547e2-istio-podinfo\") pod \"e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9\" (UID: \"75df2f44-a9e5-47df-b676-6c0882b547e2\") " pod="openshift-ingress/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9" Nov 03 02:46:46.005517 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:46.005477 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75df2f44-a9e5-47df-b676-6c0882b547e2-istio-token\") pod \"e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9\" (UID: \"75df2f44-a9e5-47df-b676-6c0882b547e2\") " pod="openshift-ingress/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9" Nov 03 02:46:46.005753 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:46.005735 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2fpx\" (UniqueName: \"kubernetes.io/projected/75df2f44-a9e5-47df-b676-6c0882b547e2-kube-api-access-g2fpx\") pod \"e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9\" (UID: \"75df2f44-a9e5-47df-b676-6c0882b547e2\") " pod="openshift-ingress/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9" Nov 03 02:46:46.017487 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:46.017389 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9" Nov 03 02:46:46.054799 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:46.054776 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7238-51ddba" Nov 03 02:46:46.153946 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:46.153915 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f429d9da-333f-459d-9aa1-da1533b0082b-kube-api-access-g7chf\" (UniqueName: \"kubernetes.io/projected/f429d9da-333f-459d-9aa1-da1533b0082b-kube-api-access-g7chf\") pod \"f429d9da-333f-459d-9aa1-da1533b0082b\" (UID: \"f429d9da-333f-459d-9aa1-da1533b0082b\") " Nov 03 02:46:46.155911 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:46.155889 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f429d9da-333f-459d-9aa1-da1533b0082b-kube-api-access-g7chf" pod "f429d9da-333f-459d-9aa1-da1533b0082b" (UID: "f429d9da-333f-459d-9aa1-da1533b0082b"). InnerVolumeSpecName "kube-api-access-g7chf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:46:46.255616 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:46.255551 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7chf\" (UniqueName: \"kubernetes.io/projected/f429d9da-333f-459d-9aa1-da1533b0082b-kube-api-access-g7chf\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:46.446608 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:46.446574 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9"] Nov 03 02:46:46.452304 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:46:46.452274 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75df2f44_a9e5_47df_b676_6c0882b547e2.slice/crio-e0e59e4a44883b9a6a64475681a1664962af5fdda882bddc519888e0162b3805 WatchSource:0}: Error finding container e0e59e4a44883b9a6a64475681a1664962af5fdda882bddc519888e0162b3805: Status 404 returned error can't find the container with id e0e59e4a44883b9a6a64475681a1664962af5fdda882bddc519888e0162b3805 Nov 03 02:46:46.455098 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:46.455063 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:46:46.455213 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:46.455129 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:46:46.455213 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:46.455164 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:46:46.831886 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:46.831849 5605 generic.go:356] "Generic (PLEG): container finished" podID="f429d9da-333f-459d-9aa1-da1533b0082b" containerID="fe13ce5040695560d9a8a4cdb4e738ddb69f9951c0031f3c5dbd7cf256a45372" exitCode=137 Nov 03 02:46:46.832303 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:46.831895 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7238-51ddba" Nov 03 02:46:46.832303 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:46.831899 5605 scope.go:122] "RemoveContainer" containerID="fe13ce5040695560d9a8a4cdb4e738ddb69f9951c0031f3c5dbd7cf256a45372" Nov 03 02:46:46.834028 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:46.834001 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9" event={"ID":"75df2f44-a9e5-47df-b676-6c0882b547e2","Type":"ContainerStarted","Data":"3d41884c4d69f4ffac583df5d18221b6659d52081231a80eb1a485f69c91d1c2"} Nov 03 02:46:46.834164 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:46.834050 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9" event={"ID":"75df2f44-a9e5-47df-b676-6c0882b547e2","Type":"ContainerStarted","Data":"e0e59e4a44883b9a6a64475681a1664962af5fdda882bddc519888e0162b3805"} Nov 03 02:46:46.843061 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:46.843041 5605 scope.go:122] "RemoveContainer" containerID="fe13ce5040695560d9a8a4cdb4e738ddb69f9951c0031f3c5dbd7cf256a45372" Nov 03 02:46:46.843386 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:46:46.843343 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe13ce5040695560d9a8a4cdb4e738ddb69f9951c0031f3c5dbd7cf256a45372\": container with ID starting with fe13ce5040695560d9a8a4cdb4e738ddb69f9951c0031f3c5dbd7cf256a45372 not found: ID does not exist" containerID="fe13ce5040695560d9a8a4cdb4e738ddb69f9951c0031f3c5dbd7cf256a45372" Nov 03 02:46:46.843487 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:46.843404 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe13ce5040695560d9a8a4cdb4e738ddb69f9951c0031f3c5dbd7cf256a45372"} err="failed to get container status \"fe13ce5040695560d9a8a4cdb4e738ddb69f9951c0031f3c5dbd7cf256a45372\": rpc error: code = NotFound desc = could not find container \"fe13ce5040695560d9a8a4cdb4e738ddb69f9951c0031f3c5dbd7cf256a45372\": container with ID starting with fe13ce5040695560d9a8a4cdb4e738ddb69f9951c0031f3c5dbd7cf256a45372 not found: ID does not exist" Nov 03 02:46:46.850685 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:46.850662 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f429d9da-333f-459d-9aa1-da1533b0082b" path="/var/lib/kubelet/pods/f429d9da-333f-459d-9aa1-da1533b0082b/volumes" Nov 03 02:46:46.948990 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:46.948944 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9" podStartSLOduration=1.9489326550000001 podStartE2EDuration="1.948932655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:46:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:46:46.946893532 +0000 UTC m=+3284.206091685" watchObservedRunningTime="2026-11-03 02:46:46.948932655 +0000 UTC m=+3284.208130808" Nov 03 02:46:47.017976 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:47.017949 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9" Nov 03 02:46:47.023271 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:47.023254 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9" Nov 03 02:46:47.841264 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:47.841231 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9" Nov 03 02:46:47.842105 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:47.842087 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9" Nov 03 02:46:48.319608 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:48.319581 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:46:53.428717 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:53.428682 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e3cc05-66dfb8cc6c-8xgvh"] Nov 03 02:46:53.433715 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:53.433692 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e3cc05-66dfb8cc6c-8xgvh" Nov 03 02:46:53.439649 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:53.439626 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9bpv\" (UniqueName: \"kubernetes.io/projected/1773b46a-1dde-4944-b001-06cda4bedcd6-kube-api-access-m9bpv\") pod \"payload-pre-processing-e2e-extmod-a-e3cc05-66dfb8cc6c-8xgvh\" (UID: \"1773b46a-1dde-4944-b001-06cda4bedcd6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e3cc05-66dfb8cc6c-8xgvh" Nov 03 02:46:53.439759 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:53.439675 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1773b46a-1dde-4944-b001-06cda4bedcd6-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-e3cc05-66dfb8cc6c-8xgvh\" (UID: \"1773b46a-1dde-4944-b001-06cda4bedcd6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e3cc05-66dfb8cc6c-8xgvh" Nov 03 02:46:53.485572 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:53.485541 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e3cc05-66dfb8cc6c-8xgvh"] Nov 03 02:46:53.532386 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:53.530460 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-e3cc05\"" Nov 03 02:46:53.532386 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:53.530604 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-e3cc05-dockercfg-dhc9m\"" Nov 03 02:46:53.555380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:53.552159 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1773b46a-1dde-4944-b001-06cda4bedcd6-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-e3cc05-66dfb8cc6c-8xgvh\" (UID: \"1773b46a-1dde-4944-b001-06cda4bedcd6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e3cc05-66dfb8cc6c-8xgvh" Nov 03 02:46:53.556836 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:53.554108 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1773b46a-1dde-4944-b001-06cda4bedcd6-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-e3cc05-66dfb8cc6c-8xgvh\" (UID: \"1773b46a-1dde-4944-b001-06cda4bedcd6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e3cc05-66dfb8cc6c-8xgvh" Nov 03 02:46:53.557037 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:53.556984 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9bpv\" (UniqueName: \"kubernetes.io/projected/1773b46a-1dde-4944-b001-06cda4bedcd6-kube-api-access-m9bpv\") pod \"payload-pre-processing-e2e-extmod-a-e3cc05-66dfb8cc6c-8xgvh\" (UID: \"1773b46a-1dde-4944-b001-06cda4bedcd6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e3cc05-66dfb8cc6c-8xgvh" Nov 03 02:46:53.574073 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:53.574036 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e3cc05-6468f7548f-rl2hr"] Nov 03 02:46:53.578411 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:53.578387 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-e3cc05-6468f7548f-rl2hr" Nov 03 02:46:53.610855 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:53.610828 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9bpv\" (UniqueName: \"kubernetes.io/projected/1773b46a-1dde-4944-b001-06cda4bedcd6-kube-api-access-m9bpv\") pod \"payload-pre-processing-e2e-extmod-a-e3cc05-66dfb8cc6c-8xgvh\" (UID: \"1773b46a-1dde-4944-b001-06cda4bedcd6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e3cc05-66dfb8cc6c-8xgvh" Nov 03 02:46:53.631391 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:53.631344 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e3cc05-6468f7548f-rl2hr"] Nov 03 02:46:53.658619 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:53.658561 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8t8t\" (UniqueName: \"kubernetes.io/projected/198ce753-de2f-471c-894f-872099dbef2c-kube-api-access-f8t8t\") pod \"payload-processing-e2e-extmod-a-e3cc05-6468f7548f-rl2hr\" (UID: \"198ce753-de2f-471c-894f-872099dbef2c\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e3cc05-6468f7548f-rl2hr" Nov 03 02:46:53.658919 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:53.658894 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/198ce753-de2f-471c-894f-872099dbef2c-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-e3cc05-6468f7548f-rl2hr\" (UID: \"198ce753-de2f-471c-894f-872099dbef2c\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e3cc05-6468f7548f-rl2hr" Nov 03 02:46:53.748508 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:53.748427 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e3cc05-66dfb8cc6c-8xgvh" Nov 03 02:46:53.760446 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:53.760408 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/198ce753-de2f-471c-894f-872099dbef2c-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-e3cc05-6468f7548f-rl2hr\" (UID: \"198ce753-de2f-471c-894f-872099dbef2c\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e3cc05-6468f7548f-rl2hr" Nov 03 02:46:53.760562 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:53.760484 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8t8t\" (UniqueName: \"kubernetes.io/projected/198ce753-de2f-471c-894f-872099dbef2c-kube-api-access-f8t8t\") pod \"payload-processing-e2e-extmod-a-e3cc05-6468f7548f-rl2hr\" (UID: \"198ce753-de2f-471c-894f-872099dbef2c\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e3cc05-6468f7548f-rl2hr" Nov 03 02:46:53.761206 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:53.761184 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/198ce753-de2f-471c-894f-872099dbef2c-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-e3cc05-6468f7548f-rl2hr\" (UID: \"198ce753-de2f-471c-894f-872099dbef2c\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e3cc05-6468f7548f-rl2hr" Nov 03 02:46:53.822983 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:53.822762 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8t8t\" (UniqueName: \"kubernetes.io/projected/198ce753-de2f-471c-894f-872099dbef2c-kube-api-access-f8t8t\") pod \"payload-processing-e2e-extmod-a-e3cc05-6468f7548f-rl2hr\" (UID: \"198ce753-de2f-471c-894f-872099dbef2c\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e3cc05-6468f7548f-rl2hr" Nov 03 02:46:53.894295 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:53.894264 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-e3cc05-6468f7548f-rl2hr" Nov 03 02:46:53.917381 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:46:53.917334 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:46:54.118728 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:54.118697 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e3cc05-66dfb8cc6c-8xgvh"] Nov 03 02:46:54.123879 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:46:54.123852 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1773b46a_1dde_4944_b001_06cda4bedcd6.slice/crio-a3374ddbb15bc88717a66584c0b2b4f4b67f30aa5c60d473fbfe03ee85aadf42 WatchSource:0}: Error finding container a3374ddbb15bc88717a66584c0b2b4f4b67f30aa5c60d473fbfe03ee85aadf42: Status 404 returned error can't find the container with id a3374ddbb15bc88717a66584c0b2b4f4b67f30aa5c60d473fbfe03ee85aadf42 Nov 03 02:46:54.270425 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:54.270399 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e3cc05-6468f7548f-rl2hr"] Nov 03 02:46:54.274773 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:46:54.274736 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod198ce753_de2f_471c_894f_872099dbef2c.slice/crio-a0abe2771b7aaf48c7d77690f6f24f2255a8cec7eae5be80a152363577bf7439 WatchSource:0}: Error finding container a0abe2771b7aaf48c7d77690f6f24f2255a8cec7eae5be80a152363577bf7439: Status 404 returned error can't find the container with id a0abe2771b7aaf48c7d77690f6f24f2255a8cec7eae5be80a152363577bf7439 Nov 03 02:46:54.882005 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:54.881961 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e3cc05-66dfb8cc6c-8xgvh" event={"ID":"1773b46a-1dde-4944-b001-06cda4bedcd6","Type":"ContainerStarted","Data":"e9cce2470453155af47146e17d4275fd9bdb8007f9f08d673eda0097d4f38597"} Nov 03 02:46:54.882475 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:54.882024 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e3cc05-66dfb8cc6c-8xgvh" event={"ID":"1773b46a-1dde-4944-b001-06cda4bedcd6","Type":"ContainerStarted","Data":"a3374ddbb15bc88717a66584c0b2b4f4b67f30aa5c60d473fbfe03ee85aadf42"} Nov 03 02:46:54.882475 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:54.882045 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e3cc05-66dfb8cc6c-8xgvh" Nov 03 02:46:54.883645 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:54.883616 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-e3cc05-6468f7548f-rl2hr" event={"ID":"198ce753-de2f-471c-894f-872099dbef2c","Type":"ContainerStarted","Data":"c71b184be1d2af5d16fc1b1e2f50ddd0a22a9065dd23adaec7cb750ec7ec27aa"} Nov 03 02:46:54.883746 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:54.883662 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-e3cc05-6468f7548f-rl2hr" event={"ID":"198ce753-de2f-471c-894f-872099dbef2c","Type":"ContainerStarted","Data":"a0abe2771b7aaf48c7d77690f6f24f2255a8cec7eae5be80a152363577bf7439"} Nov 03 02:46:54.883746 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:54.883702 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-e3cc05-6468f7548f-rl2hr" Nov 03 02:46:54.990885 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:54.990832 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e3cc05-66dfb8cc6c-8xgvh" podStartSLOduration=1.990819107 podStartE2EDuration="1.990819107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:46:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:46:54.988218556 +0000 UTC m=+3292.247416711" watchObservedRunningTime="2026-11-03 02:46:54.990819107 +0000 UTC m=+3292.250017259" Nov 03 02:46:55.096575 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:46:55.096523 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-e3cc05-6468f7548f-rl2hr" podStartSLOduration=2.096509637 podStartE2EDuration="2.096509637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:46:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:46:55.093824094 +0000 UTC m=+3292.353022247" watchObservedRunningTime="2026-11-03 02:46:55.096509637 +0000 UTC m=+3292.355707790" Nov 03 02:47:05.892350 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:47:05.892320 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-e3cc05-6468f7548f-rl2hr" Nov 03 02:47:05.892781 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:47:05.892423 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e3cc05-66dfb8cc6c-8xgvh" Nov 03 02:47:17.052008 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:47:17.051976 5605 generic.go:356] "Generic (PLEG): container finished" podID="3bf95767-b937-4d51-86ff-a41db2ffc3a4" containerID="158e577028ff4dce01c75829bfb06516735f50c8ef5e1ba438d0c6020dabe8b2" exitCode=0 Nov 03 02:47:17.052538 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:47:17.052050 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx" event={"ID":"3bf95767-b937-4d51-86ff-a41db2ffc3a4","Type":"ContainerDied","Data":"158e577028ff4dce01c75829bfb06516735f50c8ef5e1ba438d0c6020dabe8b2"} Nov 03 02:47:17.052538 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:47:17.052506 5605 scope.go:122] "RemoveContainer" containerID="158e577028ff4dce01c75829bfb06516735f50c8ef5e1ba438d0c6020dabe8b2" Nov 03 02:47:17.054062 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:47:17.054029 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:47:17.054130 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:47:17.054107 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:47:17.054175 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:47:17.054141 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:47:17.075906 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:47:17.075878 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx" Nov 03 02:47:18.085156 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:47:18.085122 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx" event={"ID":"3bf95767-b937-4d51-86ff-a41db2ffc3a4","Type":"ContainerStarted","Data":"e8ac827fbcdc946dd425af6281b7ece4a97995efe070da2be9906073783b32da"} Nov 03 02:47:19.069779 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:47:19.069739 5605 patch_prober.go:29] interesting pod/e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Nov 03 02:47:19.079571 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:47:19.070054 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx" podUID="3bf95767-b937-4d51-86ff-a41db2ffc3a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Nov 03 02:47:19.079571 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:47:19.074420 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx" Nov 03 02:47:20.070838 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:47:20.070794 5605 patch_prober.go:29] interesting pod/e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Nov 03 02:47:20.071496 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:47:20.071459 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx" podUID="3bf95767-b937-4d51-86ff-a41db2ffc3a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Nov 03 02:47:21.063818 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:47:21.063775 5605 patch_prober.go:29] interesting pod/e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Nov 03 02:47:21.064232 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:47:21.064196 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx" podUID="3bf95767-b937-4d51-86ff-a41db2ffc3a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Nov 03 02:47:22.065835 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:47:22.065791 5605 patch_prober.go:29] interesting pod/e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Nov 03 02:47:22.066497 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:47:22.066454 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx" podUID="3bf95767-b937-4d51-86ff-a41db2ffc3a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Nov 03 02:47:23.068380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:47:23.067781 5605 patch_prober.go:29] interesting pod/e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Nov 03 02:47:23.068380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:47:23.067847 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx" podUID="3bf95767-b937-4d51-86ff-a41db2ffc3a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Nov 03 02:47:23.528047 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:47:23.527961 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:47:23.916339 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:47:23.916296 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:47:24.066723 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:47:24.066693 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx" Nov 03 02:47:24.067183 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:47:24.067168 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx" Nov 03 02:47:24.067860 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:47:24.067838 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx" Nov 03 02:47:30.516826 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:47:30.516787 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-038c21/explicit-model-038c21-kserve-b65657597-c4mmt"] Nov 03 02:47:30.522156 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:47:30.522134 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-038c21/explicit-model-038c21-kserve-b65657597-c4mmt" Nov 03 02:47:30.574100 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:47:30.574068 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-038c21/explicit-model-038c21-kserve-b65657597-c4mmt"] Nov 03 02:47:30.616838 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:47:30.616800 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-038c21\"/\"openshift-service-ca.crt\"" Nov 03 02:47:30.616838 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:47:30.616808 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-038c21\"/\"kube-root-ca.crt\"" Nov 03 02:47:30.617082 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:47:30.616904 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-038c21\"/\"explicit-model-038c21-kserve-self-signed-certs\"" Nov 03 02:47:30.618107 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:47:30.618086 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-038c21\"/\"default-dockercfg-hn6sm\"" Nov 03 02:47:30.618505 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:47:30.618482 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpwzk\" (UniqueName: \"kubernetes.io/projected/1f935c46-6569-4583-a6df-d45c36c1aa62-kube-api-access-vpwzk\") pod \"explicit-model-038c21-kserve-b65657597-c4mmt\" (UID: \"1f935c46-6569-4583-a6df-d45c36c1aa62\") " pod="ai-tenant-e2e-explicit-038c21/explicit-model-038c21-kserve-b65657597-c4mmt" Nov 03 02:47:30.618563 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:47:30.618539 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f935c46-6569-4583-a6df-d45c36c1aa62-dshm\") pod \"explicit-model-038c21-kserve-b65657597-c4mmt\" (UID: \"1f935c46-6569-4583-a6df-d45c36c1aa62\") " pod="ai-tenant-e2e-explicit-038c21/explicit-model-038c21-kserve-b65657597-c4mmt" Nov 03 02:47:30.618654 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:47:30.618633 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f935c46-6569-4583-a6df-d45c36c1aa62-home\") pod \"explicit-model-038c21-kserve-b65657597-c4mmt\" (UID: \"1f935c46-6569-4583-a6df-d45c36c1aa62\") " pod="ai-tenant-e2e-explicit-038c21/explicit-model-038c21-kserve-b65657597-c4mmt" Nov 03 02:47:30.618715 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:47:30.618699 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f935c46-6569-4583-a6df-d45c36c1aa62-tls-certs\") pod \"explicit-model-038c21-kserve-b65657597-c4mmt\" (UID: \"1f935c46-6569-4583-a6df-d45c36c1aa62\") " pod="ai-tenant-e2e-explicit-038c21/explicit-model-038c21-kserve-b65657597-c4mmt" Nov 03 02:47:30.618824 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:47:30.618805 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f935c46-6569-4583-a6df-d45c36c1aa62-tmp-dir\") pod \"explicit-model-038c21-kserve-b65657597-c4mmt\" (UID: \"1f935c46-6569-4583-a6df-d45c36c1aa62\") " pod="ai-tenant-e2e-explicit-038c21/explicit-model-038c21-kserve-b65657597-c4mmt" Nov 03 02:47:30.618887 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:47:30.618875 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f935c46-6569-4583-a6df-d45c36c1aa62-model-cache\") pod \"explicit-model-038c21-kserve-b65657597-c4mmt\" (UID: \"1f935c46-6569-4583-a6df-d45c36c1aa62\") " pod="ai-tenant-e2e-explicit-038c21/explicit-model-038c21-kserve-b65657597-c4mmt" Nov 03 02:47:30.720175 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:47:30.720139 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpwzk\" (UniqueName: \"kubernetes.io/projected/1f935c46-6569-4583-a6df-d45c36c1aa62-kube-api-access-vpwzk\") pod \"explicit-model-038c21-kserve-b65657597-c4mmt\" (UID: \"1f935c46-6569-4583-a6df-d45c36c1aa62\") " pod="ai-tenant-e2e-explicit-038c21/explicit-model-038c21-kserve-b65657597-c4mmt" Nov 03 02:47:30.720349 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:47:30.720195 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f935c46-6569-4583-a6df-d45c36c1aa62-dshm\") pod \"explicit-model-038c21-kserve-b65657597-c4mmt\" (UID: \"1f935c46-6569-4583-a6df-d45c36c1aa62\") " pod="ai-tenant-e2e-explicit-038c21/explicit-model-038c21-kserve-b65657597-c4mmt" Nov 03 02:47:30.720349 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:47:30.720225 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f935c46-6569-4583-a6df-d45c36c1aa62-home\") pod \"explicit-model-038c21-kserve-b65657597-c4mmt\" (UID: \"1f935c46-6569-4583-a6df-d45c36c1aa62\") " pod="ai-tenant-e2e-explicit-038c21/explicit-model-038c21-kserve-b65657597-c4mmt" Nov 03 02:47:30.720349 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:47:30.720262 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f935c46-6569-4583-a6df-d45c36c1aa62-tls-certs\") pod \"explicit-model-038c21-kserve-b65657597-c4mmt\" (UID: \"1f935c46-6569-4583-a6df-d45c36c1aa62\") " pod="ai-tenant-e2e-explicit-038c21/explicit-model-038c21-kserve-b65657597-c4mmt" Nov 03 02:47:30.720579 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:47:30.720420 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f935c46-6569-4583-a6df-d45c36c1aa62-tmp-dir\") pod \"explicit-model-038c21-kserve-b65657597-c4mmt\" (UID: \"1f935c46-6569-4583-a6df-d45c36c1aa62\") " pod="ai-tenant-e2e-explicit-038c21/explicit-model-038c21-kserve-b65657597-c4mmt" Nov 03 02:47:30.720579 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:47:30.720493 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f935c46-6569-4583-a6df-d45c36c1aa62-model-cache\") pod \"explicit-model-038c21-kserve-b65657597-c4mmt\" (UID: \"1f935c46-6569-4583-a6df-d45c36c1aa62\") " pod="ai-tenant-e2e-explicit-038c21/explicit-model-038c21-kserve-b65657597-c4mmt" Nov 03 02:47:30.720748 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:47:30.720722 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f935c46-6569-4583-a6df-d45c36c1aa62-home\") pod \"explicit-model-038c21-kserve-b65657597-c4mmt\" (UID: \"1f935c46-6569-4583-a6df-d45c36c1aa62\") " pod="ai-tenant-e2e-explicit-038c21/explicit-model-038c21-kserve-b65657597-c4mmt" Nov 03 02:47:30.720865 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:47:30.720841 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f935c46-6569-4583-a6df-d45c36c1aa62-tmp-dir\") pod \"explicit-model-038c21-kserve-b65657597-c4mmt\" (UID: \"1f935c46-6569-4583-a6df-d45c36c1aa62\") " pod="ai-tenant-e2e-explicit-038c21/explicit-model-038c21-kserve-b65657597-c4mmt" Nov 03 02:47:30.720906 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:47:30.720891 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f935c46-6569-4583-a6df-d45c36c1aa62-model-cache\") pod \"explicit-model-038c21-kserve-b65657597-c4mmt\" (UID: \"1f935c46-6569-4583-a6df-d45c36c1aa62\") " pod="ai-tenant-e2e-explicit-038c21/explicit-model-038c21-kserve-b65657597-c4mmt" Nov 03 02:47:30.722674 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:47:30.722653 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f935c46-6569-4583-a6df-d45c36c1aa62-dshm\") pod \"explicit-model-038c21-kserve-b65657597-c4mmt\" (UID: \"1f935c46-6569-4583-a6df-d45c36c1aa62\") " pod="ai-tenant-e2e-explicit-038c21/explicit-model-038c21-kserve-b65657597-c4mmt" Nov 03 02:47:30.722948 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:47:30.722931 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f935c46-6569-4583-a6df-d45c36c1aa62-tls-certs\") pod \"explicit-model-038c21-kserve-b65657597-c4mmt\" (UID: \"1f935c46-6569-4583-a6df-d45c36c1aa62\") " pod="ai-tenant-e2e-explicit-038c21/explicit-model-038c21-kserve-b65657597-c4mmt" Nov 03 02:47:30.784307 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:47:30.784222 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpwzk\" (UniqueName: \"kubernetes.io/projected/1f935c46-6569-4583-a6df-d45c36c1aa62-kube-api-access-vpwzk\") pod \"explicit-model-038c21-kserve-b65657597-c4mmt\" (UID: \"1f935c46-6569-4583-a6df-d45c36c1aa62\") " pod="ai-tenant-e2e-explicit-038c21/explicit-model-038c21-kserve-b65657597-c4mmt" Nov 03 02:47:30.834328 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:47:30.834295 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-038c21/explicit-model-038c21-kserve-b65657597-c4mmt" Nov 03 02:47:31.211524 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:47:31.211491 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-038c21/explicit-model-038c21-kserve-b65657597-c4mmt"] Nov 03 02:47:31.216500 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:47:31.216471 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1f935c46_6569_4583_a6df_d45c36c1aa62.slice/crio-981d66c6244c876c96d46e6efc7020fb4477e680faffa47a9fb7ef5d5b6680c8 WatchSource:0}: Error finding container 981d66c6244c876c96d46e6efc7020fb4477e680faffa47a9fb7ef5d5b6680c8: Status 404 returned error can't find the container with id 981d66c6244c876c96d46e6efc7020fb4477e680faffa47a9fb7ef5d5b6680c8 Nov 03 02:47:32.179568 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:47:32.179526 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-038c21/explicit-model-038c21-kserve-b65657597-c4mmt" event={"ID":"1f935c46-6569-4583-a6df-d45c36c1aa62","Type":"ContainerStarted","Data":"bce673d2b17e8fe7a6c3d8ea37b1994a51f08413ba2492b8156262d69d2f85cc"} Nov 03 02:47:32.179943 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:47:32.179582 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-038c21/explicit-model-038c21-kserve-b65657597-c4mmt" event={"ID":"1f935c46-6569-4583-a6df-d45c36c1aa62","Type":"ContainerStarted","Data":"981d66c6244c876c96d46e6efc7020fb4477e680faffa47a9fb7ef5d5b6680c8"} Nov 03 02:47:32.287647 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:47:32.287600 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-038c21/explicit-model-038c21-kserve-b65657597-c4mmt" podStartSLOduration=2.287587502 podStartE2EDuration="2.287587502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:47:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:47:32.286511622 +0000 UTC m=+3329.545709772" watchObservedRunningTime="2026-11-03 02:47:32.287587502 +0000 UTC m=+3329.546785653" Nov 03 02:47:32.834917 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:47:32.834884 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-038c21/explicit-model-038c21-kserve-b65657597-c4mmt" Nov 03 02:47:40.835464 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:47:40.835384 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-038c21/explicit-model-038c21-kserve-b65657597-c4mmt" Nov 03 02:47:40.851520 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:47:40.851488 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-038c21/explicit-model-038c21-kserve-b65657597-c4mmt" Nov 03 02:47:41.247412 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:47:41.247306 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-038c21/explicit-model-038c21-kserve-b65657597-c4mmt" Nov 03 02:47:53.917290 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:47:53.917257 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:48:12.984016 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:12.983978 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:48:16.422608 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:16.422564 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-038c21/explicit-model-038c21-kserve-b65657597-c4mmt"] Nov 03 02:48:19.043521 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:19.043486 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1249ddb0-tb64z"] Nov 03 02:48:19.118377 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:19.118326 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1249ddb0-tb64z"] Nov 03 02:48:20.856563 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:20.856514 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="806470e7-bfd4-48c0-aecd-be72ce7f78f9" path="/var/lib/kubelet/pods/806470e7-bfd4-48c0-aecd-be72ce7f78f9/volumes" Nov 03 02:48:22.664586 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:22.664541 5605 patch_prober.go:29] interesting pod/e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:48:22.665038 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:22.664598 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp" podUID="d98a3d42-36f0-48d4-b6dc-7dd978b953ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:48:23.551713 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:23.551678 5605 generic.go:356] "Generic (PLEG): container finished" podID="955ee3cb-72bd-4dab-91ce-faffcbbe130d" containerID="7584787c15af69972b9bda79ce98e2f7221feeb167c253d85c111ee287ddfcd9" exitCode=0 Nov 03 02:48:23.553854 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:23.551750 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2" event={"ID":"955ee3cb-72bd-4dab-91ce-faffcbbe130d","Type":"ContainerDied","Data":"7584787c15af69972b9bda79ce98e2f7221feeb167c253d85c111ee287ddfcd9"} Nov 03 02:48:23.553854 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:23.552158 5605 scope.go:122] "RemoveContainer" containerID="7584787c15af69972b9bda79ce98e2f7221feeb167c253d85c111ee287ddfcd9" Nov 03 02:48:23.553854 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:23.553701 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:48:23.553854 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:23.553769 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:48:23.553854 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:23.553814 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:48:23.914246 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:48:23.914154 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:48:24.584546 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:24.584507 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2" event={"ID":"955ee3cb-72bd-4dab-91ce-faffcbbe130d","Type":"ContainerStarted","Data":"590738e66fb753edb2f7efa9b29a281759b31c9409800d0cec384d6e43f077cf"} Nov 03 02:48:24.586253 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:24.586227 5605 generic.go:356] "Generic (PLEG): container finished" podID="d98a3d42-36f0-48d4-b6dc-7dd978b953ae" containerID="be592416eb06a16a33616e403d22f5e788f0a6eb30c86edc20fe36fddb62bd1a" exitCode=0 Nov 03 02:48:24.586409 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:24.586300 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp" event={"ID":"d98a3d42-36f0-48d4-b6dc-7dd978b953ae","Type":"ContainerDied","Data":"be592416eb06a16a33616e403d22f5e788f0a6eb30c86edc20fe36fddb62bd1a"} Nov 03 02:48:24.586668 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:24.586653 5605 scope.go:122] "RemoveContainer" containerID="be592416eb06a16a33616e403d22f5e788f0a6eb30c86edc20fe36fddb62bd1a" Nov 03 02:48:24.588166 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:24.588136 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:48:24.588243 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:24.588202 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:48:24.588243 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:24.588234 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:48:24.950432 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:24.950029 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2" Nov 03 02:48:24.950432 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:24.950155 5605 patch_prober.go:29] interesting pod/e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body= Nov 03 02:48:24.950432 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:24.950202 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2" podUID="955ee3cb-72bd-4dab-91ce-faffcbbe130d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" Nov 03 02:48:25.609574 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:25.609522 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp" event={"ID":"d98a3d42-36f0-48d4-b6dc-7dd978b953ae","Type":"ContainerStarted","Data":"ebfba3cd615cd3f06257fabaebca3ddcc1cb8777ebddf77bfc71c43b456cb701"} Nov 03 02:48:25.950668 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:25.950569 5605 patch_prober.go:29] interesting pod/e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body= Nov 03 02:48:25.950668 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:25.950636 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2" podUID="955ee3cb-72bd-4dab-91ce-faffcbbe130d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" Nov 03 02:48:26.585378 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:26.584761 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp" Nov 03 02:48:26.585378 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:26.584895 5605 patch_prober.go:29] interesting pod/e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Nov 03 02:48:26.585378 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:26.584962 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp" podUID="d98a3d42-36f0-48d4-b6dc-7dd978b953ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Nov 03 02:48:26.950824 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:26.950739 5605 patch_prober.go:29] interesting pod/e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body= Nov 03 02:48:26.950824 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:26.950793 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2" podUID="955ee3cb-72bd-4dab-91ce-faffcbbe130d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" Nov 03 02:48:27.223036 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:27.222958 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-c8ee6b/autores-model-c8ee6b-kserve-6f47664dc7-v9c4p"] Nov 03 02:48:27.227311 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:27.227287 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-c8ee6b/autores-model-c8ee6b-kserve-6f47664dc7-v9c4p" Nov 03 02:48:27.347433 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:27.347406 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-c8ee6b\"/\"default-dockercfg-lbv2b\"" Nov 03 02:48:27.347596 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:27.347500 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-c8ee6b\"/\"autores-model-c8ee6b-kserve-self-signed-certs\"" Nov 03 02:48:27.347596 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:27.347565 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-c8ee6b\"/\"kube-root-ca.crt\"" Nov 03 02:48:27.347711 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:27.347697 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-c8ee6b\"/\"openshift-service-ca.crt\"" Nov 03 02:48:27.350813 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:27.350794 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-c8ee6b/autores-model-c8ee6b-kserve-6f47664dc7-v9c4p"] Nov 03 02:48:27.427034 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:27.426988 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db0c38c1-8105-49de-9561-319a632581bd-home\") pod \"autores-model-c8ee6b-kserve-6f47664dc7-v9c4p\" (UID: \"db0c38c1-8105-49de-9561-319a632581bd\") " pod="ai-tenant-e2e-autores-c8ee6b/autores-model-c8ee6b-kserve-6f47664dc7-v9c4p" Nov 03 02:48:27.427034 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:27.427034 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db0c38c1-8105-49de-9561-319a632581bd-dshm\") pod \"autores-model-c8ee6b-kserve-6f47664dc7-v9c4p\" (UID: \"db0c38c1-8105-49de-9561-319a632581bd\") " pod="ai-tenant-e2e-autores-c8ee6b/autores-model-c8ee6b-kserve-6f47664dc7-v9c4p" Nov 03 02:48:27.427323 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:27.427058 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9k6j\" (UniqueName: \"kubernetes.io/projected/db0c38c1-8105-49de-9561-319a632581bd-kube-api-access-c9k6j\") pod \"autores-model-c8ee6b-kserve-6f47664dc7-v9c4p\" (UID: \"db0c38c1-8105-49de-9561-319a632581bd\") " pod="ai-tenant-e2e-autores-c8ee6b/autores-model-c8ee6b-kserve-6f47664dc7-v9c4p" Nov 03 02:48:27.427323 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:27.427112 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db0c38c1-8105-49de-9561-319a632581bd-model-cache\") pod \"autores-model-c8ee6b-kserve-6f47664dc7-v9c4p\" (UID: \"db0c38c1-8105-49de-9561-319a632581bd\") " pod="ai-tenant-e2e-autores-c8ee6b/autores-model-c8ee6b-kserve-6f47664dc7-v9c4p" Nov 03 02:48:27.427323 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:27.427165 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db0c38c1-8105-49de-9561-319a632581bd-tls-certs\") pod \"autores-model-c8ee6b-kserve-6f47664dc7-v9c4p\" (UID: \"db0c38c1-8105-49de-9561-319a632581bd\") " pod="ai-tenant-e2e-autores-c8ee6b/autores-model-c8ee6b-kserve-6f47664dc7-v9c4p" Nov 03 02:48:27.427323 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:27.427240 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db0c38c1-8105-49de-9561-319a632581bd-tmp-dir\") pod \"autores-model-c8ee6b-kserve-6f47664dc7-v9c4p\" (UID: \"db0c38c1-8105-49de-9561-319a632581bd\") " pod="ai-tenant-e2e-autores-c8ee6b/autores-model-c8ee6b-kserve-6f47664dc7-v9c4p" Nov 03 02:48:27.528231 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:27.528140 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db0c38c1-8105-49de-9561-319a632581bd-tls-certs\") pod \"autores-model-c8ee6b-kserve-6f47664dc7-v9c4p\" (UID: \"db0c38c1-8105-49de-9561-319a632581bd\") " pod="ai-tenant-e2e-autores-c8ee6b/autores-model-c8ee6b-kserve-6f47664dc7-v9c4p" Nov 03 02:48:27.528420 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:27.528276 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db0c38c1-8105-49de-9561-319a632581bd-tmp-dir\") pod \"autores-model-c8ee6b-kserve-6f47664dc7-v9c4p\" (UID: \"db0c38c1-8105-49de-9561-319a632581bd\") " pod="ai-tenant-e2e-autores-c8ee6b/autores-model-c8ee6b-kserve-6f47664dc7-v9c4p" Nov 03 02:48:27.528420 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:27.528317 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db0c38c1-8105-49de-9561-319a632581bd-home\") pod \"autores-model-c8ee6b-kserve-6f47664dc7-v9c4p\" (UID: \"db0c38c1-8105-49de-9561-319a632581bd\") " pod="ai-tenant-e2e-autores-c8ee6b/autores-model-c8ee6b-kserve-6f47664dc7-v9c4p" Nov 03 02:48:27.528842 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:27.528817 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db0c38c1-8105-49de-9561-319a632581bd-tmp-dir\") pod \"autores-model-c8ee6b-kserve-6f47664dc7-v9c4p\" (UID: \"db0c38c1-8105-49de-9561-319a632581bd\") " pod="ai-tenant-e2e-autores-c8ee6b/autores-model-c8ee6b-kserve-6f47664dc7-v9c4p" Nov 03 02:48:27.528884 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:27.528383 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db0c38c1-8105-49de-9561-319a632581bd-dshm\") pod \"autores-model-c8ee6b-kserve-6f47664dc7-v9c4p\" (UID: \"db0c38c1-8105-49de-9561-319a632581bd\") " pod="ai-tenant-e2e-autores-c8ee6b/autores-model-c8ee6b-kserve-6f47664dc7-v9c4p" Nov 03 02:48:27.528920 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:27.528874 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db0c38c1-8105-49de-9561-319a632581bd-home\") pod \"autores-model-c8ee6b-kserve-6f47664dc7-v9c4p\" (UID: \"db0c38c1-8105-49de-9561-319a632581bd\") " pod="ai-tenant-e2e-autores-c8ee6b/autores-model-c8ee6b-kserve-6f47664dc7-v9c4p" Nov 03 02:48:27.529376 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:27.529327 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9k6j\" (UniqueName: \"kubernetes.io/projected/db0c38c1-8105-49de-9561-319a632581bd-kube-api-access-c9k6j\") pod \"autores-model-c8ee6b-kserve-6f47664dc7-v9c4p\" (UID: \"db0c38c1-8105-49de-9561-319a632581bd\") " pod="ai-tenant-e2e-autores-c8ee6b/autores-model-c8ee6b-kserve-6f47664dc7-v9c4p" Nov 03 02:48:27.529527 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:27.529435 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db0c38c1-8105-49de-9561-319a632581bd-model-cache\") pod \"autores-model-c8ee6b-kserve-6f47664dc7-v9c4p\" (UID: \"db0c38c1-8105-49de-9561-319a632581bd\") " pod="ai-tenant-e2e-autores-c8ee6b/autores-model-c8ee6b-kserve-6f47664dc7-v9c4p" Nov 03 02:48:27.529784 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:27.529766 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db0c38c1-8105-49de-9561-319a632581bd-model-cache\") pod \"autores-model-c8ee6b-kserve-6f47664dc7-v9c4p\" (UID: \"db0c38c1-8105-49de-9561-319a632581bd\") " pod="ai-tenant-e2e-autores-c8ee6b/autores-model-c8ee6b-kserve-6f47664dc7-v9c4p" Nov 03 02:48:27.531622 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:27.531598 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db0c38c1-8105-49de-9561-319a632581bd-tls-certs\") pod \"autores-model-c8ee6b-kserve-6f47664dc7-v9c4p\" (UID: \"db0c38c1-8105-49de-9561-319a632581bd\") " pod="ai-tenant-e2e-autores-c8ee6b/autores-model-c8ee6b-kserve-6f47664dc7-v9c4p" Nov 03 02:48:27.531812 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:27.531779 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db0c38c1-8105-49de-9561-319a632581bd-dshm\") pod \"autores-model-c8ee6b-kserve-6f47664dc7-v9c4p\" (UID: \"db0c38c1-8105-49de-9561-319a632581bd\") " pod="ai-tenant-e2e-autores-c8ee6b/autores-model-c8ee6b-kserve-6f47664dc7-v9c4p" Nov 03 02:48:27.585792 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:27.585753 5605 patch_prober.go:29] interesting pod/e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Nov 03 02:48:27.586148 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:27.586114 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp" podUID="d98a3d42-36f0-48d4-b6dc-7dd978b953ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Nov 03 02:48:27.607228 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:27.607150 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9k6j\" (UniqueName: \"kubernetes.io/projected/db0c38c1-8105-49de-9561-319a632581bd-kube-api-access-c9k6j\") pod \"autores-model-c8ee6b-kserve-6f47664dc7-v9c4p\" (UID: \"db0c38c1-8105-49de-9561-319a632581bd\") " pod="ai-tenant-e2e-autores-c8ee6b/autores-model-c8ee6b-kserve-6f47664dc7-v9c4p" Nov 03 02:48:27.839618 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:27.839576 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-c8ee6b/autores-model-c8ee6b-kserve-6f47664dc7-v9c4p" Nov 03 02:48:27.951579 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:27.951545 5605 patch_prober.go:29] interesting pod/e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body= Nov 03 02:48:27.952036 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:27.951595 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2" podUID="955ee3cb-72bd-4dab-91ce-faffcbbe130d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" Nov 03 02:48:28.372628 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:28.372319 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-c8ee6b/autores-model-c8ee6b-kserve-6f47664dc7-v9c4p"] Nov 03 02:48:28.585382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:28.584777 5605 patch_prober.go:29] interesting pod/e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Nov 03 02:48:28.585382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:28.584839 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp" podUID="d98a3d42-36f0-48d4-b6dc-7dd978b953ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Nov 03 02:48:28.639301 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:28.639214 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-c8ee6b/autores-model-c8ee6b-kserve-6f47664dc7-v9c4p" event={"ID":"db0c38c1-8105-49de-9561-319a632581bd","Type":"ContainerStarted","Data":"3247d35a542207ccefb3e7151dbf89916bcbc2e3eece73b56d644e9d4ccf349b"} Nov 03 02:48:28.639301 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:28.639270 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-c8ee6b/autores-model-c8ee6b-kserve-6f47664dc7-v9c4p" event={"ID":"db0c38c1-8105-49de-9561-319a632581bd","Type":"ContainerStarted","Data":"7ff520b6c38adb9e5204189861c5614aef1abd6ab1d70872e7451963a31038a2"} Nov 03 02:48:28.753176 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:28.753122 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-autores-c8ee6b/autores-model-c8ee6b-kserve-6f47664dc7-v9c4p" podStartSLOduration=1.753108579 podStartE2EDuration="1.753108579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:48:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:48:28.750326508 +0000 UTC m=+3386.009524662" watchObservedRunningTime="2026-11-03 02:48:28.753108579 +0000 UTC m=+3386.012306732" Nov 03 02:48:28.840982 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:28.840931 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-c8ee6b/autores-model-c8ee6b-kserve-6f47664dc7-v9c4p" Nov 03 02:48:28.950948 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:28.950859 5605 patch_prober.go:29] interesting pod/e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body= Nov 03 02:48:28.950948 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:28.950911 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2" podUID="955ee3cb-72bd-4dab-91ce-faffcbbe130d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" Nov 03 02:48:29.584782 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:29.584745 5605 patch_prober.go:29] interesting pod/e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Nov 03 02:48:29.585443 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:29.585400 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp" podUID="d98a3d42-36f0-48d4-b6dc-7dd978b953ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Nov 03 02:48:29.964437 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:29.963670 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2" Nov 03 02:48:29.964952 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:29.964927 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2" Nov 03 02:48:29.970576 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:29.970554 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2" Nov 03 02:48:30.585182 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:30.585148 5605 patch_prober.go:29] interesting pod/e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Nov 03 02:48:30.585637 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:30.585199 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp" podUID="d98a3d42-36f0-48d4-b6dc-7dd978b953ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Nov 03 02:48:31.445277 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:31.445234 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-038c21/explicit-model-038c21-kserve-b65657597-c4mmt" podUID="1f935c46-6569-4583-a6df-d45c36c1aa62" containerName="main" containerID="cri-o://bce673d2b17e8fe7a6c3d8ea37b1994a51f08413ba2492b8156262d69d2f85cc" gracePeriod=45 Nov 03 02:48:31.589051 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:31.589021 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp" Nov 03 02:48:31.589544 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:31.589326 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp" Nov 03 02:48:31.590082 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:31.590057 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp" Nov 03 02:48:31.661119 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:31.661087 5605 generic.go:356] "Generic (PLEG): container finished" podID="1f935c46-6569-4583-a6df-d45c36c1aa62" containerID="bce673d2b17e8fe7a6c3d8ea37b1994a51f08413ba2492b8156262d69d2f85cc" exitCode=0 Nov 03 02:48:31.661289 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:31.661158 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-038c21/explicit-model-038c21-kserve-b65657597-c4mmt" event={"ID":"1f935c46-6569-4583-a6df-d45c36c1aa62","Type":"ContainerDied","Data":"bce673d2b17e8fe7a6c3d8ea37b1994a51f08413ba2492b8156262d69d2f85cc"} Nov 03 02:48:31.711295 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:31.711244 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-038c21/explicit-model-038c21-kserve-b65657597-c4mmt" Nov 03 02:48:31.830721 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:31.830691 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f935c46-6569-4583-a6df-d45c36c1aa62-kube-api-access-vpwzk\" (UniqueName: \"kubernetes.io/projected/1f935c46-6569-4583-a6df-d45c36c1aa62-kube-api-access-vpwzk\") pod \"1f935c46-6569-4583-a6df-d45c36c1aa62\" (UID: \"1f935c46-6569-4583-a6df-d45c36c1aa62\") " Nov 03 02:48:31.830872 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:31.830729 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f935c46-6569-4583-a6df-d45c36c1aa62-tls-certs\" (UniqueName: \"kubernetes.io/secret/1f935c46-6569-4583-a6df-d45c36c1aa62-tls-certs\") pod \"1f935c46-6569-4583-a6df-d45c36c1aa62\" (UID: \"1f935c46-6569-4583-a6df-d45c36c1aa62\") " Nov 03 02:48:31.830872 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:31.830756 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f935c46-6569-4583-a6df-d45c36c1aa62-home\" (UniqueName: \"kubernetes.io/empty-dir/1f935c46-6569-4583-a6df-d45c36c1aa62-home\") pod \"1f935c46-6569-4583-a6df-d45c36c1aa62\" (UID: \"1f935c46-6569-4583-a6df-d45c36c1aa62\") " Nov 03 02:48:31.830971 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:31.830894 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f935c46-6569-4583-a6df-d45c36c1aa62-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f935c46-6569-4583-a6df-d45c36c1aa62-tmp-dir\") pod \"1f935c46-6569-4583-a6df-d45c36c1aa62\" (UID: \"1f935c46-6569-4583-a6df-d45c36c1aa62\") " Nov 03 02:48:31.830971 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:31.830944 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f935c46-6569-4583-a6df-d45c36c1aa62-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f935c46-6569-4583-a6df-d45c36c1aa62-dshm\") pod \"1f935c46-6569-4583-a6df-d45c36c1aa62\" (UID: \"1f935c46-6569-4583-a6df-d45c36c1aa62\") " Nov 03 02:48:31.831061 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:31.831031 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f935c46-6569-4583-a6df-d45c36c1aa62-home" pod "1f935c46-6569-4583-a6df-d45c36c1aa62" (UID: "1f935c46-6569-4583-a6df-d45c36c1aa62"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:48:31.831061 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:31.831050 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f935c46-6569-4583-a6df-d45c36c1aa62-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f935c46-6569-4583-a6df-d45c36c1aa62-model-cache\") pod \"1f935c46-6569-4583-a6df-d45c36c1aa62\" (UID: \"1f935c46-6569-4583-a6df-d45c36c1aa62\") " Nov 03 02:48:31.831181 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:31.831111 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f935c46-6569-4583-a6df-d45c36c1aa62-tmp-dir" pod "1f935c46-6569-4583-a6df-d45c36c1aa62" (UID: "1f935c46-6569-4583-a6df-d45c36c1aa62"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:48:31.831242 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:31.831232 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f935c46-6569-4583-a6df-d45c36c1aa62-model-cache" pod "1f935c46-6569-4583-a6df-d45c36c1aa62" (UID: "1f935c46-6569-4583-a6df-d45c36c1aa62"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:48:31.831542 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:31.831514 5605 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f935c46-6569-4583-a6df-d45c36c1aa62-model-cache\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:48:31.831678 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:31.831545 5605 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f935c46-6569-4583-a6df-d45c36c1aa62-home\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:48:31.831678 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:31.831561 5605 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f935c46-6569-4583-a6df-d45c36c1aa62-tmp-dir\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:48:31.832879 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:31.832852 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f935c46-6569-4583-a6df-d45c36c1aa62-tls-certs" pod "1f935c46-6569-4583-a6df-d45c36c1aa62" (UID: "1f935c46-6569-4583-a6df-d45c36c1aa62"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:48:31.832988 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:31.832902 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f935c46-6569-4583-a6df-d45c36c1aa62-dshm" pod "1f935c46-6569-4583-a6df-d45c36c1aa62" (UID: "1f935c46-6569-4583-a6df-d45c36c1aa62"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:48:31.832988 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:31.832914 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f935c46-6569-4583-a6df-d45c36c1aa62-kube-api-access-vpwzk" pod "1f935c46-6569-4583-a6df-d45c36c1aa62" (UID: "1f935c46-6569-4583-a6df-d45c36c1aa62"). InnerVolumeSpecName "kube-api-access-vpwzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:48:31.933310 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:31.933279 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpwzk\" (UniqueName: \"kubernetes.io/projected/1f935c46-6569-4583-a6df-d45c36c1aa62-kube-api-access-vpwzk\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:48:31.933460 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:31.933314 5605 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f935c46-6569-4583-a6df-d45c36c1aa62-tls-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:48:31.933460 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:31.933330 5605 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f935c46-6569-4583-a6df-d45c36c1aa62-dshm\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:48:32.668165 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:32.668136 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-038c21/explicit-model-038c21-kserve-b65657597-c4mmt" Nov 03 02:48:32.668628 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:32.668177 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-038c21/explicit-model-038c21-kserve-b65657597-c4mmt" event={"ID":"1f935c46-6569-4583-a6df-d45c36c1aa62","Type":"ContainerDied","Data":"981d66c6244c876c96d46e6efc7020fb4477e680faffa47a9fb7ef5d5b6680c8"} Nov 03 02:48:32.668628 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:32.668228 5605 scope.go:122] "RemoveContainer" containerID="bce673d2b17e8fe7a6c3d8ea37b1994a51f08413ba2492b8156262d69d2f85cc" Nov 03 02:48:32.821953 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:32.821927 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-038c21/explicit-model-038c21-kserve-b65657597-c4mmt"] Nov 03 02:48:32.868670 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:32.868643 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-038c21/explicit-model-038c21-kserve-b65657597-c4mmt"] Nov 03 02:48:34.850881 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:34.850850 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f935c46-6569-4583-a6df-d45c36c1aa62" path="/var/lib/kubelet/pods/1f935c46-6569-4583-a6df-d45c36c1aa62/volumes" Nov 03 02:48:37.840919 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:37.840880 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-c8ee6b/autores-model-c8ee6b-kserve-6f47664dc7-v9c4p" Nov 03 02:48:37.853078 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:37.853053 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-c8ee6b/autores-model-c8ee6b-kserve-6f47664dc7-v9c4p" Nov 03 02:48:38.737995 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:38.737965 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-c8ee6b/autores-model-c8ee6b-kserve-6f47664dc7-v9c4p" Nov 03 02:48:43.047083 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:43.047040 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94dc3580-5hlmr"] Nov 03 02:48:43.093385 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:43.093335 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94dc3580-5hlmr"] Nov 03 02:48:44.850856 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:44.850823 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="439ed92d-93fa-4b98-83ea-3eac12315fd1" path="/var/lib/kubelet/pods/439ed92d-93fa-4b98-83ea-3eac12315fd1/volumes" Nov 03 02:48:45.773383 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:45.773286 5605 generic.go:356] "Generic (PLEG): container finished" podID="75df2f44-a9e5-47df-b676-6c0882b547e2" containerID="3d41884c4d69f4ffac583df5d18221b6659d52081231a80eb1a485f69c91d1c2" exitCode=0 Nov 03 02:48:45.773383 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:45.773371 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9" event={"ID":"75df2f44-a9e5-47df-b676-6c0882b547e2","Type":"ContainerDied","Data":"3d41884c4d69f4ffac583df5d18221b6659d52081231a80eb1a485f69c91d1c2"} Nov 03 02:48:45.773787 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:45.773773 5605 scope.go:122] "RemoveContainer" containerID="3d41884c4d69f4ffac583df5d18221b6659d52081231a80eb1a485f69c91d1c2" Nov 03 02:48:45.775537 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:45.775507 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:48:45.775631 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:45.775587 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:48:45.775631 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:45.775621 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:48:46.787153 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:46.787114 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9" event={"ID":"75df2f44-a9e5-47df-b676-6c0882b547e2","Type":"ContainerStarted","Data":"2d161114e12ffdc79185a7db8f0d4db5692014db8c44d17036380748a3ec35d1"} Nov 03 02:48:47.018603 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:47.018565 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9" Nov 03 02:48:47.019628 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:47.019603 5605 patch_prober.go:29] interesting pod/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Nov 03 02:48:47.019691 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:47.019657 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9" podUID="75df2f44-a9e5-47df-b676-6c0882b547e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Nov 03 02:48:47.841204 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:47.841171 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9" Nov 03 02:48:48.018487 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:48.018437 5605 patch_prober.go:29] interesting pod/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Nov 03 02:48:48.018661 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:48.018503 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9" podUID="75df2f44-a9e5-47df-b676-6c0882b547e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Nov 03 02:48:49.018262 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:49.018225 5605 patch_prober.go:29] interesting pod/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Nov 03 02:48:49.018765 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:49.018280 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9" podUID="75df2f44-a9e5-47df-b676-6c0882b547e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Nov 03 02:48:50.017875 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:50.017840 5605 patch_prober.go:29] interesting pod/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Nov 03 02:48:50.018045 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:50.017893 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9" podUID="75df2f44-a9e5-47df-b676-6c0882b547e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Nov 03 02:48:51.018436 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:51.018403 5605 patch_prober.go:29] interesting pod/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Nov 03 02:48:51.018847 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:51.018467 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9" podUID="75df2f44-a9e5-47df-b676-6c0882b547e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Nov 03 02:48:52.018299 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:52.018266 5605 patch_prober.go:29] interesting pod/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Nov 03 02:48:52.018491 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:52.018327 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9" podUID="75df2f44-a9e5-47df-b676-6c0882b547e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Nov 03 02:48:52.815537 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:52.815508 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-038c21-c54876867-hxprn"] Nov 03 02:48:52.921660 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:52.921472 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-038c21-5d649bd5f-tcdq5"] Nov 03 02:48:53.022298 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:53.022263 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9" Nov 03 02:48:53.023307 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:53.023281 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9" Nov 03 02:48:53.913082 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:48:53.913047 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:48:54.043193 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:54.043160 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa6a259d-ql28g"] Nov 03 02:48:54.046300 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:54.046278 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa6a259d-ql28g"] Nov 03 02:48:54.855552 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:54.853123 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cf7e386-7d8e-487c-90fb-b367eb765d2a" path="/var/lib/kubelet/pods/1cf7e386-7d8e-487c-90fb-b367eb765d2a/volumes" Nov 03 02:48:57.816737 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:57.816690 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-038c21-c54876867-hxprn" podUID="d63996c8-1493-441a-927c-09b9f7394f12" containerName="payload-processing" containerID="cri-o://a1688a5c0a6b6349d11020da63822bc309e670752119a52058ee5062739a6a47" gracePeriod=25 Nov 03 02:48:57.922985 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:57.922925 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-038c21-5d649bd5f-tcdq5" podUID="9b5077e6-2db7-45f4-85e4-6fca6f27b117" containerName="payload-pre-processing" containerID="cri-o://9cd99fcf0b614a01120461f7025e7a07f81a7a46986b65e94ad3859d42d347d6" gracePeriod=25 Nov 03 02:48:58.087882 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:58.087809 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-038c21-c54876867-hxprn" Nov 03 02:48:58.195145 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:58.195116 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-038c21-5d649bd5f-tcdq5" Nov 03 02:48:58.227567 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:58.227534 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d63996c8-1493-441a-927c-09b9f7394f12-kube-api-access-l48rw\" (UniqueName: \"kubernetes.io/projected/d63996c8-1493-441a-927c-09b9f7394f12-kube-api-access-l48rw\") pod \"d63996c8-1493-441a-927c-09b9f7394f12\" (UID: \"d63996c8-1493-441a-927c-09b9f7394f12\") " Nov 03 02:48:58.227793 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:58.227585 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d63996c8-1493-441a-927c-09b9f7394f12-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d63996c8-1493-441a-927c-09b9f7394f12-plugins-config-volume\") pod \"d63996c8-1493-441a-927c-09b9f7394f12\" (UID: \"d63996c8-1493-441a-927c-09b9f7394f12\") " Nov 03 02:48:58.228078 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:58.228048 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d63996c8-1493-441a-927c-09b9f7394f12-plugins-config-volume" pod "d63996c8-1493-441a-927c-09b9f7394f12" (UID: "d63996c8-1493-441a-927c-09b9f7394f12"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:48:58.228226 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:58.228207 5605 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d63996c8-1493-441a-927c-09b9f7394f12-plugins-config-volume\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:48:58.229784 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:58.229751 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d63996c8-1493-441a-927c-09b9f7394f12-kube-api-access-l48rw" pod "d63996c8-1493-441a-927c-09b9f7394f12" (UID: "d63996c8-1493-441a-927c-09b9f7394f12"). InnerVolumeSpecName "kube-api-access-l48rw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:48:58.328865 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:58.328823 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b5077e6-2db7-45f4-85e4-6fca6f27b117-kube-api-access-r5s6x\" (UniqueName: \"kubernetes.io/projected/9b5077e6-2db7-45f4-85e4-6fca6f27b117-kube-api-access-r5s6x\") pod \"9b5077e6-2db7-45f4-85e4-6fca6f27b117\" (UID: \"9b5077e6-2db7-45f4-85e4-6fca6f27b117\") " Nov 03 02:48:58.329073 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:58.329001 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b5077e6-2db7-45f4-85e4-6fca6f27b117-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b5077e6-2db7-45f4-85e4-6fca6f27b117-plugins-config-volume\") pod \"9b5077e6-2db7-45f4-85e4-6fca6f27b117\" (UID: \"9b5077e6-2db7-45f4-85e4-6fca6f27b117\") " Nov 03 02:48:58.329395 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:58.329348 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b5077e6-2db7-45f4-85e4-6fca6f27b117-plugins-config-volume" pod "9b5077e6-2db7-45f4-85e4-6fca6f27b117" (UID: "9b5077e6-2db7-45f4-85e4-6fca6f27b117"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:48:58.329534 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:58.329505 5605 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b5077e6-2db7-45f4-85e4-6fca6f27b117-plugins-config-volume\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:48:58.329534 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:58.329529 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l48rw\" (UniqueName: \"kubernetes.io/projected/d63996c8-1493-441a-927c-09b9f7394f12-kube-api-access-l48rw\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:48:58.331007 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:58.330985 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b5077e6-2db7-45f4-85e4-6fca6f27b117-kube-api-access-r5s6x" pod "9b5077e6-2db7-45f4-85e4-6fca6f27b117" (UID: "9b5077e6-2db7-45f4-85e4-6fca6f27b117"). InnerVolumeSpecName "kube-api-access-r5s6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:48:58.430630 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:58.430543 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5s6x\" (UniqueName: \"kubernetes.io/projected/9b5077e6-2db7-45f4-85e4-6fca6f27b117-kube-api-access-r5s6x\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:48:58.865189 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:58.865149 5605 generic.go:356] "Generic (PLEG): container finished" podID="9b5077e6-2db7-45f4-85e4-6fca6f27b117" containerID="9cd99fcf0b614a01120461f7025e7a07f81a7a46986b65e94ad3859d42d347d6" exitCode=0 Nov 03 02:48:58.866035 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:58.866000 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-038c21-5d649bd5f-tcdq5" event={"ID":"9b5077e6-2db7-45f4-85e4-6fca6f27b117","Type":"ContainerDied","Data":"9cd99fcf0b614a01120461f7025e7a07f81a7a46986b65e94ad3859d42d347d6"} Nov 03 02:48:58.866471 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:58.866451 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-038c21-5d649bd5f-tcdq5" event={"ID":"9b5077e6-2db7-45f4-85e4-6fca6f27b117","Type":"ContainerDied","Data":"84a272a259f7c3aa57cd33d99dd462ee13bc300e38eb60ce37c900e8bbc99822"} Nov 03 02:48:58.866697 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:58.866681 5605 scope.go:122] "RemoveContainer" containerID="9cd99fcf0b614a01120461f7025e7a07f81a7a46986b65e94ad3859d42d347d6" Nov 03 02:48:58.866907 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:58.866024 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-038c21-5d649bd5f-tcdq5" Nov 03 02:48:58.868831 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:58.868609 5605 generic.go:356] "Generic (PLEG): container finished" podID="d63996c8-1493-441a-927c-09b9f7394f12" containerID="a1688a5c0a6b6349d11020da63822bc309e670752119a52058ee5062739a6a47" exitCode=0 Nov 03 02:48:58.868831 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:58.868686 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-038c21-c54876867-hxprn" event={"ID":"d63996c8-1493-441a-927c-09b9f7394f12","Type":"ContainerDied","Data":"a1688a5c0a6b6349d11020da63822bc309e670752119a52058ee5062739a6a47"} Nov 03 02:48:58.868831 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:58.868750 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-038c21-c54876867-hxprn" event={"ID":"d63996c8-1493-441a-927c-09b9f7394f12","Type":"ContainerDied","Data":"eb1faff8fd540331ec443aab7cd91938009b00ce614486f66382c51b88960641"} Nov 03 02:48:58.869868 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:58.869847 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-038c21-c54876867-hxprn" Nov 03 02:48:58.878744 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:58.878727 5605 scope.go:122] "RemoveContainer" containerID="9cd99fcf0b614a01120461f7025e7a07f81a7a46986b65e94ad3859d42d347d6" Nov 03 02:48:58.878985 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:48:58.878967 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cd99fcf0b614a01120461f7025e7a07f81a7a46986b65e94ad3859d42d347d6\": container with ID starting with 9cd99fcf0b614a01120461f7025e7a07f81a7a46986b65e94ad3859d42d347d6 not found: ID does not exist" containerID="9cd99fcf0b614a01120461f7025e7a07f81a7a46986b65e94ad3859d42d347d6" Nov 03 02:48:58.879063 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:58.878996 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cd99fcf0b614a01120461f7025e7a07f81a7a46986b65e94ad3859d42d347d6"} err="failed to get container status \"9cd99fcf0b614a01120461f7025e7a07f81a7a46986b65e94ad3859d42d347d6\": rpc error: code = NotFound desc = could not find container \"9cd99fcf0b614a01120461f7025e7a07f81a7a46986b65e94ad3859d42d347d6\": container with ID starting with 9cd99fcf0b614a01120461f7025e7a07f81a7a46986b65e94ad3859d42d347d6 not found: ID does not exist" Nov 03 02:48:58.879063 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:58.879018 5605 scope.go:122] "RemoveContainer" containerID="a1688a5c0a6b6349d11020da63822bc309e670752119a52058ee5062739a6a47" Nov 03 02:48:58.887283 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:58.887262 5605 scope.go:122] "RemoveContainer" containerID="a1688a5c0a6b6349d11020da63822bc309e670752119a52058ee5062739a6a47" Nov 03 02:48:58.887540 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:48:58.887521 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1688a5c0a6b6349d11020da63822bc309e670752119a52058ee5062739a6a47\": container with ID starting with a1688a5c0a6b6349d11020da63822bc309e670752119a52058ee5062739a6a47 not found: ID does not exist" containerID="a1688a5c0a6b6349d11020da63822bc309e670752119a52058ee5062739a6a47" Nov 03 02:48:58.887610 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:58.887546 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1688a5c0a6b6349d11020da63822bc309e670752119a52058ee5062739a6a47"} err="failed to get container status \"a1688a5c0a6b6349d11020da63822bc309e670752119a52058ee5062739a6a47\": rpc error: code = NotFound desc = could not find container \"a1688a5c0a6b6349d11020da63822bc309e670752119a52058ee5062739a6a47\": container with ID starting with a1688a5c0a6b6349d11020da63822bc309e670752119a52058ee5062739a6a47 not found: ID does not exist" Nov 03 02:48:59.022544 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:59.022510 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-038c21-5d649bd5f-tcdq5"] Nov 03 02:48:59.069155 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:59.069114 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-038c21-5d649bd5f-tcdq5"] Nov 03 02:48:59.170262 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:59.170177 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-038c21-c54876867-hxprn"] Nov 03 02:48:59.174241 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:48:59.174211 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-038c21-c54876867-hxprn"] Nov 03 02:49:00.850294 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:00.850257 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b5077e6-2db7-45f4-85e4-6fca6f27b117" path="/var/lib/kubelet/pods/9b5077e6-2db7-45f4-85e4-6fca6f27b117/volumes" Nov 03 02:49:00.850877 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:00.850854 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d63996c8-1493-441a-927c-09b9f7394f12" path="/var/lib/kubelet/pods/d63996c8-1493-441a-927c-09b9f7394f12/volumes" Nov 03 02:49:06.332171 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:06.332137 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:49:23.756515 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:23.756430 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ca6ef4-7f7486c797-6kt8x"] Nov 03 02:49:23.866092 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:23.866059 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ca6ef4-74f5d66687-d4vr2"] Nov 03 02:49:23.916523 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:49:23.916486 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:49:28.720179 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:28.720147 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-604da0-77cc94d8c7-bssrs"] Nov 03 02:49:28.756865 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:28.756831 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-ca6ef4-7f7486c797-6kt8x" podUID="b22351dc-095e-4437-a2d7-23a1addf255b" containerName="payload-processing" containerID="cri-o://d15e5978ef6a7c452d65969e5487411857b579b87bacb4897d83f59687901ba1" gracePeriod=25 Nov 03 02:49:28.828813 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:28.828772 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-604da0-6f5db6dc4c-98d9d"] Nov 03 02:49:28.866946 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:28.866913 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ca6ef4-74f5d66687-d4vr2" podUID="dd54f6c1-849d-49e1-9d0c-ce8f8e7ccc96" containerName="payload-pre-processing" containerID="cri-o://7a88d9a9c4e94abc57ffe8bad693960985c955457d8c3cf9994194b8d0232cb6" gracePeriod=25 Nov 03 02:49:29.025754 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:29.025720 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ca6ef4-7f7486c797-6kt8x" Nov 03 02:49:29.057919 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:29.057882 5605 generic.go:356] "Generic (PLEG): container finished" podID="b22351dc-095e-4437-a2d7-23a1addf255b" containerID="d15e5978ef6a7c452d65969e5487411857b579b87bacb4897d83f59687901ba1" exitCode=0 Nov 03 02:49:29.057919 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:29.057925 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ca6ef4-7f7486c797-6kt8x" Nov 03 02:49:29.058170 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:29.058081 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ca6ef4-7f7486c797-6kt8x" event={"ID":"b22351dc-095e-4437-a2d7-23a1addf255b","Type":"ContainerDied","Data":"d15e5978ef6a7c452d65969e5487411857b579b87bacb4897d83f59687901ba1"} Nov 03 02:49:29.058170 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:29.058125 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ca6ef4-7f7486c797-6kt8x" event={"ID":"b22351dc-095e-4437-a2d7-23a1addf255b","Type":"ContainerDied","Data":"bfcc308ec0917d0154aac8433ea76548d7669c3b8160a0f40edf2ac391c49b41"} Nov 03 02:49:29.058170 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:29.058155 5605 scope.go:122] "RemoveContainer" containerID="d15e5978ef6a7c452d65969e5487411857b579b87bacb4897d83f59687901ba1" Nov 03 02:49:29.060428 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:29.060009 5605 generic.go:356] "Generic (PLEG): container finished" podID="dd54f6c1-849d-49e1-9d0c-ce8f8e7ccc96" containerID="7a88d9a9c4e94abc57ffe8bad693960985c955457d8c3cf9994194b8d0232cb6" exitCode=0 Nov 03 02:49:29.060428 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:29.060041 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ca6ef4-74f5d66687-d4vr2" event={"ID":"dd54f6c1-849d-49e1-9d0c-ce8f8e7ccc96","Type":"ContainerDied","Data":"7a88d9a9c4e94abc57ffe8bad693960985c955457d8c3cf9994194b8d0232cb6"} Nov 03 02:49:29.072584 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:29.072525 5605 scope.go:122] "RemoveContainer" containerID="d15e5978ef6a7c452d65969e5487411857b579b87bacb4897d83f59687901ba1" Nov 03 02:49:29.073158 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:49:29.073106 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d15e5978ef6a7c452d65969e5487411857b579b87bacb4897d83f59687901ba1\": container with ID starting with d15e5978ef6a7c452d65969e5487411857b579b87bacb4897d83f59687901ba1 not found: ID does not exist" containerID="d15e5978ef6a7c452d65969e5487411857b579b87bacb4897d83f59687901ba1" Nov 03 02:49:29.073272 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:29.073169 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d15e5978ef6a7c452d65969e5487411857b579b87bacb4897d83f59687901ba1"} err="failed to get container status \"d15e5978ef6a7c452d65969e5487411857b579b87bacb4897d83f59687901ba1\": rpc error: code = NotFound desc = could not find container \"d15e5978ef6a7c452d65969e5487411857b579b87bacb4897d83f59687901ba1\": container with ID starting with d15e5978ef6a7c452d65969e5487411857b579b87bacb4897d83f59687901ba1 not found: ID does not exist" Nov 03 02:49:29.159561 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:29.154810 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b22351dc-095e-4437-a2d7-23a1addf255b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b22351dc-095e-4437-a2d7-23a1addf255b-plugins-config-volume\") pod \"b22351dc-095e-4437-a2d7-23a1addf255b\" (UID: \"b22351dc-095e-4437-a2d7-23a1addf255b\") " Nov 03 02:49:29.159561 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:29.155054 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b22351dc-095e-4437-a2d7-23a1addf255b-kube-api-access-frts2\" (UniqueName: \"kubernetes.io/projected/b22351dc-095e-4437-a2d7-23a1addf255b-kube-api-access-frts2\") pod \"b22351dc-095e-4437-a2d7-23a1addf255b\" (UID: \"b22351dc-095e-4437-a2d7-23a1addf255b\") " Nov 03 02:49:29.167565 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:29.167513 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b22351dc-095e-4437-a2d7-23a1addf255b-plugins-config-volume" pod "b22351dc-095e-4437-a2d7-23a1addf255b" (UID: "b22351dc-095e-4437-a2d7-23a1addf255b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:49:29.168938 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:29.168914 5605 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b22351dc-095e-4437-a2d7-23a1addf255b-plugins-config-volume\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:49:29.169837 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:29.169813 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b22351dc-095e-4437-a2d7-23a1addf255b-kube-api-access-frts2" pod "b22351dc-095e-4437-a2d7-23a1addf255b" (UID: "b22351dc-095e-4437-a2d7-23a1addf255b"). InnerVolumeSpecName "kube-api-access-frts2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:49:29.172970 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:29.172949 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ca6ef4-74f5d66687-d4vr2" Nov 03 02:49:29.270155 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:29.269959 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd54f6c1-849d-49e1-9d0c-ce8f8e7ccc96-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd54f6c1-849d-49e1-9d0c-ce8f8e7ccc96-plugins-config-volume\") pod \"dd54f6c1-849d-49e1-9d0c-ce8f8e7ccc96\" (UID: \"dd54f6c1-849d-49e1-9d0c-ce8f8e7ccc96\") " Nov 03 02:49:29.270155 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:29.270011 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd54f6c1-849d-49e1-9d0c-ce8f8e7ccc96-kube-api-access-vblnv\" (UniqueName: \"kubernetes.io/projected/dd54f6c1-849d-49e1-9d0c-ce8f8e7ccc96-kube-api-access-vblnv\") pod \"dd54f6c1-849d-49e1-9d0c-ce8f8e7ccc96\" (UID: \"dd54f6c1-849d-49e1-9d0c-ce8f8e7ccc96\") " Nov 03 02:49:29.272029 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:29.270836 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frts2\" (UniqueName: \"kubernetes.io/projected/b22351dc-095e-4437-a2d7-23a1addf255b-kube-api-access-frts2\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:49:29.272218 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:29.272188 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd54f6c1-849d-49e1-9d0c-ce8f8e7ccc96-plugins-config-volume" pod "dd54f6c1-849d-49e1-9d0c-ce8f8e7ccc96" (UID: "dd54f6c1-849d-49e1-9d0c-ce8f8e7ccc96"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:49:29.274541 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:29.274503 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd54f6c1-849d-49e1-9d0c-ce8f8e7ccc96-kube-api-access-vblnv" pod "dd54f6c1-849d-49e1-9d0c-ce8f8e7ccc96" (UID: "dd54f6c1-849d-49e1-9d0c-ce8f8e7ccc96"). InnerVolumeSpecName "kube-api-access-vblnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:49:29.371835 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:29.371810 5605 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd54f6c1-849d-49e1-9d0c-ce8f8e7ccc96-plugins-config-volume\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:49:29.371835 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:29.371835 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vblnv\" (UniqueName: \"kubernetes.io/projected/dd54f6c1-849d-49e1-9d0c-ce8f8e7ccc96-kube-api-access-vblnv\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:49:29.473315 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:29.473280 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ca6ef4-7f7486c797-6kt8x"] Nov 03 02:49:29.476442 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:29.476417 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ca6ef4-7f7486c797-6kt8x"] Nov 03 02:49:30.068811 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:30.068783 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ca6ef4-74f5d66687-d4vr2" Nov 03 02:49:30.068811 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:30.068795 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ca6ef4-74f5d66687-d4vr2" event={"ID":"dd54f6c1-849d-49e1-9d0c-ce8f8e7ccc96","Type":"ContainerDied","Data":"31fae1962c9104a7174228e0877d5b61b78d91bfd9aee84ca12738b5a9aee92d"} Nov 03 02:49:30.069302 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:30.068853 5605 scope.go:122] "RemoveContainer" containerID="7a88d9a9c4e94abc57ffe8bad693960985c955457d8c3cf9994194b8d0232cb6" Nov 03 02:49:30.226168 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:30.226135 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ca6ef4-74f5d66687-d4vr2"] Nov 03 02:49:30.272117 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:30.272090 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ca6ef4-74f5d66687-d4vr2"] Nov 03 02:49:30.850917 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:30.850886 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b22351dc-095e-4437-a2d7-23a1addf255b" path="/var/lib/kubelet/pods/b22351dc-095e-4437-a2d7-23a1addf255b/volumes" Nov 03 02:49:30.851325 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:30.851308 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd54f6c1-849d-49e1-9d0c-ce8f8e7ccc96" path="/var/lib/kubelet/pods/dd54f6c1-849d-49e1-9d0c-ce8f8e7ccc96/volumes" Nov 03 02:49:33.721351 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:33.721290 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-604da0-77cc94d8c7-bssrs" podUID="2e7953a5-9b97-41de-96af-c3827c82254c" containerName="payload-processing" containerID="cri-o://1c430c3f1820aee3a0e99983aba6956ce12d7bc17832cb3a892ecd2e03b0d8ea" gracePeriod=25 Nov 03 02:49:33.797412 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:33.797336 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-c8ee6b/autores-model-c8ee6b-kserve-6f47664dc7-v9c4p"] Nov 03 02:49:33.830234 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:33.830197 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-604da0-6f5db6dc4c-98d9d" podUID="bd510465-adee-438c-908b-79808324282a" containerName="payload-pre-processing" containerID="cri-o://55d2dd3c161752bbb98fb98aed6722bb94c58c289069b46d2d32fdd759621f6a" gracePeriod=25 Nov 03 02:49:34.063436 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:34.062514 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-604da0-77cc94d8c7-bssrs" Nov 03 02:49:34.107739 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:34.107702 5605 generic.go:356] "Generic (PLEG): container finished" podID="2e7953a5-9b97-41de-96af-c3827c82254c" containerID="1c430c3f1820aee3a0e99983aba6956ce12d7bc17832cb3a892ecd2e03b0d8ea" exitCode=0 Nov 03 02:49:34.107936 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:34.107817 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-604da0-77cc94d8c7-bssrs" Nov 03 02:49:34.107936 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:34.107838 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-604da0-77cc94d8c7-bssrs" event={"ID":"2e7953a5-9b97-41de-96af-c3827c82254c","Type":"ContainerDied","Data":"1c430c3f1820aee3a0e99983aba6956ce12d7bc17832cb3a892ecd2e03b0d8ea"} Nov 03 02:49:34.107936 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:34.107930 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-604da0-77cc94d8c7-bssrs" event={"ID":"2e7953a5-9b97-41de-96af-c3827c82254c","Type":"ContainerDied","Data":"729fcb729fb2ef0067feeb155dcfb1408f1ddc2883b33e6305116321d0da65f8"} Nov 03 02:49:34.108118 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:34.107961 5605 scope.go:122] "RemoveContainer" containerID="1c430c3f1820aee3a0e99983aba6956ce12d7bc17832cb3a892ecd2e03b0d8ea" Nov 03 02:49:34.111790 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:34.111634 5605 generic.go:356] "Generic (PLEG): container finished" podID="bd510465-adee-438c-908b-79808324282a" containerID="55d2dd3c161752bbb98fb98aed6722bb94c58c289069b46d2d32fdd759621f6a" exitCode=0 Nov 03 02:49:34.111790 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:34.111729 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-604da0-6f5db6dc4c-98d9d" event={"ID":"bd510465-adee-438c-908b-79808324282a","Type":"ContainerDied","Data":"55d2dd3c161752bbb98fb98aed6722bb94c58c289069b46d2d32fdd759621f6a"} Nov 03 02:49:34.123140 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:34.123125 5605 scope.go:122] "RemoveContainer" containerID="1c430c3f1820aee3a0e99983aba6956ce12d7bc17832cb3a892ecd2e03b0d8ea" Nov 03 02:49:34.123713 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:49:34.123619 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c430c3f1820aee3a0e99983aba6956ce12d7bc17832cb3a892ecd2e03b0d8ea\": container with ID starting with 1c430c3f1820aee3a0e99983aba6956ce12d7bc17832cb3a892ecd2e03b0d8ea not found: ID does not exist" containerID="1c430c3f1820aee3a0e99983aba6956ce12d7bc17832cb3a892ecd2e03b0d8ea" Nov 03 02:49:34.123713 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:34.123652 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c430c3f1820aee3a0e99983aba6956ce12d7bc17832cb3a892ecd2e03b0d8ea"} err="failed to get container status \"1c430c3f1820aee3a0e99983aba6956ce12d7bc17832cb3a892ecd2e03b0d8ea\": rpc error: code = NotFound desc = could not find container \"1c430c3f1820aee3a0e99983aba6956ce12d7bc17832cb3a892ecd2e03b0d8ea\": container with ID starting with 1c430c3f1820aee3a0e99983aba6956ce12d7bc17832cb3a892ecd2e03b0d8ea not found: ID does not exist" Nov 03 02:49:34.134524 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:34.133188 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e7953a5-9b97-41de-96af-c3827c82254c-kube-api-access-6pj95\" (UniqueName: \"kubernetes.io/projected/2e7953a5-9b97-41de-96af-c3827c82254c-kube-api-access-6pj95\") pod \"2e7953a5-9b97-41de-96af-c3827c82254c\" (UID: \"2e7953a5-9b97-41de-96af-c3827c82254c\") " Nov 03 02:49:34.134524 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:34.133301 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e7953a5-9b97-41de-96af-c3827c82254c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e7953a5-9b97-41de-96af-c3827c82254c-plugins-config-volume\") pod \"2e7953a5-9b97-41de-96af-c3827c82254c\" (UID: \"2e7953a5-9b97-41de-96af-c3827c82254c\") " Nov 03 02:49:34.134524 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:34.134111 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e7953a5-9b97-41de-96af-c3827c82254c-plugins-config-volume" pod "2e7953a5-9b97-41de-96af-c3827c82254c" (UID: "2e7953a5-9b97-41de-96af-c3827c82254c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:49:34.136527 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:34.136506 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e7953a5-9b97-41de-96af-c3827c82254c-kube-api-access-6pj95" pod "2e7953a5-9b97-41de-96af-c3827c82254c" (UID: "2e7953a5-9b97-41de-96af-c3827c82254c"). InnerVolumeSpecName "kube-api-access-6pj95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:49:34.142987 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:34.142967 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-604da0-6f5db6dc4c-98d9d" Nov 03 02:49:34.234902 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:34.234867 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd510465-adee-438c-908b-79808324282a-kube-api-access-nfpv9\" (UniqueName: \"kubernetes.io/projected/bd510465-adee-438c-908b-79808324282a-kube-api-access-nfpv9\") pod \"bd510465-adee-438c-908b-79808324282a\" (UID: \"bd510465-adee-438c-908b-79808324282a\") " Nov 03 02:49:34.235106 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:34.234982 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd510465-adee-438c-908b-79808324282a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd510465-adee-438c-908b-79808324282a-plugins-config-volume\") pod \"bd510465-adee-438c-908b-79808324282a\" (UID: \"bd510465-adee-438c-908b-79808324282a\") " Nov 03 02:49:34.235456 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:34.235433 5605 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e7953a5-9b97-41de-96af-c3827c82254c-plugins-config-volume\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:49:34.235541 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:34.235465 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pj95\" (UniqueName: \"kubernetes.io/projected/2e7953a5-9b97-41de-96af-c3827c82254c-kube-api-access-6pj95\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:49:34.235541 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:34.235432 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd510465-adee-438c-908b-79808324282a-plugins-config-volume" pod "bd510465-adee-438c-908b-79808324282a" (UID: "bd510465-adee-438c-908b-79808324282a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:49:34.237008 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:34.236983 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd510465-adee-438c-908b-79808324282a-kube-api-access-nfpv9" pod "bd510465-adee-438c-908b-79808324282a" (UID: "bd510465-adee-438c-908b-79808324282a"). InnerVolumeSpecName "kube-api-access-nfpv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:49:34.336896 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:34.336866 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfpv9\" (UniqueName: \"kubernetes.io/projected/bd510465-adee-438c-908b-79808324282a-kube-api-access-nfpv9\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:49:34.336896 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:34.336892 5605 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd510465-adee-438c-908b-79808324282a-plugins-config-volume\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:49:34.523628 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:34.523596 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-604da0-77cc94d8c7-bssrs"] Nov 03 02:49:34.526451 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:34.526426 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-604da0-77cc94d8c7-bssrs"] Nov 03 02:49:34.850284 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:34.850249 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e7953a5-9b97-41de-96af-c3827c82254c" path="/var/lib/kubelet/pods/2e7953a5-9b97-41de-96af-c3827c82254c/volumes" Nov 03 02:49:35.120200 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:35.120166 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-604da0-6f5db6dc4c-98d9d" Nov 03 02:49:35.120400 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:35.120206 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-604da0-6f5db6dc4c-98d9d" event={"ID":"bd510465-adee-438c-908b-79808324282a","Type":"ContainerDied","Data":"46be6258471609ead0d359296cd03612095e5b8202167616546cc5a17a7fb487"} Nov 03 02:49:35.120400 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:35.120270 5605 scope.go:122] "RemoveContainer" containerID="55d2dd3c161752bbb98fb98aed6722bb94c58c289069b46d2d32fdd759621f6a" Nov 03 02:49:35.274012 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:35.273977 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-604da0-6f5db6dc4c-98d9d"] Nov 03 02:49:35.280566 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:35.280537 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-604da0-6f5db6dc4c-98d9d"] Nov 03 02:49:36.851304 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:36.851273 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd510465-adee-438c-908b-79808324282a" path="/var/lib/kubelet/pods/bd510465-adee-438c-908b-79808324282a/volumes" Nov 03 02:49:42.136937 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:42.136903 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-038c21-7vqqx"] Nov 03 02:49:42.137353 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:42.137341 5605 cpu_manager.go:414] "containerMap: removing container" podUID="b22351dc-095e-4437-a2d7-23a1addf255b" containerName="payload-processing" Nov 03 02:49:42.137353 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:42.137365 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b22351dc-095e-4437-a2d7-23a1addf255b" containerName="payload-processing" Nov 03 02:49:42.137463 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:42.137377 5605 cpu_manager.go:414] "containerMap: removing container" podUID="bd510465-adee-438c-908b-79808324282a" containerName="payload-pre-processing" Nov 03 02:49:42.137463 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:42.137383 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd510465-adee-438c-908b-79808324282a" containerName="payload-pre-processing" Nov 03 02:49:42.137463 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:42.137401 5605 cpu_manager.go:414] "containerMap: removing container" podUID="d63996c8-1493-441a-927c-09b9f7394f12" containerName="payload-processing" Nov 03 02:49:42.137463 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:42.137406 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d63996c8-1493-441a-927c-09b9f7394f12" containerName="payload-processing" Nov 03 02:49:42.137463 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:42.137430 5605 cpu_manager.go:414] "containerMap: removing container" podUID="2e7953a5-9b97-41de-96af-c3827c82254c" containerName="payload-processing" Nov 03 02:49:42.137463 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:42.137437 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e7953a5-9b97-41de-96af-c3827c82254c" containerName="payload-processing" Nov 03 02:49:42.137463 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:42.137455 5605 cpu_manager.go:414] "containerMap: removing container" podUID="dd54f6c1-849d-49e1-9d0c-ce8f8e7ccc96" containerName="payload-pre-processing" Nov 03 02:49:42.137463 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:42.137462 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd54f6c1-849d-49e1-9d0c-ce8f8e7ccc96" containerName="payload-pre-processing" Nov 03 02:49:42.137722 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:42.137476 5605 cpu_manager.go:414] "containerMap: removing container" podUID="1f935c46-6569-4583-a6df-d45c36c1aa62" containerName="main" Nov 03 02:49:42.137722 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:42.137481 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f935c46-6569-4583-a6df-d45c36c1aa62" containerName="main" Nov 03 02:49:42.137722 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:42.137491 5605 cpu_manager.go:414] "containerMap: removing container" podUID="9b5077e6-2db7-45f4-85e4-6fca6f27b117" containerName="payload-pre-processing" Nov 03 02:49:42.137722 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:42.137496 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b5077e6-2db7-45f4-85e4-6fca6f27b117" containerName="payload-pre-processing" Nov 03 02:49:42.137722 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:42.137576 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="d63996c8-1493-441a-927c-09b9f7394f12" containerName="payload-processing" Nov 03 02:49:42.137722 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:42.137586 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e7953a5-9b97-41de-96af-c3827c82254c" containerName="payload-processing" Nov 03 02:49:42.137722 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:42.137594 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd54f6c1-849d-49e1-9d0c-ce8f8e7ccc96" containerName="payload-pre-processing" Nov 03 02:49:42.137722 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:42.137601 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f935c46-6569-4583-a6df-d45c36c1aa62" containerName="main" Nov 03 02:49:42.137722 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:42.137607 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b5077e6-2db7-45f4-85e4-6fca6f27b117" containerName="payload-pre-processing" Nov 03 02:49:42.137722 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:42.137614 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="b22351dc-095e-4437-a2d7-23a1addf255b" containerName="payload-processing" Nov 03 02:49:42.137722 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:42.137618 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd510465-adee-438c-908b-79808324282a" containerName="payload-pre-processing" Nov 03 02:49:42.142264 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:42.142245 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-038c21-7vqqx" Nov 03 02:49:42.238170 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:42.238106 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:49:42.238170 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:42.238105 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:49:42.238170 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:42.238140 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:49:42.238170 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:42.238152 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:49:42.240858 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:42.240833 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-038c21-7vqqx"] Nov 03 02:49:42.321165 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:42.321134 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06e3ae22-d3dd-48e0-b9e1-19e75481f880-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-038c21-7vqqx\" (UID: \"06e3ae22-d3dd-48e0-b9e1-19e75481f880\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-038c21-7vqqx" Nov 03 02:49:42.321320 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:42.321304 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06e3ae22-d3dd-48e0-b9e1-19e75481f880-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-038c21-7vqqx\" (UID: \"06e3ae22-d3dd-48e0-b9e1-19e75481f880\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-038c21-7vqqx" Nov 03 02:49:42.321399 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:42.321379 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qks5\" (UniqueName: \"kubernetes.io/projected/06e3ae22-d3dd-48e0-b9e1-19e75481f880-kube-api-access-9qks5\") pod \"maas-api-revoke-keys-e2e-explicit-038c21-7vqqx\" (UID: \"06e3ae22-d3dd-48e0-b9e1-19e75481f880\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-038c21-7vqqx" Nov 03 02:49:42.422597 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:42.422517 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06e3ae22-d3dd-48e0-b9e1-19e75481f880-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-038c21-7vqqx\" (UID: \"06e3ae22-d3dd-48e0-b9e1-19e75481f880\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-038c21-7vqqx" Nov 03 02:49:42.422769 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:42.422647 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06e3ae22-d3dd-48e0-b9e1-19e75481f880-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-038c21-7vqqx\" (UID: \"06e3ae22-d3dd-48e0-b9e1-19e75481f880\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-038c21-7vqqx" Nov 03 02:49:42.422769 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:42.422679 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qks5\" (UniqueName: \"kubernetes.io/projected/06e3ae22-d3dd-48e0-b9e1-19e75481f880-kube-api-access-9qks5\") pod \"maas-api-revoke-keys-e2e-explicit-038c21-7vqqx\" (UID: \"06e3ae22-d3dd-48e0-b9e1-19e75481f880\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-038c21-7vqqx" Nov 03 02:49:42.423252 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:42.423220 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06e3ae22-d3dd-48e0-b9e1-19e75481f880-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-038c21-7vqqx\" (UID: \"06e3ae22-d3dd-48e0-b9e1-19e75481f880\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-038c21-7vqqx" Nov 03 02:49:42.425061 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:42.425039 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06e3ae22-d3dd-48e0-b9e1-19e75481f880-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-038c21-7vqqx\" (UID: \"06e3ae22-d3dd-48e0-b9e1-19e75481f880\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-038c21-7vqqx" Nov 03 02:49:42.476218 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:42.476193 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qks5\" (UniqueName: \"kubernetes.io/projected/06e3ae22-d3dd-48e0-b9e1-19e75481f880-kube-api-access-9qks5\") pod \"maas-api-revoke-keys-e2e-explicit-038c21-7vqqx\" (UID: \"06e3ae22-d3dd-48e0-b9e1-19e75481f880\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-038c21-7vqqx" Nov 03 02:49:42.755725 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:42.755657 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-038c21-7vqqx" Nov 03 02:49:43.046135 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:43.046064 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbe69693-fb5t7"] Nov 03 02:49:43.092921 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:43.092894 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbe69693-fb5t7"] Nov 03 02:49:43.160956 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:43.160922 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-038c21-7vqqx"] Nov 03 02:49:43.166577 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:49:43.166553 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06e3ae22_d3dd_48e0_b9e1_19e75481f880.slice/crio-d89955a6d3c46ee2bec6c086c9936afd6bb9da85491fcced8ae724d9807a921a WatchSource:0}: Error finding container d89955a6d3c46ee2bec6c086c9936afd6bb9da85491fcced8ae724d9807a921a: Status 404 returned error can't find the container with id d89955a6d3c46ee2bec6c086c9936afd6bb9da85491fcced8ae724d9807a921a Nov 03 02:49:43.184056 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:43.184032 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-038c21-7vqqx" event={"ID":"06e3ae22-d3dd-48e0-b9e1-19e75481f880","Type":"ContainerStarted","Data":"d89955a6d3c46ee2bec6c086c9936afd6bb9da85491fcced8ae724d9807a921a"} Nov 03 02:49:44.043368 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:44.043330 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9324d1e-8rvcj"] Nov 03 02:49:44.090338 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:44.090301 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9324d1e-8rvcj"] Nov 03 02:49:44.191380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:44.191319 5605 generic.go:356] "Generic (PLEG): container finished" podID="06e3ae22-d3dd-48e0-b9e1-19e75481f880" containerID="f8a3f0d4807fb3d320385417ad1b7024a03109c01b21af494473392b8f78bbfe" exitCode=0 Nov 03 02:49:44.191779 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:44.191394 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-038c21-7vqqx" event={"ID":"06e3ae22-d3dd-48e0-b9e1-19e75481f880","Type":"ContainerDied","Data":"f8a3f0d4807fb3d320385417ad1b7024a03109c01b21af494473392b8f78bbfe"} Nov 03 02:49:44.850198 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:44.850164 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4777275b-ba13-499d-81ef-1706e481f3f2" path="/var/lib/kubelet/pods/4777275b-ba13-499d-81ef-1706e481f3f2/volumes" Nov 03 02:49:44.850668 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:44.850652 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c10e9e9-acb8-4f2b-ae14-a8fde1a01aa4" path="/var/lib/kubelet/pods/9c10e9e9-acb8-4f2b-ae14-a8fde1a01aa4/volumes" Nov 03 02:49:45.340886 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:45.340861 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-038c21-7vqqx" Nov 03 02:49:45.354027 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:45.354002 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06e3ae22-d3dd-48e0-b9e1-19e75481f880-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06e3ae22-d3dd-48e0-b9e1-19e75481f880-maas-api-openshift-ca\") pod \"06e3ae22-d3dd-48e0-b9e1-19e75481f880\" (UID: \"06e3ae22-d3dd-48e0-b9e1-19e75481f880\") " Nov 03 02:49:45.354229 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:45.354213 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06e3ae22-d3dd-48e0-b9e1-19e75481f880-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06e3ae22-d3dd-48e0-b9e1-19e75481f880-maas-api-xks-ca\") pod \"06e3ae22-d3dd-48e0-b9e1-19e75481f880\" (UID: \"06e3ae22-d3dd-48e0-b9e1-19e75481f880\") " Nov 03 02:49:45.354318 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:45.354302 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06e3ae22-d3dd-48e0-b9e1-19e75481f880-kube-api-access-9qks5\" (UniqueName: \"kubernetes.io/projected/06e3ae22-d3dd-48e0-b9e1-19e75481f880-kube-api-access-9qks5\") pod \"06e3ae22-d3dd-48e0-b9e1-19e75481f880\" (UID: \"06e3ae22-d3dd-48e0-b9e1-19e75481f880\") " Nov 03 02:49:45.354465 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:45.354432 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06e3ae22-d3dd-48e0-b9e1-19e75481f880-maas-api-openshift-ca" pod "06e3ae22-d3dd-48e0-b9e1-19e75481f880" (UID: "06e3ae22-d3dd-48e0-b9e1-19e75481f880"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:49:45.354765 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:45.354735 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06e3ae22-d3dd-48e0-b9e1-19e75481f880-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:49:45.356458 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:45.356417 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06e3ae22-d3dd-48e0-b9e1-19e75481f880-kube-api-access-9qks5" pod "06e3ae22-d3dd-48e0-b9e1-19e75481f880" (UID: "06e3ae22-d3dd-48e0-b9e1-19e75481f880"). InnerVolumeSpecName "kube-api-access-9qks5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:49:45.356566 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:45.356472 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06e3ae22-d3dd-48e0-b9e1-19e75481f880-maas-api-xks-ca" pod "06e3ae22-d3dd-48e0-b9e1-19e75481f880" (UID: "06e3ae22-d3dd-48e0-b9e1-19e75481f880"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:49:45.455866 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:45.455780 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qks5\" (UniqueName: \"kubernetes.io/projected/06e3ae22-d3dd-48e0-b9e1-19e75481f880-kube-api-access-9qks5\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:49:45.455866 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:45.455820 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06e3ae22-d3dd-48e0-b9e1-19e75481f880-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:49:46.204814 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:46.204767 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-038c21-7vqqx" Nov 03 02:49:46.204989 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:46.204769 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-038c21-7vqqx" event={"ID":"06e3ae22-d3dd-48e0-b9e1-19e75481f880","Type":"ContainerDied","Data":"d89955a6d3c46ee2bec6c086c9936afd6bb9da85491fcced8ae724d9807a921a"} Nov 03 02:49:46.204989 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:46.204897 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d89955a6d3c46ee2bec6c086c9936afd6bb9da85491fcced8ae724d9807a921a" Nov 03 02:49:48.817639 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:48.817604 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-autores-c8ee6b/autores-model-c8ee6b-kserve-6f47664dc7-v9c4p" podUID="db0c38c1-8105-49de-9561-319a632581bd" containerName="main" containerID="cri-o://3247d35a542207ccefb3e7151dbf89916bcbc2e3eece73b56d644e9d4ccf349b" gracePeriod=45 Nov 03 02:49:49.062757 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:49.062730 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-c8ee6b/autores-model-c8ee6b-kserve-6f47664dc7-v9c4p" Nov 03 02:49:49.093122 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:49.093049 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db0c38c1-8105-49de-9561-319a632581bd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db0c38c1-8105-49de-9561-319a632581bd-tmp-dir\") pod \"db0c38c1-8105-49de-9561-319a632581bd\" (UID: \"db0c38c1-8105-49de-9561-319a632581bd\") " Nov 03 02:49:49.093257 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:49.093165 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db0c38c1-8105-49de-9561-319a632581bd-kube-api-access-c9k6j\" (UniqueName: \"kubernetes.io/projected/db0c38c1-8105-49de-9561-319a632581bd-kube-api-access-c9k6j\") pod \"db0c38c1-8105-49de-9561-319a632581bd\" (UID: \"db0c38c1-8105-49de-9561-319a632581bd\") " Nov 03 02:49:49.093257 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:49.093223 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db0c38c1-8105-49de-9561-319a632581bd-home\" (UniqueName: \"kubernetes.io/empty-dir/db0c38c1-8105-49de-9561-319a632581bd-home\") pod \"db0c38c1-8105-49de-9561-319a632581bd\" (UID: \"db0c38c1-8105-49de-9561-319a632581bd\") " Nov 03 02:49:49.093382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:49.093285 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db0c38c1-8105-49de-9561-319a632581bd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/db0c38c1-8105-49de-9561-319a632581bd-dshm\") pod \"db0c38c1-8105-49de-9561-319a632581bd\" (UID: \"db0c38c1-8105-49de-9561-319a632581bd\") " Nov 03 02:49:49.093382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:49.093306 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db0c38c1-8105-49de-9561-319a632581bd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db0c38c1-8105-49de-9561-319a632581bd-model-cache\") pod \"db0c38c1-8105-49de-9561-319a632581bd\" (UID: \"db0c38c1-8105-49de-9561-319a632581bd\") " Nov 03 02:49:49.093382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:49.093332 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db0c38c1-8105-49de-9561-319a632581bd-tls-certs\" (UniqueName: \"kubernetes.io/secret/db0c38c1-8105-49de-9561-319a632581bd-tls-certs\") pod \"db0c38c1-8105-49de-9561-319a632581bd\" (UID: \"db0c38c1-8105-49de-9561-319a632581bd\") " Nov 03 02:49:49.093548 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:49.093300 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db0c38c1-8105-49de-9561-319a632581bd-tmp-dir" pod "db0c38c1-8105-49de-9561-319a632581bd" (UID: "db0c38c1-8105-49de-9561-319a632581bd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:49:49.093609 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:49.093568 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db0c38c1-8105-49de-9561-319a632581bd-home" pod "db0c38c1-8105-49de-9561-319a632581bd" (UID: "db0c38c1-8105-49de-9561-319a632581bd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:49:49.093665 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:49.093625 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db0c38c1-8105-49de-9561-319a632581bd-model-cache" pod "db0c38c1-8105-49de-9561-319a632581bd" (UID: "db0c38c1-8105-49de-9561-319a632581bd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:49:49.093759 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:49.093738 5605 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db0c38c1-8105-49de-9561-319a632581bd-home\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:49:49.093834 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:49.093767 5605 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db0c38c1-8105-49de-9561-319a632581bd-model-cache\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:49:49.093834 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:49.093784 5605 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db0c38c1-8105-49de-9561-319a632581bd-tmp-dir\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:49:49.095526 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:49.095501 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db0c38c1-8105-49de-9561-319a632581bd-dshm" pod "db0c38c1-8105-49de-9561-319a632581bd" (UID: "db0c38c1-8105-49de-9561-319a632581bd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:49:49.095526 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:49.095518 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db0c38c1-8105-49de-9561-319a632581bd-kube-api-access-c9k6j" pod "db0c38c1-8105-49de-9561-319a632581bd" (UID: "db0c38c1-8105-49de-9561-319a632581bd"). InnerVolumeSpecName "kube-api-access-c9k6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:49:49.095668 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:49.095571 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db0c38c1-8105-49de-9561-319a632581bd-tls-certs" pod "db0c38c1-8105-49de-9561-319a632581bd" (UID: "db0c38c1-8105-49de-9561-319a632581bd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:49:49.194498 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:49.194469 5605 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db0c38c1-8105-49de-9561-319a632581bd-dshm\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:49:49.194498 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:49.194492 5605 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db0c38c1-8105-49de-9561-319a632581bd-tls-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:49:49.194498 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:49.194502 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9k6j\" (UniqueName: \"kubernetes.io/projected/db0c38c1-8105-49de-9561-319a632581bd-kube-api-access-c9k6j\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:49:49.224722 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:49.224688 5605 generic.go:356] "Generic (PLEG): container finished" podID="db0c38c1-8105-49de-9561-319a632581bd" containerID="3247d35a542207ccefb3e7151dbf89916bcbc2e3eece73b56d644e9d4ccf349b" exitCode=0 Nov 03 02:49:49.224857 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:49.224740 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-c8ee6b/autores-model-c8ee6b-kserve-6f47664dc7-v9c4p" Nov 03 02:49:49.224857 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:49.224775 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-c8ee6b/autores-model-c8ee6b-kserve-6f47664dc7-v9c4p" event={"ID":"db0c38c1-8105-49de-9561-319a632581bd","Type":"ContainerDied","Data":"3247d35a542207ccefb3e7151dbf89916bcbc2e3eece73b56d644e9d4ccf349b"} Nov 03 02:49:49.224857 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:49.224817 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-c8ee6b/autores-model-c8ee6b-kserve-6f47664dc7-v9c4p" event={"ID":"db0c38c1-8105-49de-9561-319a632581bd","Type":"ContainerDied","Data":"7ff520b6c38adb9e5204189861c5614aef1abd6ab1d70872e7451963a31038a2"} Nov 03 02:49:49.224857 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:49.224836 5605 scope.go:122] "RemoveContainer" containerID="3247d35a542207ccefb3e7151dbf89916bcbc2e3eece73b56d644e9d4ccf349b" Nov 03 02:49:49.234595 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:49.234577 5605 scope.go:122] "RemoveContainer" containerID="3247d35a542207ccefb3e7151dbf89916bcbc2e3eece73b56d644e9d4ccf349b" Nov 03 02:49:49.234845 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:49:49.234827 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3247d35a542207ccefb3e7151dbf89916bcbc2e3eece73b56d644e9d4ccf349b\": container with ID starting with 3247d35a542207ccefb3e7151dbf89916bcbc2e3eece73b56d644e9d4ccf349b not found: ID does not exist" containerID="3247d35a542207ccefb3e7151dbf89916bcbc2e3eece73b56d644e9d4ccf349b" Nov 03 02:49:49.234908 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:49.234852 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3247d35a542207ccefb3e7151dbf89916bcbc2e3eece73b56d644e9d4ccf349b"} err="failed to get container status \"3247d35a542207ccefb3e7151dbf89916bcbc2e3eece73b56d644e9d4ccf349b\": rpc error: code = NotFound desc = could not find container \"3247d35a542207ccefb3e7151dbf89916bcbc2e3eece73b56d644e9d4ccf349b\": container with ID starting with 3247d35a542207ccefb3e7151dbf89916bcbc2e3eece73b56d644e9d4ccf349b not found: ID does not exist" Nov 03 02:49:49.377961 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:49.377889 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-c8ee6b/autores-model-c8ee6b-kserve-6f47664dc7-v9c4p"] Nov 03 02:49:49.380011 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:49.379978 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-c8ee6b/autores-model-c8ee6b-kserve-6f47664dc7-v9c4p"] Nov 03 02:49:50.851531 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:50.851500 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db0c38c1-8105-49de-9561-319a632581bd" path="/var/lib/kubelet/pods/db0c38c1-8105-49de-9561-319a632581bd/volumes" Nov 03 02:49:53.042880 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:53.042845 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6408ce5d-t5sb8"] Nov 03 02:49:53.089152 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:53.089126 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6408ce5d-t5sb8"] Nov 03 02:49:53.915333 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:49:53.915272 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:49:54.043494 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:54.043457 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79f36937-fgnzb"] Nov 03 02:49:54.089728 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:54.089699 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79f36937-fgnzb"] Nov 03 02:49:54.327045 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:54.327007 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e3cc05-6468f7548f-rl2hr"] Nov 03 02:49:54.431444 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:54.431407 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e3cc05-66dfb8cc6c-8xgvh"] Nov 03 02:49:54.872425 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:54.872383 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6472c30-5f73-499e-a92f-a3950c2c40f0" path="/var/lib/kubelet/pods/a6472c30-5f73-499e-a92f-a3950c2c40f0/volumes" Nov 03 02:49:54.873167 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:54.873081 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1a4aff6-eb0a-4234-9642-e5bbe77e3102" path="/var/lib/kubelet/pods/d1a4aff6-eb0a-4234-9642-e5bbe77e3102/volumes" Nov 03 02:49:57.778742 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:57.778707 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:49:59.327465 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:59.327422 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-e3cc05-6468f7548f-rl2hr" podUID="198ce753-de2f-471c-894f-872099dbef2c" containerName="payload-processing" containerID="cri-o://c71b184be1d2af5d16fc1b1e2f50ddd0a22a9065dd23adaec7cb750ec7ec27aa" gracePeriod=25 Nov 03 02:49:59.432747 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:59.432713 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e3cc05-66dfb8cc6c-8xgvh" podUID="1773b46a-1dde-4944-b001-06cda4bedcd6" containerName="payload-pre-processing" containerID="cri-o://e9cce2470453155af47146e17d4275fd9bdb8007f9f08d673eda0097d4f38597" gracePeriod=25 Nov 03 02:49:59.593955 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:59.593855 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-e3cc05-6468f7548f-rl2hr" Nov 03 02:49:59.702033 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:59.702002 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e3cc05-66dfb8cc6c-8xgvh" Nov 03 02:49:59.718108 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:59.718080 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/198ce753-de2f-471c-894f-872099dbef2c-kube-api-access-f8t8t\" (UniqueName: \"kubernetes.io/projected/198ce753-de2f-471c-894f-872099dbef2c-kube-api-access-f8t8t\") pod \"198ce753-de2f-471c-894f-872099dbef2c\" (UID: \"198ce753-de2f-471c-894f-872099dbef2c\") " Nov 03 02:49:59.718349 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:59.718211 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/198ce753-de2f-471c-894f-872099dbef2c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/198ce753-de2f-471c-894f-872099dbef2c-plugins-config-volume\") pod \"198ce753-de2f-471c-894f-872099dbef2c\" (UID: \"198ce753-de2f-471c-894f-872099dbef2c\") " Nov 03 02:49:59.718660 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:59.718632 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/198ce753-de2f-471c-894f-872099dbef2c-plugins-config-volume" pod "198ce753-de2f-471c-894f-872099dbef2c" (UID: "198ce753-de2f-471c-894f-872099dbef2c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:49:59.720197 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:59.720177 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/198ce753-de2f-471c-894f-872099dbef2c-kube-api-access-f8t8t" pod "198ce753-de2f-471c-894f-872099dbef2c" (UID: "198ce753-de2f-471c-894f-872099dbef2c"). InnerVolumeSpecName "kube-api-access-f8t8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:49:59.819474 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:59.819440 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1773b46a-1dde-4944-b001-06cda4bedcd6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1773b46a-1dde-4944-b001-06cda4bedcd6-plugins-config-volume\") pod \"1773b46a-1dde-4944-b001-06cda4bedcd6\" (UID: \"1773b46a-1dde-4944-b001-06cda4bedcd6\") " Nov 03 02:49:59.819659 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:59.819550 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1773b46a-1dde-4944-b001-06cda4bedcd6-kube-api-access-m9bpv\" (UniqueName: \"kubernetes.io/projected/1773b46a-1dde-4944-b001-06cda4bedcd6-kube-api-access-m9bpv\") pod \"1773b46a-1dde-4944-b001-06cda4bedcd6\" (UID: \"1773b46a-1dde-4944-b001-06cda4bedcd6\") " Nov 03 02:49:59.819817 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:59.819792 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1773b46a-1dde-4944-b001-06cda4bedcd6-plugins-config-volume" pod "1773b46a-1dde-4944-b001-06cda4bedcd6" (UID: "1773b46a-1dde-4944-b001-06cda4bedcd6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:49:59.819965 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:59.819950 5605 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1773b46a-1dde-4944-b001-06cda4bedcd6-plugins-config-volume\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:49:59.820031 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:59.819967 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8t8t\" (UniqueName: \"kubernetes.io/projected/198ce753-de2f-471c-894f-872099dbef2c-kube-api-access-f8t8t\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:49:59.820031 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:59.819977 5605 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/198ce753-de2f-471c-894f-872099dbef2c-plugins-config-volume\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:49:59.821778 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:59.821750 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1773b46a-1dde-4944-b001-06cda4bedcd6-kube-api-access-m9bpv" pod "1773b46a-1dde-4944-b001-06cda4bedcd6" (UID: "1773b46a-1dde-4944-b001-06cda4bedcd6"). InnerVolumeSpecName "kube-api-access-m9bpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:49:59.921403 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:49:59.921314 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9bpv\" (UniqueName: \"kubernetes.io/projected/1773b46a-1dde-4944-b001-06cda4bedcd6-kube-api-access-m9bpv\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:50:00.294010 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:00.293921 5605 generic.go:356] "Generic (PLEG): container finished" podID="198ce753-de2f-471c-894f-872099dbef2c" containerID="c71b184be1d2af5d16fc1b1e2f50ddd0a22a9065dd23adaec7cb750ec7ec27aa" exitCode=0 Nov 03 02:50:00.294010 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:00.293965 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-e3cc05-6468f7548f-rl2hr" Nov 03 02:50:00.294259 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:00.294011 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-e3cc05-6468f7548f-rl2hr" event={"ID":"198ce753-de2f-471c-894f-872099dbef2c","Type":"ContainerDied","Data":"c71b184be1d2af5d16fc1b1e2f50ddd0a22a9065dd23adaec7cb750ec7ec27aa"} Nov 03 02:50:00.294259 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:00.294069 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-e3cc05-6468f7548f-rl2hr" event={"ID":"198ce753-de2f-471c-894f-872099dbef2c","Type":"ContainerDied","Data":"a0abe2771b7aaf48c7d77690f6f24f2255a8cec7eae5be80a152363577bf7439"} Nov 03 02:50:00.294259 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:00.294101 5605 scope.go:122] "RemoveContainer" containerID="c71b184be1d2af5d16fc1b1e2f50ddd0a22a9065dd23adaec7cb750ec7ec27aa" Nov 03 02:50:00.295507 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:00.295348 5605 generic.go:356] "Generic (PLEG): container finished" podID="1773b46a-1dde-4944-b001-06cda4bedcd6" containerID="e9cce2470453155af47146e17d4275fd9bdb8007f9f08d673eda0097d4f38597" exitCode=0 Nov 03 02:50:00.295507 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:00.295399 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e3cc05-66dfb8cc6c-8xgvh" event={"ID":"1773b46a-1dde-4944-b001-06cda4bedcd6","Type":"ContainerDied","Data":"e9cce2470453155af47146e17d4275fd9bdb8007f9f08d673eda0097d4f38597"} Nov 03 02:50:00.295507 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:00.295418 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e3cc05-66dfb8cc6c-8xgvh" Nov 03 02:50:00.295507 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:00.295449 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e3cc05-66dfb8cc6c-8xgvh" event={"ID":"1773b46a-1dde-4944-b001-06cda4bedcd6","Type":"ContainerDied","Data":"a3374ddbb15bc88717a66584c0b2b4f4b67f30aa5c60d473fbfe03ee85aadf42"} Nov 03 02:50:00.304643 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:00.304627 5605 scope.go:122] "RemoveContainer" containerID="c71b184be1d2af5d16fc1b1e2f50ddd0a22a9065dd23adaec7cb750ec7ec27aa" Nov 03 02:50:00.305004 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:50:00.304904 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c71b184be1d2af5d16fc1b1e2f50ddd0a22a9065dd23adaec7cb750ec7ec27aa\": container with ID starting with c71b184be1d2af5d16fc1b1e2f50ddd0a22a9065dd23adaec7cb750ec7ec27aa not found: ID does not exist" containerID="c71b184be1d2af5d16fc1b1e2f50ddd0a22a9065dd23adaec7cb750ec7ec27aa" Nov 03 02:50:00.305004 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:00.304929 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c71b184be1d2af5d16fc1b1e2f50ddd0a22a9065dd23adaec7cb750ec7ec27aa"} err="failed to get container status \"c71b184be1d2af5d16fc1b1e2f50ddd0a22a9065dd23adaec7cb750ec7ec27aa\": rpc error: code = NotFound desc = could not find container \"c71b184be1d2af5d16fc1b1e2f50ddd0a22a9065dd23adaec7cb750ec7ec27aa\": container with ID starting with c71b184be1d2af5d16fc1b1e2f50ddd0a22a9065dd23adaec7cb750ec7ec27aa not found: ID does not exist" Nov 03 02:50:00.305004 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:00.304949 5605 scope.go:122] "RemoveContainer" containerID="e9cce2470453155af47146e17d4275fd9bdb8007f9f08d673eda0097d4f38597" Nov 03 02:50:00.313801 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:00.313785 5605 scope.go:122] "RemoveContainer" containerID="e9cce2470453155af47146e17d4275fd9bdb8007f9f08d673eda0097d4f38597" Nov 03 02:50:00.314063 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:50:00.314046 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9cce2470453155af47146e17d4275fd9bdb8007f9f08d673eda0097d4f38597\": container with ID starting with e9cce2470453155af47146e17d4275fd9bdb8007f9f08d673eda0097d4f38597 not found: ID does not exist" containerID="e9cce2470453155af47146e17d4275fd9bdb8007f9f08d673eda0097d4f38597" Nov 03 02:50:00.314114 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:00.314069 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9cce2470453155af47146e17d4275fd9bdb8007f9f08d673eda0097d4f38597"} err="failed to get container status \"e9cce2470453155af47146e17d4275fd9bdb8007f9f08d673eda0097d4f38597\": rpc error: code = NotFound desc = could not find container \"e9cce2470453155af47146e17d4275fd9bdb8007f9f08d673eda0097d4f38597\": container with ID starting with e9cce2470453155af47146e17d4275fd9bdb8007f9f08d673eda0097d4f38597 not found: ID does not exist" Nov 03 02:50:00.447518 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:00.447489 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e3cc05-6468f7548f-rl2hr"] Nov 03 02:50:00.493764 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:00.493731 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e3cc05-6468f7548f-rl2hr"] Nov 03 02:50:00.552876 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:00.552850 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e3cc05-66dfb8cc6c-8xgvh"] Nov 03 02:50:00.555031 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:00.555011 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e3cc05-66dfb8cc6c-8xgvh"] Nov 03 02:50:00.850840 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:00.850768 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1773b46a-1dde-4944-b001-06cda4bedcd6" path="/var/lib/kubelet/pods/1773b46a-1dde-4944-b001-06cda4bedcd6/volumes" Nov 03 02:50:00.851150 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:00.851137 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="198ce753-de2f-471c-894f-872099dbef2c" path="/var/lib/kubelet/pods/198ce753-de2f-471c-894f-872099dbef2c/volumes" Nov 03 02:50:04.442581 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:04.442553 5605 scope.go:122] "RemoveContainer" containerID="de7fbd833b45d502e8b8fc19b1496e4b30c97e6c20c181575b55ba77470ed375" Nov 03 02:50:04.451444 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:04.451421 5605 scope.go:122] "RemoveContainer" containerID="0caa287f58f24ee33f5b9f9911f42b3010a3cfe015cd14d685a5c96548d5f67e" Nov 03 02:50:04.462768 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:04.462732 5605 scope.go:122] "RemoveContainer" containerID="8eee4ac8500805be420fec4a7351f3e6b0d33aaa6e0bf20971f5af9b44d5d5c1" Nov 03 02:50:22.883757 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:22.883723 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-038c21-7vqqx"] Nov 03 02:50:22.930239 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:22.930212 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-038c21-7vqqx"] Nov 03 02:50:23.041631 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:23.041598 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-876aed4a-66qz6"] Nov 03 02:50:23.043961 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:23.043938 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-876aed4a-66qz6"] Nov 03 02:50:23.152795 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:23.152712 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-038c21-openshift-default-5d5c88569c-4r995"] Nov 03 02:50:23.152962 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:23.152921 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-038c21-openshift-default-5d5c88569c-4r995" podUID="f7c23f7b-4a52-4a24-8fce-ba087104fca5" containerName="istio-proxy" containerID="cri-o://cd1a796693e719811c9ea87d3f42db5145e34c4a7353a3a61e0b2c2205843ec8" gracePeriod=30 Nov 03 02:50:23.917716 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:50:23.917677 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:50:24.850198 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:24.850167 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06e3ae22-d3dd-48e0-b9e1-19e75481f880" path="/var/lib/kubelet/pods/06e3ae22-d3dd-48e0-b9e1-19e75481f880/volumes" Nov 03 02:50:24.850614 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:24.850599 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a07adb9-2325-4532-97d0-00bdd2c8d6f0" path="/var/lib/kubelet/pods/6a07adb9-2325-4532-97d0-00bdd2c8d6f0/volumes" Nov 03 02:50:25.750331 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:25.750288 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r"] Nov 03 02:50:25.750838 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:25.750778 5605 cpu_manager.go:414] "containerMap: removing container" podUID="06e3ae22-d3dd-48e0-b9e1-19e75481f880" containerName="revoke-keys" Nov 03 02:50:25.750838 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:25.750791 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06e3ae22-d3dd-48e0-b9e1-19e75481f880" containerName="revoke-keys" Nov 03 02:50:25.750838 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:25.750808 5605 cpu_manager.go:414] "containerMap: removing container" podUID="198ce753-de2f-471c-894f-872099dbef2c" containerName="payload-processing" Nov 03 02:50:25.750838 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:25.750814 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="198ce753-de2f-471c-894f-872099dbef2c" containerName="payload-processing" Nov 03 02:50:25.750838 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:25.750824 5605 cpu_manager.go:414] "containerMap: removing container" podUID="1773b46a-1dde-4944-b001-06cda4bedcd6" containerName="payload-pre-processing" Nov 03 02:50:25.750838 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:25.750832 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1773b46a-1dde-4944-b001-06cda4bedcd6" containerName="payload-pre-processing" Nov 03 02:50:25.751173 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:25.750853 5605 cpu_manager.go:414] "containerMap: removing container" podUID="db0c38c1-8105-49de-9561-319a632581bd" containerName="main" Nov 03 02:50:25.751173 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:25.750858 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db0c38c1-8105-49de-9561-319a632581bd" containerName="main" Nov 03 02:50:25.751173 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:25.750932 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="db0c38c1-8105-49de-9561-319a632581bd" containerName="main" Nov 03 02:50:25.751173 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:25.750943 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="06e3ae22-d3dd-48e0-b9e1-19e75481f880" containerName="revoke-keys" Nov 03 02:50:25.751173 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:25.750950 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="198ce753-de2f-471c-894f-872099dbef2c" containerName="payload-processing" Nov 03 02:50:25.751173 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:25.750958 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="1773b46a-1dde-4944-b001-06cda4bedcd6" containerName="payload-pre-processing" Nov 03 02:50:25.755753 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:25.755728 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r" Nov 03 02:50:25.801890 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:25.801859 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53640730-1899-46e3-95de-b5a502efbd89-credential-socket\") pod \"e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r\" (UID: \"53640730-1899-46e3-95de-b5a502efbd89\") " pod="openshift-ingress/e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r" Nov 03 02:50:25.802060 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:25.801904 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53640730-1899-46e3-95de-b5a502efbd89-istiod-ca-cert\") pod \"e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r\" (UID: \"53640730-1899-46e3-95de-b5a502efbd89\") " pod="openshift-ingress/e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r" Nov 03 02:50:25.802060 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:25.801988 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53640730-1899-46e3-95de-b5a502efbd89-istio-podinfo\") pod \"e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r\" (UID: \"53640730-1899-46e3-95de-b5a502efbd89\") " pod="openshift-ingress/e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r" Nov 03 02:50:25.802162 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:25.802125 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53640730-1899-46e3-95de-b5a502efbd89-istio-token\") pod \"e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r\" (UID: \"53640730-1899-46e3-95de-b5a502efbd89\") " pod="openshift-ingress/e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r" Nov 03 02:50:25.802252 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:25.802235 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53640730-1899-46e3-95de-b5a502efbd89-workload-certs\") pod \"e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r\" (UID: \"53640730-1899-46e3-95de-b5a502efbd89\") " pod="openshift-ingress/e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r" Nov 03 02:50:25.802315 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:25.802290 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xmhj\" (UniqueName: \"kubernetes.io/projected/53640730-1899-46e3-95de-b5a502efbd89-kube-api-access-6xmhj\") pod \"e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r\" (UID: \"53640730-1899-46e3-95de-b5a502efbd89\") " pod="openshift-ingress/e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r" Nov 03 02:50:25.802419 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:25.802404 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53640730-1899-46e3-95de-b5a502efbd89-istio-envoy\") pod \"e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r\" (UID: \"53640730-1899-46e3-95de-b5a502efbd89\") " pod="openshift-ingress/e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r" Nov 03 02:50:25.802471 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:25.802436 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53640730-1899-46e3-95de-b5a502efbd89-workload-socket\") pod \"e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r\" (UID: \"53640730-1899-46e3-95de-b5a502efbd89\") " pod="openshift-ingress/e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r" Nov 03 02:50:25.802471 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:25.802462 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53640730-1899-46e3-95de-b5a502efbd89-istio-data\") pod \"e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r\" (UID: \"53640730-1899-46e3-95de-b5a502efbd89\") " pod="openshift-ingress/e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r" Nov 03 02:50:25.824435 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:25.824409 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-e039cb-openshift-default-dockercfg-nzvp6\"" Nov 03 02:50:25.854651 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:25.854621 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r"] Nov 03 02:50:25.903805 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:25.903769 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53640730-1899-46e3-95de-b5a502efbd89-istio-token\") pod \"e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r\" (UID: \"53640730-1899-46e3-95de-b5a502efbd89\") " pod="openshift-ingress/e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r" Nov 03 02:50:25.904138 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:25.904116 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53640730-1899-46e3-95de-b5a502efbd89-workload-certs\") pod \"e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r\" (UID: \"53640730-1899-46e3-95de-b5a502efbd89\") " pod="openshift-ingress/e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r" Nov 03 02:50:25.905117 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:25.905092 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xmhj\" (UniqueName: \"kubernetes.io/projected/53640730-1899-46e3-95de-b5a502efbd89-kube-api-access-6xmhj\") pod \"e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r\" (UID: \"53640730-1899-46e3-95de-b5a502efbd89\") " pod="openshift-ingress/e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r" Nov 03 02:50:25.905752 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:25.905729 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53640730-1899-46e3-95de-b5a502efbd89-workload-certs\") pod \"e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r\" (UID: \"53640730-1899-46e3-95de-b5a502efbd89\") " pod="openshift-ingress/e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r" Nov 03 02:50:25.905931 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:25.905911 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53640730-1899-46e3-95de-b5a502efbd89-istio-envoy\") pod \"e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r\" (UID: \"53640730-1899-46e3-95de-b5a502efbd89\") " pod="openshift-ingress/e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r" Nov 03 02:50:25.906062 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:25.906045 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53640730-1899-46e3-95de-b5a502efbd89-workload-socket\") pod \"e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r\" (UID: \"53640730-1899-46e3-95de-b5a502efbd89\") " pod="openshift-ingress/e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r" Nov 03 02:50:25.906181 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:25.906166 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53640730-1899-46e3-95de-b5a502efbd89-istio-data\") pod \"e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r\" (UID: \"53640730-1899-46e3-95de-b5a502efbd89\") " pod="openshift-ingress/e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r" Nov 03 02:50:25.906420 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:25.906397 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53640730-1899-46e3-95de-b5a502efbd89-credential-socket\") pod \"e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r\" (UID: \"53640730-1899-46e3-95de-b5a502efbd89\") " pod="openshift-ingress/e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r" Nov 03 02:50:25.906553 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:25.906536 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53640730-1899-46e3-95de-b5a502efbd89-istiod-ca-cert\") pod \"e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r\" (UID: \"53640730-1899-46e3-95de-b5a502efbd89\") " pod="openshift-ingress/e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r" Nov 03 02:50:25.906696 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:25.906679 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53640730-1899-46e3-95de-b5a502efbd89-istio-podinfo\") pod \"e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r\" (UID: \"53640730-1899-46e3-95de-b5a502efbd89\") " pod="openshift-ingress/e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r" Nov 03 02:50:25.909169 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:25.909145 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53640730-1899-46e3-95de-b5a502efbd89-workload-socket\") pod \"e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r\" (UID: \"53640730-1899-46e3-95de-b5a502efbd89\") " pod="openshift-ingress/e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r" Nov 03 02:50:25.909603 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:25.909581 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53640730-1899-46e3-95de-b5a502efbd89-istio-data\") pod \"e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r\" (UID: \"53640730-1899-46e3-95de-b5a502efbd89\") " pod="openshift-ingress/e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r" Nov 03 02:50:25.917294 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:25.910773 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53640730-1899-46e3-95de-b5a502efbd89-istio-podinfo\") pod \"e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r\" (UID: \"53640730-1899-46e3-95de-b5a502efbd89\") " pod="openshift-ingress/e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r" Nov 03 02:50:25.917294 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:25.911524 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53640730-1899-46e3-95de-b5a502efbd89-credential-socket\") pod \"e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r\" (UID: \"53640730-1899-46e3-95de-b5a502efbd89\") " pod="openshift-ingress/e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r" Nov 03 02:50:25.917294 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:25.912099 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53640730-1899-46e3-95de-b5a502efbd89-istiod-ca-cert\") pod \"e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r\" (UID: \"53640730-1899-46e3-95de-b5a502efbd89\") " pod="openshift-ingress/e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r" Nov 03 02:50:25.959652 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:25.959191 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53640730-1899-46e3-95de-b5a502efbd89-istio-envoy\") pod \"e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r\" (UID: \"53640730-1899-46e3-95de-b5a502efbd89\") " pod="openshift-ingress/e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r" Nov 03 02:50:25.994218 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:25.994185 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53640730-1899-46e3-95de-b5a502efbd89-istio-token\") pod \"e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r\" (UID: \"53640730-1899-46e3-95de-b5a502efbd89\") " pod="openshift-ingress/e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r" Nov 03 02:50:26.008979 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:26.008895 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xmhj\" (UniqueName: \"kubernetes.io/projected/53640730-1899-46e3-95de-b5a502efbd89-kube-api-access-6xmhj\") pod \"e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r\" (UID: \"53640730-1899-46e3-95de-b5a502efbd89\") " pod="openshift-ingress/e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r" Nov 03 02:50:26.069741 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:26.069711 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r" Nov 03 02:50:26.493814 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:26.493781 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r"] Nov 03 02:50:26.499190 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:50:26.499162 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53640730_1899_46e3_95de_b5a502efbd89.slice/crio-6952053cc83242f3c820ec180b75df0848429b4da7a6b93b7e4b898d72a3658f WatchSource:0}: Error finding container 6952053cc83242f3c820ec180b75df0848429b4da7a6b93b7e4b898d72a3658f: Status 404 returned error can't find the container with id 6952053cc83242f3c820ec180b75df0848429b4da7a6b93b7e4b898d72a3658f Nov 03 02:50:26.501702 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:26.501658 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:50:26.501805 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:26.501736 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:50:26.501805 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:26.501784 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:50:27.475997 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:27.475960 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r" event={"ID":"53640730-1899-46e3-95de-b5a502efbd89","Type":"ContainerStarted","Data":"0f00a946e3edd3ee50f79cac07a83974f97e755c7cd8aaaf9ac5f719a06c3964"} Nov 03 02:50:27.476412 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:27.476023 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r" event={"ID":"53640730-1899-46e3-95de-b5a502efbd89","Type":"ContainerStarted","Data":"6952053cc83242f3c820ec180b75df0848429b4da7a6b93b7e4b898d72a3658f"} Nov 03 02:50:27.631928 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:27.631872 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r" podStartSLOduration=2.631857285 podStartE2EDuration="2.631857285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:50:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:50:27.630460401 +0000 UTC m=+3504.889658568" watchObservedRunningTime="2026-11-03 02:50:27.631857285 +0000 UTC m=+3504.891055504" Nov 03 02:50:28.069973 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:28.069943 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r" Nov 03 02:50:28.075662 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:28.075635 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r" Nov 03 02:50:28.419945 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:28.419922 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-038c21-openshift-default-5d5c88569c-4r995" Nov 03 02:50:28.483439 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:28.483412 5605 generic.go:356] "Generic (PLEG): container finished" podID="f7c23f7b-4a52-4a24-8fce-ba087104fca5" containerID="cd1a796693e719811c9ea87d3f42db5145e34c4a7353a3a61e0b2c2205843ec8" exitCode=0 Nov 03 02:50:28.483762 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:28.483463 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-038c21-openshift-default-5d5c88569c-4r995" Nov 03 02:50:28.483762 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:28.483500 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-038c21-openshift-default-5d5c88569c-4r995" event={"ID":"f7c23f7b-4a52-4a24-8fce-ba087104fca5","Type":"ContainerDied","Data":"cd1a796693e719811c9ea87d3f42db5145e34c4a7353a3a61e0b2c2205843ec8"} Nov 03 02:50:28.483762 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:28.483556 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-038c21-openshift-default-5d5c88569c-4r995" event={"ID":"f7c23f7b-4a52-4a24-8fce-ba087104fca5","Type":"ContainerDied","Data":"7c71323da9150db2af796be58db56f0b9a80c234f81b37f761a474201e2d9beb"} Nov 03 02:50:28.483762 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:28.483596 5605 scope.go:122] "RemoveContainer" containerID="cd1a796693e719811c9ea87d3f42db5145e34c4a7353a3a61e0b2c2205843ec8" Nov 03 02:50:28.483762 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:28.483715 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r" Nov 03 02:50:28.484745 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:28.484718 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r" Nov 03 02:50:28.492793 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:28.492774 5605 scope.go:122] "RemoveContainer" containerID="ea5fea9d3bd0b499ada713f2daec3443f2cf147eb3aabdc778cc85c2e515ae44" Nov 03 02:50:28.501219 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:28.501197 5605 scope.go:122] "RemoveContainer" containerID="cd1a796693e719811c9ea87d3f42db5145e34c4a7353a3a61e0b2c2205843ec8" Nov 03 02:50:28.501506 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:50:28.501483 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd1a796693e719811c9ea87d3f42db5145e34c4a7353a3a61e0b2c2205843ec8\": container with ID starting with cd1a796693e719811c9ea87d3f42db5145e34c4a7353a3a61e0b2c2205843ec8 not found: ID does not exist" containerID="cd1a796693e719811c9ea87d3f42db5145e34c4a7353a3a61e0b2c2205843ec8" Nov 03 02:50:28.501576 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:28.501511 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd1a796693e719811c9ea87d3f42db5145e34c4a7353a3a61e0b2c2205843ec8"} err="failed to get container status \"cd1a796693e719811c9ea87d3f42db5145e34c4a7353a3a61e0b2c2205843ec8\": rpc error: code = NotFound desc = could not find container \"cd1a796693e719811c9ea87d3f42db5145e34c4a7353a3a61e0b2c2205843ec8\": container with ID starting with cd1a796693e719811c9ea87d3f42db5145e34c4a7353a3a61e0b2c2205843ec8 not found: ID does not exist" Nov 03 02:50:28.501576 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:28.501529 5605 scope.go:122] "RemoveContainer" containerID="ea5fea9d3bd0b499ada713f2daec3443f2cf147eb3aabdc778cc85c2e515ae44" Nov 03 02:50:28.501758 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:50:28.501739 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea5fea9d3bd0b499ada713f2daec3443f2cf147eb3aabdc778cc85c2e515ae44\": container with ID starting with ea5fea9d3bd0b499ada713f2daec3443f2cf147eb3aabdc778cc85c2e515ae44 not found: ID does not exist" containerID="ea5fea9d3bd0b499ada713f2daec3443f2cf147eb3aabdc778cc85c2e515ae44" Nov 03 02:50:28.501805 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:28.501764 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea5fea9d3bd0b499ada713f2daec3443f2cf147eb3aabdc778cc85c2e515ae44"} err="failed to get container status \"ea5fea9d3bd0b499ada713f2daec3443f2cf147eb3aabdc778cc85c2e515ae44\": rpc error: code = NotFound desc = could not find container \"ea5fea9d3bd0b499ada713f2daec3443f2cf147eb3aabdc778cc85c2e515ae44\": container with ID starting with ea5fea9d3bd0b499ada713f2daec3443f2cf147eb3aabdc778cc85c2e515ae44 not found: ID does not exist" Nov 03 02:50:28.537653 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:28.537622 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7c23f7b-4a52-4a24-8fce-ba087104fca5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7c23f7b-4a52-4a24-8fce-ba087104fca5-workload-socket\") pod \"f7c23f7b-4a52-4a24-8fce-ba087104fca5\" (UID: \"f7c23f7b-4a52-4a24-8fce-ba087104fca5\") " Nov 03 02:50:28.537788 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:28.537683 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7c23f7b-4a52-4a24-8fce-ba087104fca5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7c23f7b-4a52-4a24-8fce-ba087104fca5-istio-envoy\") pod \"f7c23f7b-4a52-4a24-8fce-ba087104fca5\" (UID: \"f7c23f7b-4a52-4a24-8fce-ba087104fca5\") " Nov 03 02:50:28.537788 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:28.537738 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7c23f7b-4a52-4a24-8fce-ba087104fca5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7c23f7b-4a52-4a24-8fce-ba087104fca5-credential-socket\") pod \"f7c23f7b-4a52-4a24-8fce-ba087104fca5\" (UID: \"f7c23f7b-4a52-4a24-8fce-ba087104fca5\") " Nov 03 02:50:28.537788 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:28.537775 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7c23f7b-4a52-4a24-8fce-ba087104fca5-istio-token\" (UniqueName: \"kubernetes.io/projected/f7c23f7b-4a52-4a24-8fce-ba087104fca5-istio-token\") pod \"f7c23f7b-4a52-4a24-8fce-ba087104fca5\" (UID: \"f7c23f7b-4a52-4a24-8fce-ba087104fca5\") " Nov 03 02:50:28.537954 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:28.537827 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7c23f7b-4a52-4a24-8fce-ba087104fca5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7c23f7b-4a52-4a24-8fce-ba087104fca5-istio-data\") pod \"f7c23f7b-4a52-4a24-8fce-ba087104fca5\" (UID: \"f7c23f7b-4a52-4a24-8fce-ba087104fca5\") " Nov 03 02:50:28.537954 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:28.537934 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7c23f7b-4a52-4a24-8fce-ba087104fca5-workload-socket" pod "f7c23f7b-4a52-4a24-8fce-ba087104fca5" (UID: "f7c23f7b-4a52-4a24-8fce-ba087104fca5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:50:28.538066 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:28.538037 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7c23f7b-4a52-4a24-8fce-ba087104fca5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7c23f7b-4a52-4a24-8fce-ba087104fca5-workload-certs\") pod \"f7c23f7b-4a52-4a24-8fce-ba087104fca5\" (UID: \"f7c23f7b-4a52-4a24-8fce-ba087104fca5\") " Nov 03 02:50:28.538123 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:28.538064 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7c23f7b-4a52-4a24-8fce-ba087104fca5-istio-data" pod "f7c23f7b-4a52-4a24-8fce-ba087104fca5" (UID: "f7c23f7b-4a52-4a24-8fce-ba087104fca5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:50:28.538191 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:28.538117 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7c23f7b-4a52-4a24-8fce-ba087104fca5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7c23f7b-4a52-4a24-8fce-ba087104fca5-istiod-ca-cert\") pod \"f7c23f7b-4a52-4a24-8fce-ba087104fca5\" (UID: \"f7c23f7b-4a52-4a24-8fce-ba087104fca5\") " Nov 03 02:50:28.538245 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:28.538208 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7c23f7b-4a52-4a24-8fce-ba087104fca5-kube-api-access-djn5v\" (UniqueName: \"kubernetes.io/projected/f7c23f7b-4a52-4a24-8fce-ba087104fca5-kube-api-access-djn5v\") pod \"f7c23f7b-4a52-4a24-8fce-ba087104fca5\" (UID: \"f7c23f7b-4a52-4a24-8fce-ba087104fca5\") " Nov 03 02:50:28.538337 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:28.538319 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f7c23f7b-4a52-4a24-8fce-ba087104fca5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7c23f7b-4a52-4a24-8fce-ba087104fca5-istio-podinfo\") pod \"f7c23f7b-4a52-4a24-8fce-ba087104fca5\" (UID: \"f7c23f7b-4a52-4a24-8fce-ba087104fca5\") " Nov 03 02:50:28.538420 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:28.538335 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7c23f7b-4a52-4a24-8fce-ba087104fca5-credential-socket" pod "f7c23f7b-4a52-4a24-8fce-ba087104fca5" (UID: "f7c23f7b-4a52-4a24-8fce-ba087104fca5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:50:28.538565 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:28.538541 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7c23f7b-4a52-4a24-8fce-ba087104fca5-istiod-ca-cert" pod "f7c23f7b-4a52-4a24-8fce-ba087104fca5" (UID: "f7c23f7b-4a52-4a24-8fce-ba087104fca5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:50:28.538892 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:28.538862 5605 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7c23f7b-4a52-4a24-8fce-ba087104fca5-credential-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:50:28.538892 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:28.538887 5605 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7c23f7b-4a52-4a24-8fce-ba087104fca5-istio-data\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:50:28.539038 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:28.538902 5605 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7c23f7b-4a52-4a24-8fce-ba087104fca5-istiod-ca-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:50:28.539038 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:28.538916 5605 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7c23f7b-4a52-4a24-8fce-ba087104fca5-workload-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:50:28.539154 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:28.539110 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7c23f7b-4a52-4a24-8fce-ba087104fca5-workload-certs" pod "f7c23f7b-4a52-4a24-8fce-ba087104fca5" (UID: "f7c23f7b-4a52-4a24-8fce-ba087104fca5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:50:28.541036 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:28.541008 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f7c23f7b-4a52-4a24-8fce-ba087104fca5-istio-podinfo" pod "f7c23f7b-4a52-4a24-8fce-ba087104fca5" (UID: "f7c23f7b-4a52-4a24-8fce-ba087104fca5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:50:28.541705 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:28.541677 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7c23f7b-4a52-4a24-8fce-ba087104fca5-istio-token" pod "f7c23f7b-4a52-4a24-8fce-ba087104fca5" (UID: "f7c23f7b-4a52-4a24-8fce-ba087104fca5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:50:28.541705 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:28.541684 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7c23f7b-4a52-4a24-8fce-ba087104fca5-istio-envoy" pod "f7c23f7b-4a52-4a24-8fce-ba087104fca5" (UID: "f7c23f7b-4a52-4a24-8fce-ba087104fca5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:50:28.541886 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:28.541862 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7c23f7b-4a52-4a24-8fce-ba087104fca5-kube-api-access-djn5v" pod "f7c23f7b-4a52-4a24-8fce-ba087104fca5" (UID: "f7c23f7b-4a52-4a24-8fce-ba087104fca5"). InnerVolumeSpecName "kube-api-access-djn5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:50:28.639757 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:28.639688 5605 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7c23f7b-4a52-4a24-8fce-ba087104fca5-istio-envoy\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:50:28.639757 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:28.639714 5605 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7c23f7b-4a52-4a24-8fce-ba087104fca5-istio-token\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:50:28.639757 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:28.639724 5605 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7c23f7b-4a52-4a24-8fce-ba087104fca5-workload-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:50:28.639757 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:28.639734 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djn5v\" (UniqueName: \"kubernetes.io/projected/f7c23f7b-4a52-4a24-8fce-ba087104fca5-kube-api-access-djn5v\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:50:28.639757 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:28.639743 5605 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7c23f7b-4a52-4a24-8fce-ba087104fca5-istio-podinfo\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:50:28.899233 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:28.899130 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-038c21-openshift-default-5d5c88569c-4r995"] Nov 03 02:50:28.901155 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:28.901131 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-038c21-openshift-default-5d5c88569c-4r995"] Nov 03 02:50:30.855906 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:30.855874 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7c23f7b-4a52-4a24-8fce-ba087104fca5" path="/var/lib/kubelet/pods/f7c23f7b-4a52-4a24-8fce-ba087104fca5/volumes" Nov 03 02:50:33.044214 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:33.044181 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d2c4664-tjjzg"] Nov 03 02:50:33.090457 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:33.090411 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d2c4664-tjjzg"] Nov 03 02:50:34.597653 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:34.597619 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e039cb-6f47f9bfd7-9zqfn"] Nov 03 02:50:34.598199 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:34.598181 5605 cpu_manager.go:414] "containerMap: removing container" podUID="f7c23f7b-4a52-4a24-8fce-ba087104fca5" containerName="istio-proxy" Nov 03 02:50:34.598262 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:34.598201 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7c23f7b-4a52-4a24-8fce-ba087104fca5" containerName="istio-proxy" Nov 03 02:50:34.598262 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:34.598222 5605 cpu_manager.go:414] "containerMap: removing container" podUID="f7c23f7b-4a52-4a24-8fce-ba087104fca5" containerName="istio-proxy" Nov 03 02:50:34.598262 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:34.598229 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7c23f7b-4a52-4a24-8fce-ba087104fca5" containerName="istio-proxy" Nov 03 02:50:34.598423 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:34.598336 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7c23f7b-4a52-4a24-8fce-ba087104fca5" containerName="istio-proxy" Nov 03 02:50:34.598423 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:34.598349 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7c23f7b-4a52-4a24-8fce-ba087104fca5" containerName="istio-proxy" Nov 03 02:50:34.601764 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:34.601739 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-e039cb-6f47f9bfd7-9zqfn" Nov 03 02:50:34.654415 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:34.654392 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-e039cb\"" Nov 03 02:50:34.699342 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:34.699316 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-e039cb-dockercfg-2cv5n\"" Nov 03 02:50:34.701663 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:34.701632 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e039cb-6f47f9bfd7-9zqfn"] Nov 03 02:50:34.702127 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:34.702105 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vxv8\" (UniqueName: \"kubernetes.io/projected/080ef559-4fcc-46ed-83d2-d9fa1e168792-kube-api-access-2vxv8\") pod \"payload-processing-e2e-extmod-a-e039cb-6f47f9bfd7-9zqfn\" (UID: \"080ef559-4fcc-46ed-83d2-d9fa1e168792\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e039cb-6f47f9bfd7-9zqfn" Nov 03 02:50:34.702239 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:34.702224 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/080ef559-4fcc-46ed-83d2-d9fa1e168792-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-e039cb-6f47f9bfd7-9zqfn\" (UID: \"080ef559-4fcc-46ed-83d2-d9fa1e168792\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e039cb-6f47f9bfd7-9zqfn" Nov 03 02:50:34.703489 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:34.703470 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e039cb-5bbb449d87-t8dnw"] Nov 03 02:50:34.707054 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:34.707036 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e039cb-5bbb449d87-t8dnw" Nov 03 02:50:34.770190 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:34.770158 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e039cb-5bbb449d87-t8dnw"] Nov 03 02:50:34.803110 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:34.803084 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vxv8\" (UniqueName: \"kubernetes.io/projected/080ef559-4fcc-46ed-83d2-d9fa1e168792-kube-api-access-2vxv8\") pod \"payload-processing-e2e-extmod-a-e039cb-6f47f9bfd7-9zqfn\" (UID: \"080ef559-4fcc-46ed-83d2-d9fa1e168792\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e039cb-6f47f9bfd7-9zqfn" Nov 03 02:50:34.803239 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:34.803172 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/080ef559-4fcc-46ed-83d2-d9fa1e168792-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-e039cb-6f47f9bfd7-9zqfn\" (UID: \"080ef559-4fcc-46ed-83d2-d9fa1e168792\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e039cb-6f47f9bfd7-9zqfn" Nov 03 02:50:34.803239 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:34.803226 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cec8117c-77b5-4cf8-8c6c-f3e5bffbf72b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-e039cb-5bbb449d87-t8dnw\" (UID: \"cec8117c-77b5-4cf8-8c6c-f3e5bffbf72b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e039cb-5bbb449d87-t8dnw" Nov 03 02:50:34.803368 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:34.803250 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpkqr\" (UniqueName: \"kubernetes.io/projected/cec8117c-77b5-4cf8-8c6c-f3e5bffbf72b-kube-api-access-fpkqr\") pod \"payload-pre-processing-e2e-extmod-a-e039cb-5bbb449d87-t8dnw\" (UID: \"cec8117c-77b5-4cf8-8c6c-f3e5bffbf72b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e039cb-5bbb449d87-t8dnw" Nov 03 02:50:34.804161 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:34.804132 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/080ef559-4fcc-46ed-83d2-d9fa1e168792-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-e039cb-6f47f9bfd7-9zqfn\" (UID: \"080ef559-4fcc-46ed-83d2-d9fa1e168792\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e039cb-6f47f9bfd7-9zqfn" Nov 03 02:50:34.850380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:34.850313 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfa077e8-2701-4b9c-a30a-cdc528a66a11" path="/var/lib/kubelet/pods/dfa077e8-2701-4b9c-a30a-cdc528a66a11/volumes" Nov 03 02:50:34.856646 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:34.856620 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vxv8\" (UniqueName: \"kubernetes.io/projected/080ef559-4fcc-46ed-83d2-d9fa1e168792-kube-api-access-2vxv8\") pod \"payload-processing-e2e-extmod-a-e039cb-6f47f9bfd7-9zqfn\" (UID: \"080ef559-4fcc-46ed-83d2-d9fa1e168792\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e039cb-6f47f9bfd7-9zqfn" Nov 03 02:50:34.904044 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:34.904012 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cec8117c-77b5-4cf8-8c6c-f3e5bffbf72b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-e039cb-5bbb449d87-t8dnw\" (UID: \"cec8117c-77b5-4cf8-8c6c-f3e5bffbf72b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e039cb-5bbb449d87-t8dnw" Nov 03 02:50:34.904044 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:34.904039 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpkqr\" (UniqueName: \"kubernetes.io/projected/cec8117c-77b5-4cf8-8c6c-f3e5bffbf72b-kube-api-access-fpkqr\") pod \"payload-pre-processing-e2e-extmod-a-e039cb-5bbb449d87-t8dnw\" (UID: \"cec8117c-77b5-4cf8-8c6c-f3e5bffbf72b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e039cb-5bbb449d87-t8dnw" Nov 03 02:50:34.904783 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:34.904765 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cec8117c-77b5-4cf8-8c6c-f3e5bffbf72b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-e039cb-5bbb449d87-t8dnw\" (UID: \"cec8117c-77b5-4cf8-8c6c-f3e5bffbf72b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e039cb-5bbb449d87-t8dnw" Nov 03 02:50:34.915678 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:34.915661 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-e039cb-6f47f9bfd7-9zqfn" Nov 03 02:50:34.957882 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:34.957849 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpkqr\" (UniqueName: \"kubernetes.io/projected/cec8117c-77b5-4cf8-8c6c-f3e5bffbf72b-kube-api-access-fpkqr\") pod \"payload-pre-processing-e2e-extmod-a-e039cb-5bbb449d87-t8dnw\" (UID: \"cec8117c-77b5-4cf8-8c6c-f3e5bffbf72b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e039cb-5bbb449d87-t8dnw" Nov 03 02:50:35.019837 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:35.019806 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e039cb-5bbb449d87-t8dnw" Nov 03 02:50:35.283244 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:35.283162 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e039cb-6f47f9bfd7-9zqfn"] Nov 03 02:50:35.289229 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:50:35.289202 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod080ef559_4fcc_46ed_83d2_d9fa1e168792.slice/crio-04564c6a8d3d80e3f7075ea61a1c7d1e6aa6d2eb5cd0e804d1712050e5be8c68 WatchSource:0}: Error finding container 04564c6a8d3d80e3f7075ea61a1c7d1e6aa6d2eb5cd0e804d1712050e5be8c68: Status 404 returned error can't find the container with id 04564c6a8d3d80e3f7075ea61a1c7d1e6aa6d2eb5cd0e804d1712050e5be8c68 Nov 03 02:50:35.394795 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:35.394758 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e039cb-5bbb449d87-t8dnw"] Nov 03 02:50:35.400833 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:50:35.400800 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcec8117c_77b5_4cf8_8c6c_f3e5bffbf72b.slice/crio-f8f5e394fe2a88544b719275c192a163e9546cbec1556a7423eac04dc6bd5d6a WatchSource:0}: Error finding container f8f5e394fe2a88544b719275c192a163e9546cbec1556a7423eac04dc6bd5d6a: Status 404 returned error can't find the container with id f8f5e394fe2a88544b719275c192a163e9546cbec1556a7423eac04dc6bd5d6a Nov 03 02:50:35.524976 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:35.524948 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e039cb-5bbb449d87-t8dnw" event={"ID":"cec8117c-77b5-4cf8-8c6c-f3e5bffbf72b","Type":"ContainerStarted","Data":"936cbadc551e0d220087f8d1d86bffd3ac2ae1e68c249f684c25217c5f604707"} Nov 03 02:50:35.525109 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:35.525001 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e039cb-5bbb449d87-t8dnw" event={"ID":"cec8117c-77b5-4cf8-8c6c-f3e5bffbf72b","Type":"ContainerStarted","Data":"f8f5e394fe2a88544b719275c192a163e9546cbec1556a7423eac04dc6bd5d6a"} Nov 03 02:50:35.525176 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:35.525112 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e039cb-5bbb449d87-t8dnw" Nov 03 02:50:35.526658 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:35.526633 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-e039cb-6f47f9bfd7-9zqfn" event={"ID":"080ef559-4fcc-46ed-83d2-d9fa1e168792","Type":"ContainerStarted","Data":"052fb46f0b7a12f70c595a298bb2633a1fba76400d37da8cb2f21b00374d6e82"} Nov 03 02:50:35.526769 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:35.526673 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-e039cb-6f47f9bfd7-9zqfn" event={"ID":"080ef559-4fcc-46ed-83d2-d9fa1e168792","Type":"ContainerStarted","Data":"04564c6a8d3d80e3f7075ea61a1c7d1e6aa6d2eb5cd0e804d1712050e5be8c68"} Nov 03 02:50:35.526769 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:35.526691 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-e039cb-6f47f9bfd7-9zqfn" Nov 03 02:50:35.632960 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:35.632913 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e039cb-5bbb449d87-t8dnw" podStartSLOduration=1.632901849 podStartE2EDuration="1.632901849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:50:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:50:35.631330476 +0000 UTC m=+3512.890528631" watchObservedRunningTime="2026-11-03 02:50:35.632901849 +0000 UTC m=+3512.892100002" Nov 03 02:50:35.738128 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:35.738080 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-e039cb-6f47f9bfd7-9zqfn" podStartSLOduration=1.738068491 podStartE2EDuration="1.738068491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:50:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:50:35.737449997 +0000 UTC m=+3512.996648151" watchObservedRunningTime="2026-11-03 02:50:35.738068491 +0000 UTC m=+3512.997266635" Nov 03 02:50:36.543035 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:36.543006 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:50:46.536736 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:46.536661 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-e039cb-6f47f9bfd7-9zqfn" Nov 03 02:50:46.536736 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:46.536732 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e039cb-5bbb449d87-t8dnw" Nov 03 02:50:46.989038 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:46.989004 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2"] Nov 03 02:50:46.989309 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:46.989263 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2" podUID="955ee3cb-72bd-4dab-91ce-faffcbbe130d" containerName="istio-proxy" containerID="cri-o://590738e66fb753edb2f7efa9b29a281759b31c9409800d0cec384d6e43f077cf" gracePeriod=30 Nov 03 02:50:52.246448 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:52.246417 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2" Nov 03 02:50:52.306924 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:52.306888 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/955ee3cb-72bd-4dab-91ce-faffcbbe130d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/955ee3cb-72bd-4dab-91ce-faffcbbe130d-workload-certs\") pod \"955ee3cb-72bd-4dab-91ce-faffcbbe130d\" (UID: \"955ee3cb-72bd-4dab-91ce-faffcbbe130d\") " Nov 03 02:50:52.307107 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:52.306973 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/955ee3cb-72bd-4dab-91ce-faffcbbe130d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/955ee3cb-72bd-4dab-91ce-faffcbbe130d-credential-socket\") pod \"955ee3cb-72bd-4dab-91ce-faffcbbe130d\" (UID: \"955ee3cb-72bd-4dab-91ce-faffcbbe130d\") " Nov 03 02:50:52.307107 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:52.307035 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/955ee3cb-72bd-4dab-91ce-faffcbbe130d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/955ee3cb-72bd-4dab-91ce-faffcbbe130d-istio-podinfo\") pod \"955ee3cb-72bd-4dab-91ce-faffcbbe130d\" (UID: \"955ee3cb-72bd-4dab-91ce-faffcbbe130d\") " Nov 03 02:50:52.307107 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:52.307069 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/955ee3cb-72bd-4dab-91ce-faffcbbe130d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/955ee3cb-72bd-4dab-91ce-faffcbbe130d-istiod-ca-cert\") pod \"955ee3cb-72bd-4dab-91ce-faffcbbe130d\" (UID: \"955ee3cb-72bd-4dab-91ce-faffcbbe130d\") " Nov 03 02:50:52.307107 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:52.307103 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/955ee3cb-72bd-4dab-91ce-faffcbbe130d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/955ee3cb-72bd-4dab-91ce-faffcbbe130d-istio-data\") pod \"955ee3cb-72bd-4dab-91ce-faffcbbe130d\" (UID: \"955ee3cb-72bd-4dab-91ce-faffcbbe130d\") " Nov 03 02:50:52.307321 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:52.307122 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/955ee3cb-72bd-4dab-91ce-faffcbbe130d-istio-token\" (UniqueName: \"kubernetes.io/projected/955ee3cb-72bd-4dab-91ce-faffcbbe130d-istio-token\") pod \"955ee3cb-72bd-4dab-91ce-faffcbbe130d\" (UID: \"955ee3cb-72bd-4dab-91ce-faffcbbe130d\") " Nov 03 02:50:52.307321 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:52.307165 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/955ee3cb-72bd-4dab-91ce-faffcbbe130d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/955ee3cb-72bd-4dab-91ce-faffcbbe130d-istio-envoy\") pod \"955ee3cb-72bd-4dab-91ce-faffcbbe130d\" (UID: \"955ee3cb-72bd-4dab-91ce-faffcbbe130d\") " Nov 03 02:50:52.307321 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:52.307195 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/955ee3cb-72bd-4dab-91ce-faffcbbe130d-kube-api-access-6k6p7\" (UniqueName: \"kubernetes.io/projected/955ee3cb-72bd-4dab-91ce-faffcbbe130d-kube-api-access-6k6p7\") pod \"955ee3cb-72bd-4dab-91ce-faffcbbe130d\" (UID: \"955ee3cb-72bd-4dab-91ce-faffcbbe130d\") " Nov 03 02:50:52.307321 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:52.307218 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/955ee3cb-72bd-4dab-91ce-faffcbbe130d-workload-certs" pod "955ee3cb-72bd-4dab-91ce-faffcbbe130d" (UID: "955ee3cb-72bd-4dab-91ce-faffcbbe130d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:50:52.307321 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:52.307273 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/955ee3cb-72bd-4dab-91ce-faffcbbe130d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/955ee3cb-72bd-4dab-91ce-faffcbbe130d-workload-socket\") pod \"955ee3cb-72bd-4dab-91ce-faffcbbe130d\" (UID: \"955ee3cb-72bd-4dab-91ce-faffcbbe130d\") " Nov 03 02:50:52.307666 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:52.307336 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/955ee3cb-72bd-4dab-91ce-faffcbbe130d-credential-socket" pod "955ee3cb-72bd-4dab-91ce-faffcbbe130d" (UID: "955ee3cb-72bd-4dab-91ce-faffcbbe130d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:50:52.307666 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:52.307553 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/955ee3cb-72bd-4dab-91ce-faffcbbe130d-istiod-ca-cert" pod "955ee3cb-72bd-4dab-91ce-faffcbbe130d" (UID: "955ee3cb-72bd-4dab-91ce-faffcbbe130d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:50:52.307787 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:52.307740 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/955ee3cb-72bd-4dab-91ce-faffcbbe130d-workload-socket" pod "955ee3cb-72bd-4dab-91ce-faffcbbe130d" (UID: "955ee3cb-72bd-4dab-91ce-faffcbbe130d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:50:52.307787 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:52.307760 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/955ee3cb-72bd-4dab-91ce-faffcbbe130d-istio-data" pod "955ee3cb-72bd-4dab-91ce-faffcbbe130d" (UID: "955ee3cb-72bd-4dab-91ce-faffcbbe130d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:50:52.307963 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:52.307941 5605 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/955ee3cb-72bd-4dab-91ce-faffcbbe130d-istiod-ca-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:50:52.308032 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:52.307967 5605 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/955ee3cb-72bd-4dab-91ce-faffcbbe130d-istio-data\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:50:52.308032 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:52.307981 5605 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/955ee3cb-72bd-4dab-91ce-faffcbbe130d-workload-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:50:52.308032 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:52.307995 5605 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/955ee3cb-72bd-4dab-91ce-faffcbbe130d-workload-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:50:52.308032 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:52.308008 5605 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/955ee3cb-72bd-4dab-91ce-faffcbbe130d-credential-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:50:52.309622 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:52.309599 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/955ee3cb-72bd-4dab-91ce-faffcbbe130d-istio-token" pod "955ee3cb-72bd-4dab-91ce-faffcbbe130d" (UID: "955ee3cb-72bd-4dab-91ce-faffcbbe130d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:50:52.309734 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:52.309632 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/955ee3cb-72bd-4dab-91ce-faffcbbe130d-kube-api-access-6k6p7" pod "955ee3cb-72bd-4dab-91ce-faffcbbe130d" (UID: "955ee3cb-72bd-4dab-91ce-faffcbbe130d"). InnerVolumeSpecName "kube-api-access-6k6p7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:50:52.309855 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:52.309840 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/955ee3cb-72bd-4dab-91ce-faffcbbe130d-istio-podinfo" pod "955ee3cb-72bd-4dab-91ce-faffcbbe130d" (UID: "955ee3cb-72bd-4dab-91ce-faffcbbe130d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:50:52.309900 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:52.309871 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/955ee3cb-72bd-4dab-91ce-faffcbbe130d-istio-envoy" pod "955ee3cb-72bd-4dab-91ce-faffcbbe130d" (UID: "955ee3cb-72bd-4dab-91ce-faffcbbe130d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:50:52.409741 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:52.409700 5605 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/955ee3cb-72bd-4dab-91ce-faffcbbe130d-istio-podinfo\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:50:52.409741 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:52.409740 5605 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/955ee3cb-72bd-4dab-91ce-faffcbbe130d-istio-token\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:50:52.409978 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:52.409755 5605 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/955ee3cb-72bd-4dab-91ce-faffcbbe130d-istio-envoy\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:50:52.409978 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:52.409769 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6k6p7\" (UniqueName: \"kubernetes.io/projected/955ee3cb-72bd-4dab-91ce-faffcbbe130d-kube-api-access-6k6p7\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:50:52.628147 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:52.628055 5605 generic.go:356] "Generic (PLEG): container finished" podID="955ee3cb-72bd-4dab-91ce-faffcbbe130d" containerID="590738e66fb753edb2f7efa9b29a281759b31c9409800d0cec384d6e43f077cf" exitCode=0 Nov 03 02:50:52.628147 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:52.628106 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2" Nov 03 02:50:52.628395 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:52.628097 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2" event={"ID":"955ee3cb-72bd-4dab-91ce-faffcbbe130d","Type":"ContainerDied","Data":"590738e66fb753edb2f7efa9b29a281759b31c9409800d0cec384d6e43f077cf"} Nov 03 02:50:52.628395 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:52.628237 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2" event={"ID":"955ee3cb-72bd-4dab-91ce-faffcbbe130d","Type":"ContainerDied","Data":"856fcde7324f4527597b1334ff70403342fb1109367f06f03e67bc1436b18fb1"} Nov 03 02:50:52.628395 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:52.628261 5605 scope.go:122] "RemoveContainer" containerID="590738e66fb753edb2f7efa9b29a281759b31c9409800d0cec384d6e43f077cf" Nov 03 02:50:52.638583 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:52.638564 5605 scope.go:122] "RemoveContainer" containerID="7584787c15af69972b9bda79ce98e2f7221feeb167c253d85c111ee287ddfcd9" Nov 03 02:50:52.648953 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:52.648923 5605 scope.go:122] "RemoveContainer" containerID="590738e66fb753edb2f7efa9b29a281759b31c9409800d0cec384d6e43f077cf" Nov 03 02:50:52.649304 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:50:52.649282 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"590738e66fb753edb2f7efa9b29a281759b31c9409800d0cec384d6e43f077cf\": container with ID starting with 590738e66fb753edb2f7efa9b29a281759b31c9409800d0cec384d6e43f077cf not found: ID does not exist" containerID="590738e66fb753edb2f7efa9b29a281759b31c9409800d0cec384d6e43f077cf" Nov 03 02:50:52.649432 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:52.649316 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"590738e66fb753edb2f7efa9b29a281759b31c9409800d0cec384d6e43f077cf"} err="failed to get container status \"590738e66fb753edb2f7efa9b29a281759b31c9409800d0cec384d6e43f077cf\": rpc error: code = NotFound desc = could not find container \"590738e66fb753edb2f7efa9b29a281759b31c9409800d0cec384d6e43f077cf\": container with ID starting with 590738e66fb753edb2f7efa9b29a281759b31c9409800d0cec384d6e43f077cf not found: ID does not exist" Nov 03 02:50:52.649432 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:52.649348 5605 scope.go:122] "RemoveContainer" containerID="7584787c15af69972b9bda79ce98e2f7221feeb167c253d85c111ee287ddfcd9" Nov 03 02:50:52.649663 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:50:52.649633 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7584787c15af69972b9bda79ce98e2f7221feeb167c253d85c111ee287ddfcd9\": container with ID starting with 7584787c15af69972b9bda79ce98e2f7221feeb167c253d85c111ee287ddfcd9 not found: ID does not exist" containerID="7584787c15af69972b9bda79ce98e2f7221feeb167c253d85c111ee287ddfcd9" Nov 03 02:50:52.649787 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:52.649667 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7584787c15af69972b9bda79ce98e2f7221feeb167c253d85c111ee287ddfcd9"} err="failed to get container status \"7584787c15af69972b9bda79ce98e2f7221feeb167c253d85c111ee287ddfcd9\": rpc error: code = NotFound desc = could not find container \"7584787c15af69972b9bda79ce98e2f7221feeb167c253d85c111ee287ddfcd9\": container with ID starting with 7584787c15af69972b9bda79ce98e2f7221feeb167c253d85c111ee287ddfcd9 not found: ID does not exist" Nov 03 02:50:52.673808 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:52.673772 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp"] Nov 03 02:50:52.674025 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:52.674001 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp" podUID="d98a3d42-36f0-48d4-b6dc-7dd978b953ae" containerName="istio-proxy" containerID="cri-o://ebfba3cd615cd3f06257fabaebca3ddcc1cb8777ebddf77bfc71c43b456cb701" gracePeriod=30 Nov 03 02:50:52.936571 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:52.936477 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2"] Nov 03 02:50:52.941543 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:52.941513 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-ca6ef4-openshift-default-7945d7df57-9shw2"] Nov 03 02:50:53.915150 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:50:53.915112 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:50:54.773164 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:54.773129 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-666b7cb4b7-rq5h6"] Nov 03 02:50:54.773868 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:54.773838 5605 cpu_manager.go:414] "containerMap: removing container" podUID="955ee3cb-72bd-4dab-91ce-faffcbbe130d" containerName="istio-proxy" Nov 03 02:50:54.773868 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:54.773864 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="955ee3cb-72bd-4dab-91ce-faffcbbe130d" containerName="istio-proxy" Nov 03 02:50:54.774046 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:54.774019 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="955ee3cb-72bd-4dab-91ce-faffcbbe130d" containerName="istio-proxy" Nov 03 02:50:54.774046 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:54.774039 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="955ee3cb-72bd-4dab-91ce-faffcbbe130d" containerName="istio-proxy" Nov 03 02:50:54.778867 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:54.778846 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-666b7cb4b7-rq5h6" Nov 03 02:50:54.846462 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:54.846423 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/11118499-1a93-4e9d-8ecf-43116090ab21-routing-overlay-volume\") pod \"payload-processing-external-model-666b7cb4b7-rq5h6\" (UID: \"11118499-1a93-4e9d-8ecf-43116090ab21\") " pod="models-as-a-service/payload-processing-external-model-666b7cb4b7-rq5h6" Nov 03 02:50:54.846629 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:54.846483 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4trdh\" (UniqueName: \"kubernetes.io/projected/11118499-1a93-4e9d-8ecf-43116090ab21-kube-api-access-4trdh\") pod \"payload-processing-external-model-666b7cb4b7-rq5h6\" (UID: \"11118499-1a93-4e9d-8ecf-43116090ab21\") " pod="models-as-a-service/payload-processing-external-model-666b7cb4b7-rq5h6" Nov 03 02:50:54.846629 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:54.846576 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/11118499-1a93-4e9d-8ecf-43116090ab21-provider-credentials\") pod \"payload-processing-external-model-666b7cb4b7-rq5h6\" (UID: \"11118499-1a93-4e9d-8ecf-43116090ab21\") " pod="models-as-a-service/payload-processing-external-model-666b7cb4b7-rq5h6" Nov 03 02:50:54.846741 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:54.846657 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11118499-1a93-4e9d-8ecf-43116090ab21-plugins-config-volume\") pod \"payload-processing-external-model-666b7cb4b7-rq5h6\" (UID: \"11118499-1a93-4e9d-8ecf-43116090ab21\") " pod="models-as-a-service/payload-processing-external-model-666b7cb4b7-rq5h6" Nov 03 02:50:54.852495 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:54.852460 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="955ee3cb-72bd-4dab-91ce-faffcbbe130d" path="/var/lib/kubelet/pods/955ee3cb-72bd-4dab-91ce-faffcbbe130d/volumes" Nov 03 02:50:54.876035 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:54.875994 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-50a3cab9906d\"" Nov 03 02:50:54.876035 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:54.876010 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 03 02:50:54.876035 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:54.876013 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 03 02:50:54.876310 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:54.876037 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 03 02:50:54.876310 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:54.876079 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 03 02:50:54.876310 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:54.876131 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-vtw9w\"" Nov 03 02:50:54.878350 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:54.878330 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-666b7cb4b7-rq5h6"] Nov 03 02:50:54.947275 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:54.947242 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/11118499-1a93-4e9d-8ecf-43116090ab21-routing-overlay-volume\") pod \"payload-processing-external-model-666b7cb4b7-rq5h6\" (UID: \"11118499-1a93-4e9d-8ecf-43116090ab21\") " pod="models-as-a-service/payload-processing-external-model-666b7cb4b7-rq5h6" Nov 03 02:50:54.947801 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:54.947320 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4trdh\" (UniqueName: \"kubernetes.io/projected/11118499-1a93-4e9d-8ecf-43116090ab21-kube-api-access-4trdh\") pod \"payload-processing-external-model-666b7cb4b7-rq5h6\" (UID: \"11118499-1a93-4e9d-8ecf-43116090ab21\") " pod="models-as-a-service/payload-processing-external-model-666b7cb4b7-rq5h6" Nov 03 02:50:54.947801 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:54.947379 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/11118499-1a93-4e9d-8ecf-43116090ab21-provider-credentials\") pod \"payload-processing-external-model-666b7cb4b7-rq5h6\" (UID: \"11118499-1a93-4e9d-8ecf-43116090ab21\") " pod="models-as-a-service/payload-processing-external-model-666b7cb4b7-rq5h6" Nov 03 02:50:54.947801 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:54.947522 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11118499-1a93-4e9d-8ecf-43116090ab21-plugins-config-volume\") pod \"payload-processing-external-model-666b7cb4b7-rq5h6\" (UID: \"11118499-1a93-4e9d-8ecf-43116090ab21\") " pod="models-as-a-service/payload-processing-external-model-666b7cb4b7-rq5h6" Nov 03 02:50:54.948039 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:54.948011 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/11118499-1a93-4e9d-8ecf-43116090ab21-routing-overlay-volume\") pod \"payload-processing-external-model-666b7cb4b7-rq5h6\" (UID: \"11118499-1a93-4e9d-8ecf-43116090ab21\") " pod="models-as-a-service/payload-processing-external-model-666b7cb4b7-rq5h6" Nov 03 02:50:54.948235 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:54.948214 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11118499-1a93-4e9d-8ecf-43116090ab21-plugins-config-volume\") pod \"payload-processing-external-model-666b7cb4b7-rq5h6\" (UID: \"11118499-1a93-4e9d-8ecf-43116090ab21\") " pod="models-as-a-service/payload-processing-external-model-666b7cb4b7-rq5h6" Nov 03 02:50:54.949891 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:54.949870 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/11118499-1a93-4e9d-8ecf-43116090ab21-provider-credentials\") pod \"payload-processing-external-model-666b7cb4b7-rq5h6\" (UID: \"11118499-1a93-4e9d-8ecf-43116090ab21\") " pod="models-as-a-service/payload-processing-external-model-666b7cb4b7-rq5h6" Nov 03 02:50:55.000659 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:55.000626 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4trdh\" (UniqueName: \"kubernetes.io/projected/11118499-1a93-4e9d-8ecf-43116090ab21-kube-api-access-4trdh\") pod \"payload-processing-external-model-666b7cb4b7-rq5h6\" (UID: \"11118499-1a93-4e9d-8ecf-43116090ab21\") " pod="models-as-a-service/payload-processing-external-model-666b7cb4b7-rq5h6" Nov 03 02:50:55.092565 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:55.092535 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-666b7cb4b7-rq5h6" Nov 03 02:50:55.462468 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:55.462377 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-666b7cb4b7-rq5h6"] Nov 03 02:50:55.467944 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:50:55.467908 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11118499_1a93_4e9d_8ecf_43116090ab21.slice/crio-ac93f127fbcf510fd512f06db472d3a30c88cf4de4ab63b0fabe8410b0751b39 WatchSource:0}: Error finding container ac93f127fbcf510fd512f06db472d3a30c88cf4de4ab63b0fabe8410b0751b39: Status 404 returned error can't find the container with id ac93f127fbcf510fd512f06db472d3a30c88cf4de4ab63b0fabe8410b0751b39 Nov 03 02:50:55.648713 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:55.648663 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-666b7cb4b7-rq5h6" event={"ID":"11118499-1a93-4e9d-8ecf-43116090ab21","Type":"ContainerStarted","Data":"ec5fe58a252d4fde5e30569c574ed0bb1989646b89a94efd01fef0f96fc565d3"} Nov 03 02:50:55.648919 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:55.648740 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-666b7cb4b7-rq5h6" Nov 03 02:50:55.648919 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:55.648771 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-666b7cb4b7-rq5h6" event={"ID":"11118499-1a93-4e9d-8ecf-43116090ab21","Type":"ContainerStarted","Data":"ac93f127fbcf510fd512f06db472d3a30c88cf4de4ab63b0fabe8410b0751b39"} Nov 03 02:50:55.756414 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:55.756275 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-666b7cb4b7-rq5h6" podStartSLOduration=1.756257094 podStartE2EDuration="1.756257094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:50:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:50:55.75459152 +0000 UTC m=+3533.013789675" watchObservedRunningTime="2026-11-03 02:50:55.756257094 +0000 UTC m=+3533.015455280" Nov 03 02:50:57.940770 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:57.940708 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp" Nov 03 02:50:57.999857 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:57.999825 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d98a3d42-36f0-48d4-b6dc-7dd978b953ae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d98a3d42-36f0-48d4-b6dc-7dd978b953ae-istio-data\") pod \"d98a3d42-36f0-48d4-b6dc-7dd978b953ae\" (UID: \"d98a3d42-36f0-48d4-b6dc-7dd978b953ae\") " Nov 03 02:50:58.000015 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:57.999877 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d98a3d42-36f0-48d4-b6dc-7dd978b953ae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98a3d42-36f0-48d4-b6dc-7dd978b953ae-workload-socket\") pod \"d98a3d42-36f0-48d4-b6dc-7dd978b953ae\" (UID: \"d98a3d42-36f0-48d4-b6dc-7dd978b953ae\") " Nov 03 02:50:58.000015 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:57.999925 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d98a3d42-36f0-48d4-b6dc-7dd978b953ae-kube-api-access-zqx6t\" (UniqueName: \"kubernetes.io/projected/d98a3d42-36f0-48d4-b6dc-7dd978b953ae-kube-api-access-zqx6t\") pod \"d98a3d42-36f0-48d4-b6dc-7dd978b953ae\" (UID: \"d98a3d42-36f0-48d4-b6dc-7dd978b953ae\") " Nov 03 02:50:58.000015 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:57.999985 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d98a3d42-36f0-48d4-b6dc-7dd978b953ae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d98a3d42-36f0-48d4-b6dc-7dd978b953ae-workload-certs\") pod \"d98a3d42-36f0-48d4-b6dc-7dd978b953ae\" (UID: \"d98a3d42-36f0-48d4-b6dc-7dd978b953ae\") " Nov 03 02:50:58.000204 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:58.000050 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d98a3d42-36f0-48d4-b6dc-7dd978b953ae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98a3d42-36f0-48d4-b6dc-7dd978b953ae-credential-socket\") pod \"d98a3d42-36f0-48d4-b6dc-7dd978b953ae\" (UID: \"d98a3d42-36f0-48d4-b6dc-7dd978b953ae\") " Nov 03 02:50:58.000204 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:58.000110 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d98a3d42-36f0-48d4-b6dc-7dd978b953ae-istio-token\" (UniqueName: \"kubernetes.io/projected/d98a3d42-36f0-48d4-b6dc-7dd978b953ae-istio-token\") pod \"d98a3d42-36f0-48d4-b6dc-7dd978b953ae\" (UID: \"d98a3d42-36f0-48d4-b6dc-7dd978b953ae\") " Nov 03 02:50:58.000204 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:58.000138 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d98a3d42-36f0-48d4-b6dc-7dd978b953ae-istio-data" pod "d98a3d42-36f0-48d4-b6dc-7dd978b953ae" (UID: "d98a3d42-36f0-48d4-b6dc-7dd978b953ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:50:58.000204 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:58.000148 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d98a3d42-36f0-48d4-b6dc-7dd978b953ae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d98a3d42-36f0-48d4-b6dc-7dd978b953ae-istio-podinfo\") pod \"d98a3d42-36f0-48d4-b6dc-7dd978b953ae\" (UID: \"d98a3d42-36f0-48d4-b6dc-7dd978b953ae\") " Nov 03 02:50:58.000476 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:58.000226 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d98a3d42-36f0-48d4-b6dc-7dd978b953ae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d98a3d42-36f0-48d4-b6dc-7dd978b953ae-istiod-ca-cert\") pod \"d98a3d42-36f0-48d4-b6dc-7dd978b953ae\" (UID: \"d98a3d42-36f0-48d4-b6dc-7dd978b953ae\") " Nov 03 02:50:58.000476 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:58.000163 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d98a3d42-36f0-48d4-b6dc-7dd978b953ae-workload-socket" pod "d98a3d42-36f0-48d4-b6dc-7dd978b953ae" (UID: "d98a3d42-36f0-48d4-b6dc-7dd978b953ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:50:58.000476 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:58.000375 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d98a3d42-36f0-48d4-b6dc-7dd978b953ae-workload-certs" pod "d98a3d42-36f0-48d4-b6dc-7dd978b953ae" (UID: "d98a3d42-36f0-48d4-b6dc-7dd978b953ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:50:58.000476 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:58.000421 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d98a3d42-36f0-48d4-b6dc-7dd978b953ae-credential-socket" pod "d98a3d42-36f0-48d4-b6dc-7dd978b953ae" (UID: "d98a3d42-36f0-48d4-b6dc-7dd978b953ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:50:58.000697 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:58.000668 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d98a3d42-36f0-48d4-b6dc-7dd978b953ae-istiod-ca-cert" pod "d98a3d42-36f0-48d4-b6dc-7dd978b953ae" (UID: "d98a3d42-36f0-48d4-b6dc-7dd978b953ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:50:58.000761 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:58.000704 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d98a3d42-36f0-48d4-b6dc-7dd978b953ae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d98a3d42-36f0-48d4-b6dc-7dd978b953ae-istio-envoy\") pod \"d98a3d42-36f0-48d4-b6dc-7dd978b953ae\" (UID: \"d98a3d42-36f0-48d4-b6dc-7dd978b953ae\") " Nov 03 02:50:58.001130 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:58.001112 5605 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d98a3d42-36f0-48d4-b6dc-7dd978b953ae-istiod-ca-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:50:58.001213 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:58.001135 5605 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d98a3d42-36f0-48d4-b6dc-7dd978b953ae-istio-data\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:50:58.001213 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:58.001150 5605 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98a3d42-36f0-48d4-b6dc-7dd978b953ae-workload-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:50:58.001213 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:58.001162 5605 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d98a3d42-36f0-48d4-b6dc-7dd978b953ae-workload-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:50:58.001213 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:58.001175 5605 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98a3d42-36f0-48d4-b6dc-7dd978b953ae-credential-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:50:58.002179 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:58.002161 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d98a3d42-36f0-48d4-b6dc-7dd978b953ae-kube-api-access-zqx6t" pod "d98a3d42-36f0-48d4-b6dc-7dd978b953ae" (UID: "d98a3d42-36f0-48d4-b6dc-7dd978b953ae"). InnerVolumeSpecName "kube-api-access-zqx6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:50:58.002347 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:58.002330 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d98a3d42-36f0-48d4-b6dc-7dd978b953ae-istio-token" pod "d98a3d42-36f0-48d4-b6dc-7dd978b953ae" (UID: "d98a3d42-36f0-48d4-b6dc-7dd978b953ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:50:58.002642 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:58.002616 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d98a3d42-36f0-48d4-b6dc-7dd978b953ae-istio-envoy" pod "d98a3d42-36f0-48d4-b6dc-7dd978b953ae" (UID: "d98a3d42-36f0-48d4-b6dc-7dd978b953ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:50:58.002642 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:58.002629 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d98a3d42-36f0-48d4-b6dc-7dd978b953ae-istio-podinfo" pod "d98a3d42-36f0-48d4-b6dc-7dd978b953ae" (UID: "d98a3d42-36f0-48d4-b6dc-7dd978b953ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:50:58.102089 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:58.102057 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqx6t\" (UniqueName: \"kubernetes.io/projected/d98a3d42-36f0-48d4-b6dc-7dd978b953ae-kube-api-access-zqx6t\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:50:58.102089 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:58.102089 5605 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d98a3d42-36f0-48d4-b6dc-7dd978b953ae-istio-token\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:50:58.102310 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:58.102102 5605 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d98a3d42-36f0-48d4-b6dc-7dd978b953ae-istio-podinfo\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:50:58.102310 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:58.102118 5605 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d98a3d42-36f0-48d4-b6dc-7dd978b953ae-istio-envoy\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:50:58.541798 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:58.541766 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-56c7fb7568-t7rlj"] Nov 03 02:50:58.542235 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:58.542223 5605 cpu_manager.go:414] "containerMap: removing container" podUID="d98a3d42-36f0-48d4-b6dc-7dd978b953ae" containerName="istio-proxy" Nov 03 02:50:58.542289 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:58.542237 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d98a3d42-36f0-48d4-b6dc-7dd978b953ae" containerName="istio-proxy" Nov 03 02:50:58.542289 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:58.542264 5605 cpu_manager.go:414] "containerMap: removing container" podUID="955ee3cb-72bd-4dab-91ce-faffcbbe130d" containerName="istio-proxy" Nov 03 02:50:58.542289 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:58.542270 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="955ee3cb-72bd-4dab-91ce-faffcbbe130d" containerName="istio-proxy" Nov 03 02:50:58.542403 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:58.542342 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="d98a3d42-36f0-48d4-b6dc-7dd978b953ae" containerName="istio-proxy" Nov 03 02:50:58.542403 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:58.542368 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="d98a3d42-36f0-48d4-b6dc-7dd978b953ae" containerName="istio-proxy" Nov 03 02:50:58.545741 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:58.545724 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56c7fb7568-t7rlj" Nov 03 02:50:58.597015 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:58.596990 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-f20e06ca5b75\"" Nov 03 02:50:58.599319 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:58.599295 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-56c7fb7568-t7rlj"] Nov 03 02:50:58.606734 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:58.606712 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb78z\" (UniqueName: \"kubernetes.io/projected/c82367ed-80d4-44de-9d37-450c1fca5281-kube-api-access-nb78z\") pod \"payload-processing-external-model-56c7fb7568-t7rlj\" (UID: \"c82367ed-80d4-44de-9d37-450c1fca5281\") " pod="models-as-a-service/payload-processing-external-model-56c7fb7568-t7rlj" Nov 03 02:50:58.606857 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:58.606751 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c82367ed-80d4-44de-9d37-450c1fca5281-plugins-config-volume\") pod \"payload-processing-external-model-56c7fb7568-t7rlj\" (UID: \"c82367ed-80d4-44de-9d37-450c1fca5281\") " pod="models-as-a-service/payload-processing-external-model-56c7fb7568-t7rlj" Nov 03 02:50:58.606960 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:58.606929 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c82367ed-80d4-44de-9d37-450c1fca5281-provider-credentials\") pod \"payload-processing-external-model-56c7fb7568-t7rlj\" (UID: \"c82367ed-80d4-44de-9d37-450c1fca5281\") " pod="models-as-a-service/payload-processing-external-model-56c7fb7568-t7rlj" Nov 03 02:50:58.606999 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:58.606986 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c82367ed-80d4-44de-9d37-450c1fca5281-routing-overlay-volume\") pod \"payload-processing-external-model-56c7fb7568-t7rlj\" (UID: \"c82367ed-80d4-44de-9d37-450c1fca5281\") " pod="models-as-a-service/payload-processing-external-model-56c7fb7568-t7rlj" Nov 03 02:50:58.668053 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:58.668019 5605 generic.go:356] "Generic (PLEG): container finished" podID="d98a3d42-36f0-48d4-b6dc-7dd978b953ae" containerID="ebfba3cd615cd3f06257fabaebca3ddcc1cb8777ebddf77bfc71c43b456cb701" exitCode=0 Nov 03 02:50:58.668188 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:58.668066 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp" Nov 03 02:50:58.668188 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:58.668088 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp" event={"ID":"d98a3d42-36f0-48d4-b6dc-7dd978b953ae","Type":"ContainerDied","Data":"ebfba3cd615cd3f06257fabaebca3ddcc1cb8777ebddf77bfc71c43b456cb701"} Nov 03 02:50:58.668188 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:58.668136 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp" event={"ID":"d98a3d42-36f0-48d4-b6dc-7dd978b953ae","Type":"ContainerDied","Data":"458f4a7ff90155a3aff7f45b0202042d3743dfa5fd7d23db2c301a0e7dd2c5b8"} Nov 03 02:50:58.668188 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:58.668160 5605 scope.go:122] "RemoveContainer" containerID="ebfba3cd615cd3f06257fabaebca3ddcc1cb8777ebddf77bfc71c43b456cb701" Nov 03 02:50:58.678539 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:58.678519 5605 scope.go:122] "RemoveContainer" containerID="be592416eb06a16a33616e403d22f5e788f0a6eb30c86edc20fe36fddb62bd1a" Nov 03 02:50:58.687834 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:58.687817 5605 scope.go:122] "RemoveContainer" containerID="ebfba3cd615cd3f06257fabaebca3ddcc1cb8777ebddf77bfc71c43b456cb701" Nov 03 02:50:58.688109 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:50:58.688091 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebfba3cd615cd3f06257fabaebca3ddcc1cb8777ebddf77bfc71c43b456cb701\": container with ID starting with ebfba3cd615cd3f06257fabaebca3ddcc1cb8777ebddf77bfc71c43b456cb701 not found: ID does not exist" containerID="ebfba3cd615cd3f06257fabaebca3ddcc1cb8777ebddf77bfc71c43b456cb701" Nov 03 02:50:58.688164 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:58.688118 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebfba3cd615cd3f06257fabaebca3ddcc1cb8777ebddf77bfc71c43b456cb701"} err="failed to get container status \"ebfba3cd615cd3f06257fabaebca3ddcc1cb8777ebddf77bfc71c43b456cb701\": rpc error: code = NotFound desc = could not find container \"ebfba3cd615cd3f06257fabaebca3ddcc1cb8777ebddf77bfc71c43b456cb701\": container with ID starting with ebfba3cd615cd3f06257fabaebca3ddcc1cb8777ebddf77bfc71c43b456cb701 not found: ID does not exist" Nov 03 02:50:58.688164 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:58.688138 5605 scope.go:122] "RemoveContainer" containerID="be592416eb06a16a33616e403d22f5e788f0a6eb30c86edc20fe36fddb62bd1a" Nov 03 02:50:58.688419 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:50:58.688396 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be592416eb06a16a33616e403d22f5e788f0a6eb30c86edc20fe36fddb62bd1a\": container with ID starting with be592416eb06a16a33616e403d22f5e788f0a6eb30c86edc20fe36fddb62bd1a not found: ID does not exist" containerID="be592416eb06a16a33616e403d22f5e788f0a6eb30c86edc20fe36fddb62bd1a" Nov 03 02:50:58.688484 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:58.688430 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be592416eb06a16a33616e403d22f5e788f0a6eb30c86edc20fe36fddb62bd1a"} err="failed to get container status \"be592416eb06a16a33616e403d22f5e788f0a6eb30c86edc20fe36fddb62bd1a\": rpc error: code = NotFound desc = could not find container \"be592416eb06a16a33616e403d22f5e788f0a6eb30c86edc20fe36fddb62bd1a\": container with ID starting with be592416eb06a16a33616e403d22f5e788f0a6eb30c86edc20fe36fddb62bd1a not found: ID does not exist" Nov 03 02:50:58.708151 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:58.708123 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb78z\" (UniqueName: \"kubernetes.io/projected/c82367ed-80d4-44de-9d37-450c1fca5281-kube-api-access-nb78z\") pod \"payload-processing-external-model-56c7fb7568-t7rlj\" (UID: \"c82367ed-80d4-44de-9d37-450c1fca5281\") " pod="models-as-a-service/payload-processing-external-model-56c7fb7568-t7rlj" Nov 03 02:50:58.708291 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:58.708162 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c82367ed-80d4-44de-9d37-450c1fca5281-plugins-config-volume\") pod \"payload-processing-external-model-56c7fb7568-t7rlj\" (UID: \"c82367ed-80d4-44de-9d37-450c1fca5281\") " pod="models-as-a-service/payload-processing-external-model-56c7fb7568-t7rlj" Nov 03 02:50:58.708291 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:58.708231 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c82367ed-80d4-44de-9d37-450c1fca5281-provider-credentials\") pod \"payload-processing-external-model-56c7fb7568-t7rlj\" (UID: \"c82367ed-80d4-44de-9d37-450c1fca5281\") " pod="models-as-a-service/payload-processing-external-model-56c7fb7568-t7rlj" Nov 03 02:50:58.708291 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:58.708266 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c82367ed-80d4-44de-9d37-450c1fca5281-routing-overlay-volume\") pod \"payload-processing-external-model-56c7fb7568-t7rlj\" (UID: \"c82367ed-80d4-44de-9d37-450c1fca5281\") " pod="models-as-a-service/payload-processing-external-model-56c7fb7568-t7rlj" Nov 03 02:50:58.709037 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:58.709013 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c82367ed-80d4-44de-9d37-450c1fca5281-routing-overlay-volume\") pod \"payload-processing-external-model-56c7fb7568-t7rlj\" (UID: \"c82367ed-80d4-44de-9d37-450c1fca5281\") " pod="models-as-a-service/payload-processing-external-model-56c7fb7568-t7rlj" Nov 03 02:50:58.709104 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:58.709090 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c82367ed-80d4-44de-9d37-450c1fca5281-plugins-config-volume\") pod \"payload-processing-external-model-56c7fb7568-t7rlj\" (UID: \"c82367ed-80d4-44de-9d37-450c1fca5281\") " pod="models-as-a-service/payload-processing-external-model-56c7fb7568-t7rlj" Nov 03 02:50:58.710630 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:58.710612 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c82367ed-80d4-44de-9d37-450c1fca5281-provider-credentials\") pod \"payload-processing-external-model-56c7fb7568-t7rlj\" (UID: \"c82367ed-80d4-44de-9d37-450c1fca5281\") " pod="models-as-a-service/payload-processing-external-model-56c7fb7568-t7rlj" Nov 03 02:50:58.761586 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:58.761560 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb78z\" (UniqueName: \"kubernetes.io/projected/c82367ed-80d4-44de-9d37-450c1fca5281-kube-api-access-nb78z\") pod \"payload-processing-external-model-56c7fb7568-t7rlj\" (UID: \"c82367ed-80d4-44de-9d37-450c1fca5281\") " pod="models-as-a-service/payload-processing-external-model-56c7fb7568-t7rlj" Nov 03 02:50:58.821105 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:58.821074 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp"] Nov 03 02:50:58.824718 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:58.824692 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-604da0-openshift-default-d59b76957-7mtwp"] Nov 03 02:50:58.850896 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:58.850860 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d98a3d42-36f0-48d4-b6dc-7dd978b953ae" path="/var/lib/kubelet/pods/d98a3d42-36f0-48d4-b6dc-7dd978b953ae/volumes" Nov 03 02:50:58.858772 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:58.858749 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56c7fb7568-t7rlj" Nov 03 02:50:59.234349 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:59.234270 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-56c7fb7568-t7rlj"] Nov 03 02:50:59.240387 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:50:59.240343 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc82367ed_80d4_44de_9d37_450c1fca5281.slice/crio-3a723a79de6b12a8b2a482934db6b3808fc09ed10af5bc1e9231719e7ff590a8 WatchSource:0}: Error finding container 3a723a79de6b12a8b2a482934db6b3808fc09ed10af5bc1e9231719e7ff590a8: Status 404 returned error can't find the container with id 3a723a79de6b12a8b2a482934db6b3808fc09ed10af5bc1e9231719e7ff590a8 Nov 03 02:50:59.676225 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:59.676186 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56c7fb7568-t7rlj" event={"ID":"c82367ed-80d4-44de-9d37-450c1fca5281","Type":"ContainerStarted","Data":"df22aa38495fb9c6e39f9ebae6c462c5c326f6baf65df1dfcb1456b47e75f875"} Nov 03 02:50:59.676478 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:59.676344 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-56c7fb7568-t7rlj" Nov 03 02:50:59.676478 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:59.676408 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56c7fb7568-t7rlj" event={"ID":"c82367ed-80d4-44de-9d37-450c1fca5281","Type":"ContainerStarted","Data":"3a723a79de6b12a8b2a482934db6b3808fc09ed10af5bc1e9231719e7ff590a8"} Nov 03 02:50:59.784799 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:50:59.784747 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-56c7fb7568-t7rlj" podStartSLOduration=1.784732792 podStartE2EDuration="1.784732792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:50:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:50:59.78396146 +0000 UTC m=+3537.043159627" watchObservedRunningTime="2026-11-03 02:50:59.784732792 +0000 UTC m=+3537.043930966" Nov 03 02:51:04.523029 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:04.522998 5605 scope.go:122] "RemoveContainer" containerID="3f7315d548fdfdca574c997ab7ba3bb07f61a07b7cc55237c2e46af124b552ac" Nov 03 02:51:04.533128 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:04.533104 5605 scope.go:122] "RemoveContainer" containerID="a6fd954fae62f72abacbba2baf5fe64325520528c8dc2ca55dadeb6c1d3585e5" Nov 03 02:51:04.542593 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:04.542574 5605 scope.go:122] "RemoveContainer" containerID="6e59891937a8f7c4bc8cc3503ec6b13c9e15e118f5d5d2291bdf784f09011410" Nov 03 02:51:04.550769 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:04.550751 5605 scope.go:122] "RemoveContainer" containerID="b87d9f4ef1f14a1a59f243ef9bc86a9a2238345d84d9fa41e0c1ede853bb2aab" Nov 03 02:51:04.558402 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:04.558384 5605 scope.go:122] "RemoveContainer" containerID="1579b2d465a044636e5cb31c43a4dacf9d761ba20fed61153f0eb5e03c4d119c" Nov 03 02:51:05.049427 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:05.049396 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c8ee6b-4hn2p"] Nov 03 02:51:05.049879 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:05.049867 5605 cpu_manager.go:414] "containerMap: removing container" podUID="d98a3d42-36f0-48d4-b6dc-7dd978b953ae" containerName="istio-proxy" Nov 03 02:51:05.049932 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:05.049881 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d98a3d42-36f0-48d4-b6dc-7dd978b953ae" containerName="istio-proxy" Nov 03 02:51:05.055282 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:05.055259 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c8ee6b-4hn2p" Nov 03 02:51:05.154448 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:05.154414 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:51:05.154651 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:05.154455 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:51:05.154651 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:05.154475 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:51:05.154651 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:05.154561 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:51:05.156674 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:05.156653 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c8ee6b-4hn2p"] Nov 03 02:51:05.180829 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:05.180802 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvhvg\" (UniqueName: \"kubernetes.io/projected/ecaa74f8-112f-434d-9ebf-45e23e713cde-kube-api-access-hvhvg\") pod \"maas-api-revoke-keys-e2e-autores-c8ee6b-4hn2p\" (UID: \"ecaa74f8-112f-434d-9ebf-45e23e713cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c8ee6b-4hn2p" Nov 03 02:51:05.180979 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:05.180843 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecaa74f8-112f-434d-9ebf-45e23e713cde-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-c8ee6b-4hn2p\" (UID: \"ecaa74f8-112f-434d-9ebf-45e23e713cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c8ee6b-4hn2p" Nov 03 02:51:05.180979 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:05.180869 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecaa74f8-112f-434d-9ebf-45e23e713cde-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-c8ee6b-4hn2p\" (UID: \"ecaa74f8-112f-434d-9ebf-45e23e713cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c8ee6b-4hn2p" Nov 03 02:51:05.281805 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:05.281760 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvhvg\" (UniqueName: \"kubernetes.io/projected/ecaa74f8-112f-434d-9ebf-45e23e713cde-kube-api-access-hvhvg\") pod \"maas-api-revoke-keys-e2e-autores-c8ee6b-4hn2p\" (UID: \"ecaa74f8-112f-434d-9ebf-45e23e713cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c8ee6b-4hn2p" Nov 03 02:51:05.282016 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:05.281821 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecaa74f8-112f-434d-9ebf-45e23e713cde-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-c8ee6b-4hn2p\" (UID: \"ecaa74f8-112f-434d-9ebf-45e23e713cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c8ee6b-4hn2p" Nov 03 02:51:05.282016 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:05.281857 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecaa74f8-112f-434d-9ebf-45e23e713cde-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-c8ee6b-4hn2p\" (UID: \"ecaa74f8-112f-434d-9ebf-45e23e713cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c8ee6b-4hn2p" Nov 03 02:51:05.282718 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:05.282691 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecaa74f8-112f-434d-9ebf-45e23e713cde-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-c8ee6b-4hn2p\" (UID: \"ecaa74f8-112f-434d-9ebf-45e23e713cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c8ee6b-4hn2p" Nov 03 02:51:05.284292 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:05.284268 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecaa74f8-112f-434d-9ebf-45e23e713cde-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-c8ee6b-4hn2p\" (UID: \"ecaa74f8-112f-434d-9ebf-45e23e713cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c8ee6b-4hn2p" Nov 03 02:51:05.335968 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:05.335892 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvhvg\" (UniqueName: \"kubernetes.io/projected/ecaa74f8-112f-434d-9ebf-45e23e713cde-kube-api-access-hvhvg\") pod \"maas-api-revoke-keys-e2e-autores-c8ee6b-4hn2p\" (UID: \"ecaa74f8-112f-434d-9ebf-45e23e713cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c8ee6b-4hn2p" Nov 03 02:51:05.368112 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:05.368079 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c8ee6b-4hn2p" Nov 03 02:51:05.735852 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:05.735777 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c8ee6b-4hn2p"] Nov 03 02:51:05.742270 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:51:05.742239 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecaa74f8_112f_434d_9ebf_45e23e713cde.slice/crio-9cec7d2f90e5559d26e6cce108ec43c2fa68b50d7b3fad75b8deb424b7f5e0d9 WatchSource:0}: Error finding container 9cec7d2f90e5559d26e6cce108ec43c2fa68b50d7b3fad75b8deb424b7f5e0d9: Status 404 returned error can't find the container with id 9cec7d2f90e5559d26e6cce108ec43c2fa68b50d7b3fad75b8deb424b7f5e0d9 Nov 03 02:51:06.658205 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:06.658178 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-666b7cb4b7-rq5h6" Nov 03 02:51:06.718929 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:06.718897 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c8ee6b-4hn2p" event={"ID":"ecaa74f8-112f-434d-9ebf-45e23e713cde","Type":"ContainerStarted","Data":"7641e4a1a1616d73dc084b0c67212e4c6026397094d265b794ff393f82e85a7a"} Nov 03 02:51:06.719103 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:06.718948 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c8ee6b-4hn2p" event={"ID":"ecaa74f8-112f-434d-9ebf-45e23e713cde","Type":"ContainerStarted","Data":"9cec7d2f90e5559d26e6cce108ec43c2fa68b50d7b3fad75b8deb424b7f5e0d9"} Nov 03 02:51:06.876779 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:06.876719 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c8ee6b-4hn2p" podStartSLOduration=1.876701734 podStartE2EDuration="1.876701734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:51:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:51:06.873632283 +0000 UTC m=+3544.132830437" watchObservedRunningTime="2026-11-03 02:51:06.876701734 +0000 UTC m=+3544.135899890" Nov 03 02:51:07.726749 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:07.726714 5605 generic.go:356] "Generic (PLEG): container finished" podID="ecaa74f8-112f-434d-9ebf-45e23e713cde" containerID="7641e4a1a1616d73dc084b0c67212e4c6026397094d265b794ff393f82e85a7a" exitCode=0 Nov 03 02:51:07.726934 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:07.726799 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c8ee6b-4hn2p" event={"ID":"ecaa74f8-112f-434d-9ebf-45e23e713cde","Type":"ContainerDied","Data":"7641e4a1a1616d73dc084b0c67212e4c6026397094d265b794ff393f82e85a7a"} Nov 03 02:51:07.827303 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:07.827269 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:51:08.880062 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:08.880038 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c8ee6b-4hn2p" Nov 03 02:51:08.922865 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:08.922837 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecaa74f8-112f-434d-9ebf-45e23e713cde-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecaa74f8-112f-434d-9ebf-45e23e713cde-maas-api-openshift-ca\") pod \"ecaa74f8-112f-434d-9ebf-45e23e713cde\" (UID: \"ecaa74f8-112f-434d-9ebf-45e23e713cde\") " Nov 03 02:51:08.922999 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:08.922870 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecaa74f8-112f-434d-9ebf-45e23e713cde-kube-api-access-hvhvg\" (UniqueName: \"kubernetes.io/projected/ecaa74f8-112f-434d-9ebf-45e23e713cde-kube-api-access-hvhvg\") pod \"ecaa74f8-112f-434d-9ebf-45e23e713cde\" (UID: \"ecaa74f8-112f-434d-9ebf-45e23e713cde\") " Nov 03 02:51:08.922999 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:08.922975 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecaa74f8-112f-434d-9ebf-45e23e713cde-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecaa74f8-112f-434d-9ebf-45e23e713cde-maas-api-xks-ca\") pod \"ecaa74f8-112f-434d-9ebf-45e23e713cde\" (UID: \"ecaa74f8-112f-434d-9ebf-45e23e713cde\") " Nov 03 02:51:08.923185 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:08.923163 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecaa74f8-112f-434d-9ebf-45e23e713cde-maas-api-openshift-ca" pod "ecaa74f8-112f-434d-9ebf-45e23e713cde" (UID: "ecaa74f8-112f-434d-9ebf-45e23e713cde"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:51:08.923303 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:08.923288 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecaa74f8-112f-434d-9ebf-45e23e713cde-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:51:08.925528 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:08.925502 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecaa74f8-112f-434d-9ebf-45e23e713cde-maas-api-xks-ca" pod "ecaa74f8-112f-434d-9ebf-45e23e713cde" (UID: "ecaa74f8-112f-434d-9ebf-45e23e713cde"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:51:08.925653 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:08.925530 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecaa74f8-112f-434d-9ebf-45e23e713cde-kube-api-access-hvhvg" pod "ecaa74f8-112f-434d-9ebf-45e23e713cde" (UID: "ecaa74f8-112f-434d-9ebf-45e23e713cde"). InnerVolumeSpecName "kube-api-access-hvhvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:51:09.024042 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:09.023984 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvhvg\" (UniqueName: \"kubernetes.io/projected/ecaa74f8-112f-434d-9ebf-45e23e713cde-kube-api-access-hvhvg\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:51:09.024042 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:09.024011 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecaa74f8-112f-434d-9ebf-45e23e713cde-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:51:09.738941 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:09.738913 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c8ee6b-4hn2p" Nov 03 02:51:09.739115 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:09.738908 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c8ee6b-4hn2p" event={"ID":"ecaa74f8-112f-434d-9ebf-45e23e713cde","Type":"ContainerDied","Data":"9cec7d2f90e5559d26e6cce108ec43c2fa68b50d7b3fad75b8deb424b7f5e0d9"} Nov 03 02:51:09.739115 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:09.739038 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9cec7d2f90e5559d26e6cce108ec43c2fa68b50d7b3fad75b8deb424b7f5e0d9" Nov 03 02:51:10.685842 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:10.685812 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-56c7fb7568-t7rlj" Nov 03 02:51:10.841556 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:10.841513 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-666b7cb4b7-rq5h6"] Nov 03 02:51:15.841956 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:15.841875 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-666b7cb4b7-rq5h6" podUID="11118499-1a93-4e9d-8ecf-43116090ab21" containerName="payload-processing" containerID="cri-o://ec5fe58a252d4fde5e30569c574ed0bb1989646b89a94efd01fef0f96fc565d3" gracePeriod=25 Nov 03 02:51:16.099326 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:16.099262 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-666b7cb4b7-rq5h6" Nov 03 02:51:16.206652 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:16.206618 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11118499-1a93-4e9d-8ecf-43116090ab21-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/11118499-1a93-4e9d-8ecf-43116090ab21-routing-overlay-volume\") pod \"11118499-1a93-4e9d-8ecf-43116090ab21\" (UID: \"11118499-1a93-4e9d-8ecf-43116090ab21\") " Nov 03 02:51:16.206869 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:16.206794 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11118499-1a93-4e9d-8ecf-43116090ab21-kube-api-access-4trdh\" (UniqueName: \"kubernetes.io/projected/11118499-1a93-4e9d-8ecf-43116090ab21-kube-api-access-4trdh\") pod \"11118499-1a93-4e9d-8ecf-43116090ab21\" (UID: \"11118499-1a93-4e9d-8ecf-43116090ab21\") " Nov 03 02:51:16.207847 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:16.206952 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11118499-1a93-4e9d-8ecf-43116090ab21-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11118499-1a93-4e9d-8ecf-43116090ab21-plugins-config-volume\") pod \"11118499-1a93-4e9d-8ecf-43116090ab21\" (UID: \"11118499-1a93-4e9d-8ecf-43116090ab21\") " Nov 03 02:51:16.207847 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:16.206994 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11118499-1a93-4e9d-8ecf-43116090ab21-provider-credentials\" (UniqueName: \"kubernetes.io/projected/11118499-1a93-4e9d-8ecf-43116090ab21-provider-credentials\") pod \"11118499-1a93-4e9d-8ecf-43116090ab21\" (UID: \"11118499-1a93-4e9d-8ecf-43116090ab21\") " Nov 03 02:51:16.207847 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:16.207060 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11118499-1a93-4e9d-8ecf-43116090ab21-routing-overlay-volume" pod "11118499-1a93-4e9d-8ecf-43116090ab21" (UID: "11118499-1a93-4e9d-8ecf-43116090ab21"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:51:16.207847 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:16.207816 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11118499-1a93-4e9d-8ecf-43116090ab21-plugins-config-volume" pod "11118499-1a93-4e9d-8ecf-43116090ab21" (UID: "11118499-1a93-4e9d-8ecf-43116090ab21"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:51:16.208409 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:16.208332 5605 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11118499-1a93-4e9d-8ecf-43116090ab21-plugins-config-volume\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:51:16.208409 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:16.208382 5605 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/11118499-1a93-4e9d-8ecf-43116090ab21-routing-overlay-volume\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:51:16.210001 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:16.209947 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11118499-1a93-4e9d-8ecf-43116090ab21-kube-api-access-4trdh" pod "11118499-1a93-4e9d-8ecf-43116090ab21" (UID: "11118499-1a93-4e9d-8ecf-43116090ab21"). InnerVolumeSpecName "kube-api-access-4trdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:51:16.210540 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:16.210509 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11118499-1a93-4e9d-8ecf-43116090ab21-provider-credentials" pod "11118499-1a93-4e9d-8ecf-43116090ab21" (UID: "11118499-1a93-4e9d-8ecf-43116090ab21"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:51:16.309750 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:16.309707 5605 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/11118499-1a93-4e9d-8ecf-43116090ab21-provider-credentials\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:51:16.309750 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:16.309749 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4trdh\" (UniqueName: \"kubernetes.io/projected/11118499-1a93-4e9d-8ecf-43116090ab21-kube-api-access-4trdh\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:51:16.784653 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:16.784609 5605 generic.go:356] "Generic (PLEG): container finished" podID="11118499-1a93-4e9d-8ecf-43116090ab21" containerID="ec5fe58a252d4fde5e30569c574ed0bb1989646b89a94efd01fef0f96fc565d3" exitCode=0 Nov 03 02:51:16.784827 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:16.784655 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-666b7cb4b7-rq5h6" event={"ID":"11118499-1a93-4e9d-8ecf-43116090ab21","Type":"ContainerDied","Data":"ec5fe58a252d4fde5e30569c574ed0bb1989646b89a94efd01fef0f96fc565d3"} Nov 03 02:51:16.784827 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:16.784722 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-666b7cb4b7-rq5h6" Nov 03 02:51:16.784978 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:16.784952 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-666b7cb4b7-rq5h6" event={"ID":"11118499-1a93-4e9d-8ecf-43116090ab21","Type":"ContainerDied","Data":"ac93f127fbcf510fd512f06db472d3a30c88cf4de4ab63b0fabe8410b0751b39"} Nov 03 02:51:16.785053 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:16.785013 5605 scope.go:122] "RemoveContainer" containerID="ec5fe58a252d4fde5e30569c574ed0bb1989646b89a94efd01fef0f96fc565d3" Nov 03 02:51:16.798023 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:16.798001 5605 scope.go:122] "RemoveContainer" containerID="ec5fe58a252d4fde5e30569c574ed0bb1989646b89a94efd01fef0f96fc565d3" Nov 03 02:51:16.798460 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:51:16.798433 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec5fe58a252d4fde5e30569c574ed0bb1989646b89a94efd01fef0f96fc565d3\": container with ID starting with ec5fe58a252d4fde5e30569c574ed0bb1989646b89a94efd01fef0f96fc565d3 not found: ID does not exist" containerID="ec5fe58a252d4fde5e30569c574ed0bb1989646b89a94efd01fef0f96fc565d3" Nov 03 02:51:16.798568 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:16.798465 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec5fe58a252d4fde5e30569c574ed0bb1989646b89a94efd01fef0f96fc565d3"} err="failed to get container status \"ec5fe58a252d4fde5e30569c574ed0bb1989646b89a94efd01fef0f96fc565d3\": rpc error: code = NotFound desc = could not find container \"ec5fe58a252d4fde5e30569c574ed0bb1989646b89a94efd01fef0f96fc565d3\": container with ID starting with ec5fe58a252d4fde5e30569c574ed0bb1989646b89a94efd01fef0f96fc565d3 not found: ID does not exist" Nov 03 02:51:16.939304 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:16.939269 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-666b7cb4b7-rq5h6"] Nov 03 02:51:16.986053 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:16.986016 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-666b7cb4b7-rq5h6"] Nov 03 02:51:18.850845 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:18.850812 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11118499-1a93-4e9d-8ecf-43116090ab21" path="/var/lib/kubelet/pods/11118499-1a93-4e9d-8ecf-43116090ab21/volumes" Nov 03 02:51:19.119936 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:19.119848 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9"] Nov 03 02:51:19.120106 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:19.120067 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9" podUID="75df2f44-a9e5-47df-b676-6c0882b547e2" containerName="istio-proxy" containerID="cri-o://2d161114e12ffdc79185a7db8f0d4db5692014db8c44d17036380748a3ec35d1" gracePeriod=30 Nov 03 02:51:23.024606 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:23.024568 5605 patch_prober.go:29] interesting pod/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 02:51:23.024998 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:23.024623 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9" podUID="75df2f44-a9e5-47df-b676-6c0882b547e2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 02:51:23.914144 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:51:23.914108 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:51:24.377494 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:24.377471 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9" Nov 03 02:51:24.501658 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:24.501585 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75df2f44-a9e5-47df-b676-6c0882b547e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75df2f44-a9e5-47df-b676-6c0882b547e2-credential-socket\") pod \"75df2f44-a9e5-47df-b676-6c0882b547e2\" (UID: \"75df2f44-a9e5-47df-b676-6c0882b547e2\") " Nov 03 02:51:24.501658 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:24.501635 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75df2f44-a9e5-47df-b676-6c0882b547e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75df2f44-a9e5-47df-b676-6c0882b547e2-workload-certs\") pod \"75df2f44-a9e5-47df-b676-6c0882b547e2\" (UID: \"75df2f44-a9e5-47df-b676-6c0882b547e2\") " Nov 03 02:51:24.501891 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:24.501668 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/75df2f44-a9e5-47df-b676-6c0882b547e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75df2f44-a9e5-47df-b676-6c0882b547e2-istio-podinfo\") pod \"75df2f44-a9e5-47df-b676-6c0882b547e2\" (UID: \"75df2f44-a9e5-47df-b676-6c0882b547e2\") " Nov 03 02:51:24.501891 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:24.501692 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75df2f44-a9e5-47df-b676-6c0882b547e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75df2f44-a9e5-47df-b676-6c0882b547e2-workload-socket\") pod \"75df2f44-a9e5-47df-b676-6c0882b547e2\" (UID: \"75df2f44-a9e5-47df-b676-6c0882b547e2\") " Nov 03 02:51:24.501891 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:24.501725 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75df2f44-a9e5-47df-b676-6c0882b547e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75df2f44-a9e5-47df-b676-6c0882b547e2-istio-envoy\") pod \"75df2f44-a9e5-47df-b676-6c0882b547e2\" (UID: \"75df2f44-a9e5-47df-b676-6c0882b547e2\") " Nov 03 02:51:24.501891 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:24.501789 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75df2f44-a9e5-47df-b676-6c0882b547e2-kube-api-access-g2fpx\" (UniqueName: \"kubernetes.io/projected/75df2f44-a9e5-47df-b676-6c0882b547e2-kube-api-access-g2fpx\") pod \"75df2f44-a9e5-47df-b676-6c0882b547e2\" (UID: \"75df2f44-a9e5-47df-b676-6c0882b547e2\") " Nov 03 02:51:24.501891 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:24.501807 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75df2f44-a9e5-47df-b676-6c0882b547e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75df2f44-a9e5-47df-b676-6c0882b547e2-istio-data\") pod \"75df2f44-a9e5-47df-b676-6c0882b547e2\" (UID: \"75df2f44-a9e5-47df-b676-6c0882b547e2\") " Nov 03 02:51:24.501891 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:24.501815 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75df2f44-a9e5-47df-b676-6c0882b547e2-credential-socket" pod "75df2f44-a9e5-47df-b676-6c0882b547e2" (UID: "75df2f44-a9e5-47df-b676-6c0882b547e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:51:24.501891 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:24.501857 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75df2f44-a9e5-47df-b676-6c0882b547e2-istio-token\" (UniqueName: \"kubernetes.io/projected/75df2f44-a9e5-47df-b676-6c0882b547e2-istio-token\") pod \"75df2f44-a9e5-47df-b676-6c0882b547e2\" (UID: \"75df2f44-a9e5-47df-b676-6c0882b547e2\") " Nov 03 02:51:24.501891 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:24.501887 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75df2f44-a9e5-47df-b676-6c0882b547e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75df2f44-a9e5-47df-b676-6c0882b547e2-istiod-ca-cert\") pod \"75df2f44-a9e5-47df-b676-6c0882b547e2\" (UID: \"75df2f44-a9e5-47df-b676-6c0882b547e2\") " Nov 03 02:51:24.502321 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:24.501958 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75df2f44-a9e5-47df-b676-6c0882b547e2-workload-socket" pod "75df2f44-a9e5-47df-b676-6c0882b547e2" (UID: "75df2f44-a9e5-47df-b676-6c0882b547e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:51:24.502321 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:24.501985 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75df2f44-a9e5-47df-b676-6c0882b547e2-workload-certs" pod "75df2f44-a9e5-47df-b676-6c0882b547e2" (UID: "75df2f44-a9e5-47df-b676-6c0882b547e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:51:24.502321 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:24.502229 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75df2f44-a9e5-47df-b676-6c0882b547e2-istio-data" pod "75df2f44-a9e5-47df-b676-6c0882b547e2" (UID: "75df2f44-a9e5-47df-b676-6c0882b547e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:51:24.502504 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:24.502395 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75df2f44-a9e5-47df-b676-6c0882b547e2-istiod-ca-cert" pod "75df2f44-a9e5-47df-b676-6c0882b547e2" (UID: "75df2f44-a9e5-47df-b676-6c0882b547e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:51:24.502504 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:24.502419 5605 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75df2f44-a9e5-47df-b676-6c0882b547e2-istio-data\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:51:24.502504 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:24.502442 5605 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75df2f44-a9e5-47df-b676-6c0882b547e2-credential-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:51:24.502504 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:24.502457 5605 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75df2f44-a9e5-47df-b676-6c0882b547e2-workload-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:51:24.502504 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:24.502471 5605 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75df2f44-a9e5-47df-b676-6c0882b547e2-workload-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:51:24.504013 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:24.503990 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75df2f44-a9e5-47df-b676-6c0882b547e2-kube-api-access-g2fpx" pod "75df2f44-a9e5-47df-b676-6c0882b547e2" (UID: "75df2f44-a9e5-47df-b676-6c0882b547e2"). InnerVolumeSpecName "kube-api-access-g2fpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:51:24.504120 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:24.504036 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75df2f44-a9e5-47df-b676-6c0882b547e2-istio-envoy" pod "75df2f44-a9e5-47df-b676-6c0882b547e2" (UID: "75df2f44-a9e5-47df-b676-6c0882b547e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:51:24.504325 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:24.504308 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75df2f44-a9e5-47df-b676-6c0882b547e2-istio-podinfo" pod "75df2f44-a9e5-47df-b676-6c0882b547e2" (UID: "75df2f44-a9e5-47df-b676-6c0882b547e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:51:24.504414 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:24.504311 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75df2f44-a9e5-47df-b676-6c0882b547e2-istio-token" pod "75df2f44-a9e5-47df-b676-6c0882b547e2" (UID: "75df2f44-a9e5-47df-b676-6c0882b547e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:51:24.603739 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:24.603705 5605 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75df2f44-a9e5-47df-b676-6c0882b547e2-istio-podinfo\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:51:24.603739 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:24.603734 5605 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75df2f44-a9e5-47df-b676-6c0882b547e2-istio-envoy\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:51:24.603739 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:24.603744 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2fpx\" (UniqueName: \"kubernetes.io/projected/75df2f44-a9e5-47df-b676-6c0882b547e2-kube-api-access-g2fpx\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:51:24.603977 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:24.603754 5605 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75df2f44-a9e5-47df-b676-6c0882b547e2-istio-token\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:51:24.603977 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:24.603763 5605 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75df2f44-a9e5-47df-b676-6c0882b547e2-istiod-ca-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:51:24.833249 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:24.833215 5605 generic.go:356] "Generic (PLEG): container finished" podID="75df2f44-a9e5-47df-b676-6c0882b547e2" containerID="2d161114e12ffdc79185a7db8f0d4db5692014db8c44d17036380748a3ec35d1" exitCode=0 Nov 03 02:51:24.833429 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:24.833277 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9" Nov 03 02:51:24.833429 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:24.833311 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9" event={"ID":"75df2f44-a9e5-47df-b676-6c0882b547e2","Type":"ContainerDied","Data":"2d161114e12ffdc79185a7db8f0d4db5692014db8c44d17036380748a3ec35d1"} Nov 03 02:51:24.833429 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:24.833425 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9" event={"ID":"75df2f44-a9e5-47df-b676-6c0882b547e2","Type":"ContainerDied","Data":"e0e59e4a44883b9a6a64475681a1664962af5fdda882bddc519888e0162b3805"} Nov 03 02:51:24.833567 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:24.833459 5605 scope.go:122] "RemoveContainer" containerID="2d161114e12ffdc79185a7db8f0d4db5692014db8c44d17036380748a3ec35d1" Nov 03 02:51:24.844219 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:24.844196 5605 scope.go:122] "RemoveContainer" containerID="3d41884c4d69f4ffac583df5d18221b6659d52081231a80eb1a485f69c91d1c2" Nov 03 02:51:24.853825 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:24.853808 5605 scope.go:122] "RemoveContainer" containerID="2d161114e12ffdc79185a7db8f0d4db5692014db8c44d17036380748a3ec35d1" Nov 03 02:51:24.854078 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:51:24.854056 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d161114e12ffdc79185a7db8f0d4db5692014db8c44d17036380748a3ec35d1\": container with ID starting with 2d161114e12ffdc79185a7db8f0d4db5692014db8c44d17036380748a3ec35d1 not found: ID does not exist" containerID="2d161114e12ffdc79185a7db8f0d4db5692014db8c44d17036380748a3ec35d1" Nov 03 02:51:24.854168 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:24.854088 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d161114e12ffdc79185a7db8f0d4db5692014db8c44d17036380748a3ec35d1"} err="failed to get container status \"2d161114e12ffdc79185a7db8f0d4db5692014db8c44d17036380748a3ec35d1\": rpc error: code = NotFound desc = could not find container \"2d161114e12ffdc79185a7db8f0d4db5692014db8c44d17036380748a3ec35d1\": container with ID starting with 2d161114e12ffdc79185a7db8f0d4db5692014db8c44d17036380748a3ec35d1 not found: ID does not exist" Nov 03 02:51:24.854168 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:24.854111 5605 scope.go:122] "RemoveContainer" containerID="3d41884c4d69f4ffac583df5d18221b6659d52081231a80eb1a485f69c91d1c2" Nov 03 02:51:24.854417 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:51:24.854391 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d41884c4d69f4ffac583df5d18221b6659d52081231a80eb1a485f69c91d1c2\": container with ID starting with 3d41884c4d69f4ffac583df5d18221b6659d52081231a80eb1a485f69c91d1c2 not found: ID does not exist" containerID="3d41884c4d69f4ffac583df5d18221b6659d52081231a80eb1a485f69c91d1c2" Nov 03 02:51:24.854538 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:24.854420 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d41884c4d69f4ffac583df5d18221b6659d52081231a80eb1a485f69c91d1c2"} err="failed to get container status \"3d41884c4d69f4ffac583df5d18221b6659d52081231a80eb1a485f69c91d1c2\": rpc error: code = NotFound desc = could not find container \"3d41884c4d69f4ffac583df5d18221b6659d52081231a80eb1a485f69c91d1c2\": container with ID starting with 3d41884c4d69f4ffac583df5d18221b6659d52081231a80eb1a485f69c91d1c2 not found: ID does not exist" Nov 03 02:51:24.992007 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:24.991974 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9"] Nov 03 02:51:24.998708 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:24.998681 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-e3cc05-openshift-default-697c8b8fd9-b5rs9"] Nov 03 02:51:26.589416 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:26.589386 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-68686f579-ntp6g"] Nov 03 02:51:26.589863 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:26.589848 5605 cpu_manager.go:414] "containerMap: removing container" podUID="ecaa74f8-112f-434d-9ebf-45e23e713cde" containerName="revoke-keys" Nov 03 02:51:26.589912 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:26.589864 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecaa74f8-112f-434d-9ebf-45e23e713cde" containerName="revoke-keys" Nov 03 02:51:26.589912 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:26.589877 5605 cpu_manager.go:414] "containerMap: removing container" podUID="11118499-1a93-4e9d-8ecf-43116090ab21" containerName="payload-processing" Nov 03 02:51:26.589912 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:26.589885 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11118499-1a93-4e9d-8ecf-43116090ab21" containerName="payload-processing" Nov 03 02:51:26.590015 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:26.589922 5605 cpu_manager.go:414] "containerMap: removing container" podUID="75df2f44-a9e5-47df-b676-6c0882b547e2" containerName="istio-proxy" Nov 03 02:51:26.590015 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:26.589927 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75df2f44-a9e5-47df-b676-6c0882b547e2" containerName="istio-proxy" Nov 03 02:51:26.590078 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:26.590019 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="11118499-1a93-4e9d-8ecf-43116090ab21" containerName="payload-processing" Nov 03 02:51:26.590078 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:26.590037 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="75df2f44-a9e5-47df-b676-6c0882b547e2" containerName="istio-proxy" Nov 03 02:51:26.590078 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:26.590048 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="75df2f44-a9e5-47df-b676-6c0882b547e2" containerName="istio-proxy" Nov 03 02:51:26.590078 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:26.590056 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecaa74f8-112f-434d-9ebf-45e23e713cde" containerName="revoke-keys" Nov 03 02:51:26.594753 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:26.594731 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-68686f579-ntp6g" Nov 03 02:51:26.620385 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:26.620328 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d89cb81e-1adf-4e6c-82ea-dde66712aab7-plugins-config-volume\") pod \"payload-processing-external-model-68686f579-ntp6g\" (UID: \"d89cb81e-1adf-4e6c-82ea-dde66712aab7\") " pod="models-as-a-service/payload-processing-external-model-68686f579-ntp6g" Nov 03 02:51:26.620512 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:26.620492 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snm7b\" (UniqueName: \"kubernetes.io/projected/d89cb81e-1adf-4e6c-82ea-dde66712aab7-kube-api-access-snm7b\") pod \"payload-processing-external-model-68686f579-ntp6g\" (UID: \"d89cb81e-1adf-4e6c-82ea-dde66712aab7\") " pod="models-as-a-service/payload-processing-external-model-68686f579-ntp6g" Nov 03 02:51:26.620577 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:26.620524 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d89cb81e-1adf-4e6c-82ea-dde66712aab7-provider-credentials\") pod \"payload-processing-external-model-68686f579-ntp6g\" (UID: \"d89cb81e-1adf-4e6c-82ea-dde66712aab7\") " pod="models-as-a-service/payload-processing-external-model-68686f579-ntp6g" Nov 03 02:51:26.620687 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:26.620667 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d89cb81e-1adf-4e6c-82ea-dde66712aab7-routing-overlay-volume\") pod \"payload-processing-external-model-68686f579-ntp6g\" (UID: \"d89cb81e-1adf-4e6c-82ea-dde66712aab7\") " pod="models-as-a-service/payload-processing-external-model-68686f579-ntp6g" Nov 03 02:51:26.647228 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:26.647198 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-38bfa1738e15\"" Nov 03 02:51:26.650031 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:26.650010 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-68686f579-ntp6g"] Nov 03 02:51:26.721183 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:26.721160 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d89cb81e-1adf-4e6c-82ea-dde66712aab7-routing-overlay-volume\") pod \"payload-processing-external-model-68686f579-ntp6g\" (UID: \"d89cb81e-1adf-4e6c-82ea-dde66712aab7\") " pod="models-as-a-service/payload-processing-external-model-68686f579-ntp6g" Nov 03 02:51:26.721280 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:26.721193 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d89cb81e-1adf-4e6c-82ea-dde66712aab7-plugins-config-volume\") pod \"payload-processing-external-model-68686f579-ntp6g\" (UID: \"d89cb81e-1adf-4e6c-82ea-dde66712aab7\") " pod="models-as-a-service/payload-processing-external-model-68686f579-ntp6g" Nov 03 02:51:26.721280 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:26.721253 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snm7b\" (UniqueName: \"kubernetes.io/projected/d89cb81e-1adf-4e6c-82ea-dde66712aab7-kube-api-access-snm7b\") pod \"payload-processing-external-model-68686f579-ntp6g\" (UID: \"d89cb81e-1adf-4e6c-82ea-dde66712aab7\") " pod="models-as-a-service/payload-processing-external-model-68686f579-ntp6g" Nov 03 02:51:26.721280 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:26.721271 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d89cb81e-1adf-4e6c-82ea-dde66712aab7-provider-credentials\") pod \"payload-processing-external-model-68686f579-ntp6g\" (UID: \"d89cb81e-1adf-4e6c-82ea-dde66712aab7\") " pod="models-as-a-service/payload-processing-external-model-68686f579-ntp6g" Nov 03 02:51:26.721852 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:26.721834 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d89cb81e-1adf-4e6c-82ea-dde66712aab7-routing-overlay-volume\") pod \"payload-processing-external-model-68686f579-ntp6g\" (UID: \"d89cb81e-1adf-4e6c-82ea-dde66712aab7\") " pod="models-as-a-service/payload-processing-external-model-68686f579-ntp6g" Nov 03 02:51:26.722024 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:26.722009 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d89cb81e-1adf-4e6c-82ea-dde66712aab7-plugins-config-volume\") pod \"payload-processing-external-model-68686f579-ntp6g\" (UID: \"d89cb81e-1adf-4e6c-82ea-dde66712aab7\") " pod="models-as-a-service/payload-processing-external-model-68686f579-ntp6g" Nov 03 02:51:26.723769 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:26.723750 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d89cb81e-1adf-4e6c-82ea-dde66712aab7-provider-credentials\") pod \"payload-processing-external-model-68686f579-ntp6g\" (UID: \"d89cb81e-1adf-4e6c-82ea-dde66712aab7\") " pod="models-as-a-service/payload-processing-external-model-68686f579-ntp6g" Nov 03 02:51:26.775307 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:26.775282 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snm7b\" (UniqueName: \"kubernetes.io/projected/d89cb81e-1adf-4e6c-82ea-dde66712aab7-kube-api-access-snm7b\") pod \"payload-processing-external-model-68686f579-ntp6g\" (UID: \"d89cb81e-1adf-4e6c-82ea-dde66712aab7\") " pod="models-as-a-service/payload-processing-external-model-68686f579-ntp6g" Nov 03 02:51:26.849045 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:26.848977 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75df2f44-a9e5-47df-b676-6c0882b547e2" path="/var/lib/kubelet/pods/75df2f44-a9e5-47df-b676-6c0882b547e2/volumes" Nov 03 02:51:26.908697 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:26.908670 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-68686f579-ntp6g" Nov 03 02:51:27.281546 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:27.281467 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-68686f579-ntp6g"] Nov 03 02:51:27.286927 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:51:27.286897 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd89cb81e_1adf_4e6c_82ea_dde66712aab7.slice/crio-42b0f290bcadbdd2c859596babbfc446eb1fc1cdee140ae8d92614b7b8831e41 WatchSource:0}: Error finding container 42b0f290bcadbdd2c859596babbfc446eb1fc1cdee140ae8d92614b7b8831e41: Status 404 returned error can't find the container with id 42b0f290bcadbdd2c859596babbfc446eb1fc1cdee140ae8d92614b7b8831e41 Nov 03 02:51:27.852077 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:27.852043 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-68686f579-ntp6g" event={"ID":"d89cb81e-1adf-4e6c-82ea-dde66712aab7","Type":"ContainerStarted","Data":"8734436811ef402aa8a89f8cd7747828eea9ff341d20ce30ced6db251b33993d"} Nov 03 02:51:27.852560 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:27.852096 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-68686f579-ntp6g" event={"ID":"d89cb81e-1adf-4e6c-82ea-dde66712aab7","Type":"ContainerStarted","Data":"42b0f290bcadbdd2c859596babbfc446eb1fc1cdee140ae8d92614b7b8831e41"} Nov 03 02:51:27.852560 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:27.852140 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-68686f579-ntp6g" Nov 03 02:51:27.959654 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:27.959611 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-68686f579-ntp6g" podStartSLOduration=1.959596947 podStartE2EDuration="1.959596947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:51:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:51:27.958059894 +0000 UTC m=+3565.217258049" watchObservedRunningTime="2026-11-03 02:51:27.959596947 +0000 UTC m=+3565.218795103" Nov 03 02:51:34.062464 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:34.062427 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj"] Nov 03 02:51:34.062857 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:34.062626 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj" podUID="9aecc649-014f-45ef-849d-27fad4f34933" containerName="istio-proxy" containerID="cri-o://602dcaadbe89b58dd913ddc8e28386a8308228aba5b09e942a429efb0cfa15cb" gracePeriod=30 Nov 03 02:51:36.424478 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:36.424438 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7dn2l"] Nov 03 02:51:36.424964 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:36.424910 5605 cpu_manager.go:414] "containerMap: removing container" podUID="75df2f44-a9e5-47df-b676-6c0882b547e2" containerName="istio-proxy" Nov 03 02:51:36.424964 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:36.424921 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75df2f44-a9e5-47df-b676-6c0882b547e2" containerName="istio-proxy" Nov 03 02:51:36.428270 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:36.428252 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7dn2l" Nov 03 02:51:36.471299 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:36.471275 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jgcjh"] Nov 03 02:51:36.474912 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:36.474894 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jgcjh" Nov 03 02:51:36.514584 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:36.514552 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80871945-cbe2-40b4-8715-472c062fc27f-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jgcjh\" (UID: \"80871945-cbe2-40b4-8715-472c062fc27f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jgcjh" Nov 03 02:51:36.514710 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:36.514588 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80871945-cbe2-40b4-8715-472c062fc27f-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jgcjh\" (UID: \"80871945-cbe2-40b4-8715-472c062fc27f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jgcjh" Nov 03 02:51:36.514710 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:36.514619 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21200be4-59d6-4955-8d62-5394f4de28df-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7dn2l\" (UID: \"21200be4-59d6-4955-8d62-5394f4de28df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7dn2l" Nov 03 02:51:36.514789 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:36.514742 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21200be4-59d6-4955-8d62-5394f4de28df-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7dn2l\" (UID: \"21200be4-59d6-4955-8d62-5394f4de28df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7dn2l" Nov 03 02:51:36.514789 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:36.514772 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2trql\" (UniqueName: \"kubernetes.io/projected/80871945-cbe2-40b4-8715-472c062fc27f-kube-api-access-2trql\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jgcjh\" (UID: \"80871945-cbe2-40b4-8715-472c062fc27f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jgcjh" Nov 03 02:51:36.514860 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:36.514821 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80871945-cbe2-40b4-8715-472c062fc27f-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jgcjh\" (UID: \"80871945-cbe2-40b4-8715-472c062fc27f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jgcjh" Nov 03 02:51:36.514918 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:36.514897 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21200be4-59d6-4955-8d62-5394f4de28df-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7dn2l\" (UID: \"21200be4-59d6-4955-8d62-5394f4de28df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7dn2l" Nov 03 02:51:36.514954 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:36.514925 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21200be4-59d6-4955-8d62-5394f4de28df-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7dn2l\" (UID: \"21200be4-59d6-4955-8d62-5394f4de28df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7dn2l" Nov 03 02:51:36.514954 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:36.514944 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80871945-cbe2-40b4-8715-472c062fc27f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jgcjh\" (UID: \"80871945-cbe2-40b4-8715-472c062fc27f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jgcjh" Nov 03 02:51:36.515022 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:36.514981 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80871945-cbe2-40b4-8715-472c062fc27f-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jgcjh\" (UID: \"80871945-cbe2-40b4-8715-472c062fc27f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jgcjh" Nov 03 02:51:36.515083 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:36.515070 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80871945-cbe2-40b4-8715-472c062fc27f-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jgcjh\" (UID: \"80871945-cbe2-40b4-8715-472c062fc27f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jgcjh" Nov 03 02:51:36.515119 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:36.515091 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21200be4-59d6-4955-8d62-5394f4de28df-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7dn2l\" (UID: \"21200be4-59d6-4955-8d62-5394f4de28df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7dn2l" Nov 03 02:51:36.515119 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:36.515110 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-749g6\" (UniqueName: \"kubernetes.io/projected/21200be4-59d6-4955-8d62-5394f4de28df-kube-api-access-749g6\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7dn2l\" (UID: \"21200be4-59d6-4955-8d62-5394f4de28df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7dn2l" Nov 03 02:51:36.515201 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:36.515132 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21200be4-59d6-4955-8d62-5394f4de28df-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7dn2l\" (UID: \"21200be4-59d6-4955-8d62-5394f4de28df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7dn2l" Nov 03 02:51:36.515290 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:36.515275 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21200be4-59d6-4955-8d62-5394f4de28df-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7dn2l\" (UID: \"21200be4-59d6-4955-8d62-5394f4de28df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7dn2l" Nov 03 02:51:36.515353 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:36.515310 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80871945-cbe2-40b4-8715-472c062fc27f-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jgcjh\" (UID: \"80871945-cbe2-40b4-8715-472c062fc27f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jgcjh" Nov 03 02:51:36.515465 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:36.515450 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21200be4-59d6-4955-8d62-5394f4de28df-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7dn2l\" (UID: \"21200be4-59d6-4955-8d62-5394f4de28df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7dn2l" Nov 03 02:51:36.515534 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:36.515485 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80871945-cbe2-40b4-8715-472c062fc27f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jgcjh\" (UID: \"80871945-cbe2-40b4-8715-472c062fc27f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jgcjh" Nov 03 02:51:36.515718 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:36.515701 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-fwpk8\"" Nov 03 02:51:36.526538 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:36.526518 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7dn2l"] Nov 03 02:51:36.578862 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:36.578831 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jgcjh"] Nov 03 02:51:36.616319 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:36.616293 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80871945-cbe2-40b4-8715-472c062fc27f-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jgcjh\" (UID: \"80871945-cbe2-40b4-8715-472c062fc27f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jgcjh" Nov 03 02:51:36.616502 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:36.616324 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21200be4-59d6-4955-8d62-5394f4de28df-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7dn2l\" (UID: \"21200be4-59d6-4955-8d62-5394f4de28df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7dn2l" Nov 03 02:51:36.616502 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:36.616345 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-749g6\" (UniqueName: \"kubernetes.io/projected/21200be4-59d6-4955-8d62-5394f4de28df-kube-api-access-749g6\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7dn2l\" (UID: \"21200be4-59d6-4955-8d62-5394f4de28df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7dn2l" Nov 03 02:51:36.616502 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:36.616396 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21200be4-59d6-4955-8d62-5394f4de28df-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7dn2l\" (UID: \"21200be4-59d6-4955-8d62-5394f4de28df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7dn2l" Nov 03 02:51:36.616502 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:36.616438 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21200be4-59d6-4955-8d62-5394f4de28df-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7dn2l\" (UID: \"21200be4-59d6-4955-8d62-5394f4de28df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7dn2l" Nov 03 02:51:36.616502 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:36.616465 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80871945-cbe2-40b4-8715-472c062fc27f-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jgcjh\" (UID: \"80871945-cbe2-40b4-8715-472c062fc27f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jgcjh" Nov 03 02:51:36.616776 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:36.616542 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21200be4-59d6-4955-8d62-5394f4de28df-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7dn2l\" (UID: \"21200be4-59d6-4955-8d62-5394f4de28df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7dn2l" Nov 03 02:51:36.616776 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:36.616571 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80871945-cbe2-40b4-8715-472c062fc27f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jgcjh\" (UID: \"80871945-cbe2-40b4-8715-472c062fc27f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jgcjh" Nov 03 02:51:36.616776 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:36.616602 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80871945-cbe2-40b4-8715-472c062fc27f-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jgcjh\" (UID: \"80871945-cbe2-40b4-8715-472c062fc27f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jgcjh" Nov 03 02:51:36.616776 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:36.616631 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80871945-cbe2-40b4-8715-472c062fc27f-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jgcjh\" (UID: \"80871945-cbe2-40b4-8715-472c062fc27f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jgcjh" Nov 03 02:51:36.616776 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:36.616661 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21200be4-59d6-4955-8d62-5394f4de28df-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7dn2l\" (UID: \"21200be4-59d6-4955-8d62-5394f4de28df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7dn2l" Nov 03 02:51:36.616776 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:36.616701 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21200be4-59d6-4955-8d62-5394f4de28df-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7dn2l\" (UID: \"21200be4-59d6-4955-8d62-5394f4de28df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7dn2l" Nov 03 02:51:36.616776 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:36.616731 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2trql\" (UniqueName: \"kubernetes.io/projected/80871945-cbe2-40b4-8715-472c062fc27f-kube-api-access-2trql\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jgcjh\" (UID: \"80871945-cbe2-40b4-8715-472c062fc27f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jgcjh" Nov 03 02:51:36.616776 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:36.616766 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80871945-cbe2-40b4-8715-472c062fc27f-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jgcjh\" (UID: \"80871945-cbe2-40b4-8715-472c062fc27f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jgcjh" Nov 03 02:51:36.617223 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:36.616829 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80871945-cbe2-40b4-8715-472c062fc27f-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jgcjh\" (UID: \"80871945-cbe2-40b4-8715-472c062fc27f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jgcjh" Nov 03 02:51:36.617223 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:36.616834 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80871945-cbe2-40b4-8715-472c062fc27f-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jgcjh\" (UID: \"80871945-cbe2-40b4-8715-472c062fc27f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jgcjh" Nov 03 02:51:36.617223 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:36.616893 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21200be4-59d6-4955-8d62-5394f4de28df-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7dn2l\" (UID: \"21200be4-59d6-4955-8d62-5394f4de28df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7dn2l" Nov 03 02:51:36.617223 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:36.616901 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21200be4-59d6-4955-8d62-5394f4de28df-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7dn2l\" (UID: \"21200be4-59d6-4955-8d62-5394f4de28df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7dn2l" Nov 03 02:51:36.617223 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:36.616920 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21200be4-59d6-4955-8d62-5394f4de28df-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7dn2l\" (UID: \"21200be4-59d6-4955-8d62-5394f4de28df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7dn2l" Nov 03 02:51:36.617223 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:36.616960 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80871945-cbe2-40b4-8715-472c062fc27f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jgcjh\" (UID: \"80871945-cbe2-40b4-8715-472c062fc27f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jgcjh" Nov 03 02:51:36.617223 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:36.617157 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80871945-cbe2-40b4-8715-472c062fc27f-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jgcjh\" (UID: \"80871945-cbe2-40b4-8715-472c062fc27f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jgcjh" Nov 03 02:51:36.617654 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:36.617234 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21200be4-59d6-4955-8d62-5394f4de28df-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7dn2l\" (UID: \"21200be4-59d6-4955-8d62-5394f4de28df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7dn2l" Nov 03 02:51:36.617654 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:36.617339 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80871945-cbe2-40b4-8715-472c062fc27f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jgcjh\" (UID: \"80871945-cbe2-40b4-8715-472c062fc27f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jgcjh" Nov 03 02:51:36.617654 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:36.617346 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80871945-cbe2-40b4-8715-472c062fc27f-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jgcjh\" (UID: \"80871945-cbe2-40b4-8715-472c062fc27f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jgcjh" Nov 03 02:51:36.617654 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:36.617533 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21200be4-59d6-4955-8d62-5394f4de28df-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7dn2l\" (UID: \"21200be4-59d6-4955-8d62-5394f4de28df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7dn2l" Nov 03 02:51:36.617654 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:36.617533 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80871945-cbe2-40b4-8715-472c062fc27f-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jgcjh\" (UID: \"80871945-cbe2-40b4-8715-472c062fc27f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jgcjh" Nov 03 02:51:36.617957 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:36.617777 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21200be4-59d6-4955-8d62-5394f4de28df-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7dn2l\" (UID: \"21200be4-59d6-4955-8d62-5394f4de28df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7dn2l" Nov 03 02:51:36.618060 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:36.618036 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21200be4-59d6-4955-8d62-5394f4de28df-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7dn2l\" (UID: \"21200be4-59d6-4955-8d62-5394f4de28df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7dn2l" Nov 03 02:51:36.619005 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:36.618978 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21200be4-59d6-4955-8d62-5394f4de28df-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7dn2l\" (UID: \"21200be4-59d6-4955-8d62-5394f4de28df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7dn2l" Nov 03 02:51:36.619278 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:36.619258 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21200be4-59d6-4955-8d62-5394f4de28df-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7dn2l\" (UID: \"21200be4-59d6-4955-8d62-5394f4de28df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7dn2l" Nov 03 02:51:36.619399 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:36.619378 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80871945-cbe2-40b4-8715-472c062fc27f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jgcjh\" (UID: \"80871945-cbe2-40b4-8715-472c062fc27f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jgcjh" Nov 03 02:51:36.619493 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:36.619478 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80871945-cbe2-40b4-8715-472c062fc27f-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jgcjh\" (UID: \"80871945-cbe2-40b4-8715-472c062fc27f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jgcjh" Nov 03 02:51:36.671721 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:36.671696 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-749g6\" (UniqueName: \"kubernetes.io/projected/21200be4-59d6-4955-8d62-5394f4de28df-kube-api-access-749g6\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7dn2l\" (UID: \"21200be4-59d6-4955-8d62-5394f4de28df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7dn2l" Nov 03 02:51:36.671904 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:36.671886 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2trql\" (UniqueName: \"kubernetes.io/projected/80871945-cbe2-40b4-8715-472c062fc27f-kube-api-access-2trql\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jgcjh\" (UID: \"80871945-cbe2-40b4-8715-472c062fc27f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jgcjh" Nov 03 02:51:36.672169 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:36.672146 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21200be4-59d6-4955-8d62-5394f4de28df-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7dn2l\" (UID: \"21200be4-59d6-4955-8d62-5394f4de28df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7dn2l" Nov 03 02:51:36.673055 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:36.673037 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80871945-cbe2-40b4-8715-472c062fc27f-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jgcjh\" (UID: \"80871945-cbe2-40b4-8715-472c062fc27f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jgcjh" Nov 03 02:51:36.742339 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:36.742280 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7dn2l" Nov 03 02:51:36.787457 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:36.787424 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jgcjh" Nov 03 02:51:37.167771 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:37.167738 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7dn2l"] Nov 03 02:51:37.210902 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:51:37.210870 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21200be4_59d6_4955_8d62_5394f4de28df.slice/crio-eb9f4a497f38a8fa8dfcbae222653d3426b236c9ae3fbba48ec1cc2e0f6e8598 WatchSource:0}: Error finding container eb9f4a497f38a8fa8dfcbae222653d3426b236c9ae3fbba48ec1cc2e0f6e8598: Status 404 returned error can't find the container with id eb9f4a497f38a8fa8dfcbae222653d3426b236c9ae3fbba48ec1cc2e0f6e8598 Nov 03 02:51:37.213577 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:37.213541 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:51:37.213677 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:37.213611 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:51:37.213677 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:37.213643 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:51:37.221667 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:37.221643 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jgcjh"] Nov 03 02:51:37.225978 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:51:37.225949 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80871945_cbe2_40b4_8715_472c062fc27f.slice/crio-bbb0bb2540c66c44d1aa7feba3c94f0ddd7e96e816fe055713a08018b5031234 WatchSource:0}: Error finding container bbb0bb2540c66c44d1aa7feba3c94f0ddd7e96e816fe055713a08018b5031234: Status 404 returned error can't find the container with id bbb0bb2540c66c44d1aa7feba3c94f0ddd7e96e816fe055713a08018b5031234 Nov 03 02:51:37.228532 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:37.228504 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:51:37.228651 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:37.228563 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:51:37.228651 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:37.228594 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:51:37.919791 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:37.919702 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jgcjh" event={"ID":"80871945-cbe2-40b4-8715-472c062fc27f","Type":"ContainerStarted","Data":"ec8627709f986d040ae797ce67bac125af0f73f36320ee6e8309cfc60c2e5de9"} Nov 03 02:51:37.919791 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:37.919757 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jgcjh" event={"ID":"80871945-cbe2-40b4-8715-472c062fc27f","Type":"ContainerStarted","Data":"bbb0bb2540c66c44d1aa7feba3c94f0ddd7e96e816fe055713a08018b5031234"} Nov 03 02:51:37.922325 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:37.922293 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7dn2l" event={"ID":"21200be4-59d6-4955-8d62-5394f4de28df","Type":"ContainerStarted","Data":"c7d79248d1b67b2fcf22e4ea304961bf1019daae99df51f335af48dd78b637ca"} Nov 03 02:51:37.922466 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:37.922348 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7dn2l" event={"ID":"21200be4-59d6-4955-8d62-5394f4de28df","Type":"ContainerStarted","Data":"eb9f4a497f38a8fa8dfcbae222653d3426b236c9ae3fbba48ec1cc2e0f6e8598"} Nov 03 02:51:38.033472 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:38.033413 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jgcjh" podStartSLOduration=2.033397693 podStartE2EDuration="2.033397693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:51:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:51:38.032124124 +0000 UTC m=+3575.291322278" watchObservedRunningTime="2026-11-03 02:51:38.033397693 +0000 UTC m=+3575.292595846" Nov 03 02:51:38.742807 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:38.742770 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7dn2l" Nov 03 02:51:38.744502 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:38.744474 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-7dn2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body= Nov 03 02:51:38.744664 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:38.744527 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7dn2l" podUID="21200be4-59d6-4955-8d62-5394f4de28df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" Nov 03 02:51:38.803382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:38.799196 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jgcjh" Nov 03 02:51:38.803382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:38.799279 5605 patch_prober.go:29] interesting pod/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 02:51:38.803382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:38.799318 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj" podUID="9aecc649-014f-45ef-849d-27fad4f34933" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 02:51:38.805898 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:38.805459 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-jgcjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body= Nov 03 02:51:38.805898 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:38.805513 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jgcjh" podUID="80871945-cbe2-40b4-8715-472c062fc27f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" Nov 03 02:51:38.908421 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:38.904538 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-68686f579-ntp6g" Nov 03 02:51:39.028126 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:39.028010 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7dn2l" podStartSLOduration=3.027991048 podStartE2EDuration="3.027991048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:51:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:51:38.175571518 +0000 UTC m=+3575.434769672" watchObservedRunningTime="2026-11-03 02:51:39.027991048 +0000 UTC m=+3576.287189203" Nov 03 02:51:39.110418 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:39.088070 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-56c7fb7568-t7rlj"] Nov 03 02:51:39.403377 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:39.397983 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj" Nov 03 02:51:39.469263 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:39.454333 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9aecc649-014f-45ef-849d-27fad4f34933-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9aecc649-014f-45ef-849d-27fad4f34933-workload-certs\") pod \"9aecc649-014f-45ef-849d-27fad4f34933\" (UID: \"9aecc649-014f-45ef-849d-27fad4f34933\") " Nov 03 02:51:39.469263 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:39.454421 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9aecc649-014f-45ef-849d-27fad4f34933-istio-token\" (UniqueName: \"kubernetes.io/projected/9aecc649-014f-45ef-849d-27fad4f34933-istio-token\") pod \"9aecc649-014f-45ef-849d-27fad4f34933\" (UID: \"9aecc649-014f-45ef-849d-27fad4f34933\") " Nov 03 02:51:39.469263 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:39.454468 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9aecc649-014f-45ef-849d-27fad4f34933-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9aecc649-014f-45ef-849d-27fad4f34933-istio-podinfo\") pod \"9aecc649-014f-45ef-849d-27fad4f34933\" (UID: \"9aecc649-014f-45ef-849d-27fad4f34933\") " Nov 03 02:51:39.469263 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:39.454511 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9aecc649-014f-45ef-849d-27fad4f34933-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aecc649-014f-45ef-849d-27fad4f34933-workload-socket\") pod \"9aecc649-014f-45ef-849d-27fad4f34933\" (UID: \"9aecc649-014f-45ef-849d-27fad4f34933\") " Nov 03 02:51:39.469263 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:39.454539 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9aecc649-014f-45ef-849d-27fad4f34933-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9aecc649-014f-45ef-849d-27fad4f34933-istiod-ca-cert\") pod \"9aecc649-014f-45ef-849d-27fad4f34933\" (UID: \"9aecc649-014f-45ef-849d-27fad4f34933\") " Nov 03 02:51:39.469263 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:39.454594 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9aecc649-014f-45ef-849d-27fad4f34933-kube-api-access-92g55\" (UniqueName: \"kubernetes.io/projected/9aecc649-014f-45ef-849d-27fad4f34933-kube-api-access-92g55\") pod \"9aecc649-014f-45ef-849d-27fad4f34933\" (UID: \"9aecc649-014f-45ef-849d-27fad4f34933\") " Nov 03 02:51:39.469263 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:39.454704 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9aecc649-014f-45ef-849d-27fad4f34933-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9aecc649-014f-45ef-849d-27fad4f34933-istio-data\") pod \"9aecc649-014f-45ef-849d-27fad4f34933\" (UID: \"9aecc649-014f-45ef-849d-27fad4f34933\") " Nov 03 02:51:39.469263 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:39.454833 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9aecc649-014f-45ef-849d-27fad4f34933-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aecc649-014f-45ef-849d-27fad4f34933-credential-socket\") pod \"9aecc649-014f-45ef-849d-27fad4f34933\" (UID: \"9aecc649-014f-45ef-849d-27fad4f34933\") " Nov 03 02:51:39.469263 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:39.454860 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9aecc649-014f-45ef-849d-27fad4f34933-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9aecc649-014f-45ef-849d-27fad4f34933-istio-envoy\") pod \"9aecc649-014f-45ef-849d-27fad4f34933\" (UID: \"9aecc649-014f-45ef-849d-27fad4f34933\") " Nov 03 02:51:39.469263 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:39.464984 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aecc649-014f-45ef-849d-27fad4f34933-workload-socket" pod "9aecc649-014f-45ef-849d-27fad4f34933" (UID: "9aecc649-014f-45ef-849d-27fad4f34933"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:51:39.469263 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:39.465337 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aecc649-014f-45ef-849d-27fad4f34933-istio-data" pod "9aecc649-014f-45ef-849d-27fad4f34933" (UID: "9aecc649-014f-45ef-849d-27fad4f34933"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:51:39.469263 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:39.465652 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9aecc649-014f-45ef-849d-27fad4f34933-istiod-ca-cert" pod "9aecc649-014f-45ef-849d-27fad4f34933" (UID: "9aecc649-014f-45ef-849d-27fad4f34933"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:51:39.470086 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:39.469420 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aecc649-014f-45ef-849d-27fad4f34933-credential-socket" pod "9aecc649-014f-45ef-849d-27fad4f34933" (UID: "9aecc649-014f-45ef-849d-27fad4f34933"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:51:39.470086 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:39.469931 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aecc649-014f-45ef-849d-27fad4f34933-workload-certs" pod "9aecc649-014f-45ef-849d-27fad4f34933" (UID: "9aecc649-014f-45ef-849d-27fad4f34933"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:51:39.498088 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:39.498049 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aecc649-014f-45ef-849d-27fad4f34933-kube-api-access-92g55" pod "9aecc649-014f-45ef-849d-27fad4f34933" (UID: "9aecc649-014f-45ef-849d-27fad4f34933"). InnerVolumeSpecName "kube-api-access-92g55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:51:39.503386 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:39.501475 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aecc649-014f-45ef-849d-27fad4f34933-istio-token" pod "9aecc649-014f-45ef-849d-27fad4f34933" (UID: "9aecc649-014f-45ef-849d-27fad4f34933"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:51:39.503685 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:39.503640 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9aecc649-014f-45ef-849d-27fad4f34933-istio-podinfo" pod "9aecc649-014f-45ef-849d-27fad4f34933" (UID: "9aecc649-014f-45ef-849d-27fad4f34933"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:51:39.510390 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:39.509460 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aecc649-014f-45ef-849d-27fad4f34933-istio-envoy" pod "9aecc649-014f-45ef-849d-27fad4f34933" (UID: "9aecc649-014f-45ef-849d-27fad4f34933"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:51:39.595438 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:39.594608 5605 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9aecc649-014f-45ef-849d-27fad4f34933-istio-data\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:51:39.595438 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:39.594645 5605 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aecc649-014f-45ef-849d-27fad4f34933-credential-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:51:39.595438 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:39.594661 5605 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9aecc649-014f-45ef-849d-27fad4f34933-istio-envoy\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:51:39.595438 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:39.594687 5605 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9aecc649-014f-45ef-849d-27fad4f34933-workload-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:51:39.595438 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:39.594704 5605 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9aecc649-014f-45ef-849d-27fad4f34933-istio-token\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:51:39.595438 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:39.594717 5605 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9aecc649-014f-45ef-849d-27fad4f34933-istio-podinfo\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:51:39.595438 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:39.594730 5605 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aecc649-014f-45ef-849d-27fad4f34933-workload-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:51:39.603337 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:39.594743 5605 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9aecc649-014f-45ef-849d-27fad4f34933-istiod-ca-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:51:39.603337 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:39.602725 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92g55\" (UniqueName: \"kubernetes.io/projected/9aecc649-014f-45ef-849d-27fad4f34933-kube-api-access-92g55\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:51:39.742878 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:39.742842 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-7dn2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body= Nov 03 02:51:39.743049 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:39.742903 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7dn2l" podUID="21200be4-59d6-4955-8d62-5394f4de28df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" Nov 03 02:51:39.791833 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:39.791715 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-jgcjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body= Nov 03 02:51:39.791833 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:39.791773 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jgcjh" podUID="80871945-cbe2-40b4-8715-472c062fc27f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" Nov 03 02:51:39.933827 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:39.933738 5605 generic.go:356] "Generic (PLEG): container finished" podID="9aecc649-014f-45ef-849d-27fad4f34933" containerID="602dcaadbe89b58dd913ddc8e28386a8308228aba5b09e942a429efb0cfa15cb" exitCode=0 Nov 03 02:51:39.933827 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:39.933787 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj" Nov 03 02:51:39.934037 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:39.933818 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj" event={"ID":"9aecc649-014f-45ef-849d-27fad4f34933","Type":"ContainerDied","Data":"602dcaadbe89b58dd913ddc8e28386a8308228aba5b09e942a429efb0cfa15cb"} Nov 03 02:51:39.934037 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:39.933872 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj" event={"ID":"9aecc649-014f-45ef-849d-27fad4f34933","Type":"ContainerDied","Data":"1a1ab3e9a5f0f953e9859c7f9e5a3cad3bdf33af422be9ec27bc8da92a874b26"} Nov 03 02:51:39.934037 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:39.933893 5605 scope.go:122] "RemoveContainer" containerID="602dcaadbe89b58dd913ddc8e28386a8308228aba5b09e942a429efb0cfa15cb" Nov 03 02:51:39.945136 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:39.945118 5605 scope.go:122] "RemoveContainer" containerID="602dcaadbe89b58dd913ddc8e28386a8308228aba5b09e942a429efb0cfa15cb" Nov 03 02:51:39.945416 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:51:39.945387 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"602dcaadbe89b58dd913ddc8e28386a8308228aba5b09e942a429efb0cfa15cb\": container with ID starting with 602dcaadbe89b58dd913ddc8e28386a8308228aba5b09e942a429efb0cfa15cb not found: ID does not exist" containerID="602dcaadbe89b58dd913ddc8e28386a8308228aba5b09e942a429efb0cfa15cb" Nov 03 02:51:39.945522 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:39.945429 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"602dcaadbe89b58dd913ddc8e28386a8308228aba5b09e942a429efb0cfa15cb"} err="failed to get container status \"602dcaadbe89b58dd913ddc8e28386a8308228aba5b09e942a429efb0cfa15cb\": rpc error: code = NotFound desc = could not find container \"602dcaadbe89b58dd913ddc8e28386a8308228aba5b09e942a429efb0cfa15cb\": container with ID starting with 602dcaadbe89b58dd913ddc8e28386a8308228aba5b09e942a429efb0cfa15cb not found: ID does not exist" Nov 03 02:51:40.105872 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:40.105818 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj"] Nov 03 02:51:40.109259 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:40.109235 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-kpqgj"] Nov 03 02:51:40.742638 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:40.742611 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-7dn2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body= Nov 03 02:51:40.742874 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:40.742852 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7dn2l" podUID="21200be4-59d6-4955-8d62-5394f4de28df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" Nov 03 02:51:40.792385 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:40.791701 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-jgcjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body= Nov 03 02:51:40.792385 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:40.791756 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jgcjh" podUID="80871945-cbe2-40b4-8715-472c062fc27f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" Nov 03 02:51:40.850311 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:40.850281 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9aecc649-014f-45ef-849d-27fad4f34933" path="/var/lib/kubelet/pods/9aecc649-014f-45ef-849d-27fad4f34933/volumes" Nov 03 02:51:41.742933 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:41.742896 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-7dn2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body= Nov 03 02:51:41.743332 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:41.742945 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7dn2l" podUID="21200be4-59d6-4955-8d62-5394f4de28df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" Nov 03 02:51:41.790382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:41.789794 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-jgcjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body= Nov 03 02:51:41.790382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:41.789857 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jgcjh" podUID="80871945-cbe2-40b4-8715-472c062fc27f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" Nov 03 02:51:42.742694 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:42.742659 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-7dn2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body= Nov 03 02:51:42.742860 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:42.742717 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7dn2l" podUID="21200be4-59d6-4955-8d62-5394f4de28df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" Nov 03 02:51:42.794810 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:42.794775 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-jgcjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body= Nov 03 02:51:42.795426 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:42.795393 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jgcjh" podUID="80871945-cbe2-40b4-8715-472c062fc27f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" Nov 03 02:51:43.743492 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:43.743459 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-7dn2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body= Nov 03 02:51:43.743687 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:43.743514 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7dn2l" podUID="21200be4-59d6-4955-8d62-5394f4de28df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" Nov 03 02:51:43.788428 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:43.788395 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-jgcjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body= Nov 03 02:51:43.788555 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:43.788451 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jgcjh" podUID="80871945-cbe2-40b4-8715-472c062fc27f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" Nov 03 02:51:44.088604 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:44.088514 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-56c7fb7568-t7rlj" podUID="c82367ed-80d4-44de-9d37-450c1fca5281" containerName="payload-processing" containerID="cri-o://df22aa38495fb9c6e39f9ebae6c462c5c326f6baf65df1dfcb1456b47e75f875" gracePeriod=25 Nov 03 02:51:44.361680 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:44.361654 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56c7fb7568-t7rlj" Nov 03 02:51:44.452814 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:44.452779 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c82367ed-80d4-44de-9d37-450c1fca5281-kube-api-access-nb78z\" (UniqueName: \"kubernetes.io/projected/c82367ed-80d4-44de-9d37-450c1fca5281-kube-api-access-nb78z\") pod \"c82367ed-80d4-44de-9d37-450c1fca5281\" (UID: \"c82367ed-80d4-44de-9d37-450c1fca5281\") " Nov 03 02:51:44.453001 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:44.452879 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c82367ed-80d4-44de-9d37-450c1fca5281-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c82367ed-80d4-44de-9d37-450c1fca5281-plugins-config-volume\") pod \"c82367ed-80d4-44de-9d37-450c1fca5281\" (UID: \"c82367ed-80d4-44de-9d37-450c1fca5281\") " Nov 03 02:51:44.453001 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:44.452979 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c82367ed-80d4-44de-9d37-450c1fca5281-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c82367ed-80d4-44de-9d37-450c1fca5281-provider-credentials\") pod \"c82367ed-80d4-44de-9d37-450c1fca5281\" (UID: \"c82367ed-80d4-44de-9d37-450c1fca5281\") " Nov 03 02:51:44.453107 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:44.453010 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c82367ed-80d4-44de-9d37-450c1fca5281-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c82367ed-80d4-44de-9d37-450c1fca5281-routing-overlay-volume\") pod \"c82367ed-80d4-44de-9d37-450c1fca5281\" (UID: \"c82367ed-80d4-44de-9d37-450c1fca5281\") " Nov 03 02:51:44.453307 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:44.453278 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c82367ed-80d4-44de-9d37-450c1fca5281-plugins-config-volume" pod "c82367ed-80d4-44de-9d37-450c1fca5281" (UID: "c82367ed-80d4-44de-9d37-450c1fca5281"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:51:44.453530 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:44.453512 5605 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c82367ed-80d4-44de-9d37-450c1fca5281-plugins-config-volume\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:51:44.453687 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:44.453662 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c82367ed-80d4-44de-9d37-450c1fca5281-routing-overlay-volume" pod "c82367ed-80d4-44de-9d37-450c1fca5281" (UID: "c82367ed-80d4-44de-9d37-450c1fca5281"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:51:44.455414 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:44.455391 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c82367ed-80d4-44de-9d37-450c1fca5281-kube-api-access-nb78z" pod "c82367ed-80d4-44de-9d37-450c1fca5281" (UID: "c82367ed-80d4-44de-9d37-450c1fca5281"). InnerVolumeSpecName "kube-api-access-nb78z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:51:44.455627 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:44.455606 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c82367ed-80d4-44de-9d37-450c1fca5281-provider-credentials" pod "c82367ed-80d4-44de-9d37-450c1fca5281" (UID: "c82367ed-80d4-44de-9d37-450c1fca5281"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:51:44.554418 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:44.554381 5605 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c82367ed-80d4-44de-9d37-450c1fca5281-provider-credentials\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:51:44.554418 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:44.554418 5605 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c82367ed-80d4-44de-9d37-450c1fca5281-routing-overlay-volume\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:51:44.554657 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:44.554436 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nb78z\" (UniqueName: \"kubernetes.io/projected/c82367ed-80d4-44de-9d37-450c1fca5281-kube-api-access-nb78z\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:51:44.743566 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:44.743483 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-7dn2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body= Nov 03 02:51:44.743566 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:44.743535 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7dn2l" podUID="21200be4-59d6-4955-8d62-5394f4de28df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" Nov 03 02:51:44.788777 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:44.788742 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-jgcjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body= Nov 03 02:51:44.788911 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:44.788797 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jgcjh" podUID="80871945-cbe2-40b4-8715-472c062fc27f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" Nov 03 02:51:44.964838 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:44.964806 5605 generic.go:356] "Generic (PLEG): container finished" podID="c82367ed-80d4-44de-9d37-450c1fca5281" containerID="df22aa38495fb9c6e39f9ebae6c462c5c326f6baf65df1dfcb1456b47e75f875" exitCode=0 Nov 03 02:51:44.965006 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:44.964853 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56c7fb7568-t7rlj" Nov 03 02:51:44.965006 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:44.964895 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56c7fb7568-t7rlj" event={"ID":"c82367ed-80d4-44de-9d37-450c1fca5281","Type":"ContainerDied","Data":"df22aa38495fb9c6e39f9ebae6c462c5c326f6baf65df1dfcb1456b47e75f875"} Nov 03 02:51:44.965006 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:44.964964 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56c7fb7568-t7rlj" event={"ID":"c82367ed-80d4-44de-9d37-450c1fca5281","Type":"ContainerDied","Data":"3a723a79de6b12a8b2a482934db6b3808fc09ed10af5bc1e9231719e7ff590a8"} Nov 03 02:51:44.965006 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:44.964998 5605 scope.go:122] "RemoveContainer" containerID="df22aa38495fb9c6e39f9ebae6c462c5c326f6baf65df1dfcb1456b47e75f875" Nov 03 02:51:44.974483 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:44.974468 5605 scope.go:122] "RemoveContainer" containerID="df22aa38495fb9c6e39f9ebae6c462c5c326f6baf65df1dfcb1456b47e75f875" Nov 03 02:51:44.974797 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:51:44.974771 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df22aa38495fb9c6e39f9ebae6c462c5c326f6baf65df1dfcb1456b47e75f875\": container with ID starting with df22aa38495fb9c6e39f9ebae6c462c5c326f6baf65df1dfcb1456b47e75f875 not found: ID does not exist" containerID="df22aa38495fb9c6e39f9ebae6c462c5c326f6baf65df1dfcb1456b47e75f875" Nov 03 02:51:44.974865 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:44.974808 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df22aa38495fb9c6e39f9ebae6c462c5c326f6baf65df1dfcb1456b47e75f875"} err="failed to get container status \"df22aa38495fb9c6e39f9ebae6c462c5c326f6baf65df1dfcb1456b47e75f875\": rpc error: code = NotFound desc = could not find container \"df22aa38495fb9c6e39f9ebae6c462c5c326f6baf65df1dfcb1456b47e75f875\": container with ID starting with df22aa38495fb9c6e39f9ebae6c462c5c326f6baf65df1dfcb1456b47e75f875 not found: ID does not exist" Nov 03 02:51:45.120324 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:45.120300 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-56c7fb7568-t7rlj"] Nov 03 02:51:45.125829 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:45.125807 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-56c7fb7568-t7rlj"] Nov 03 02:51:46.136516 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:46.136480 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7dn2l" Nov 03 02:51:46.137002 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:46.136916 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7dn2l" Nov 03 02:51:46.138124 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:46.138104 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7dn2l" Nov 03 02:51:46.172405 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:46.172323 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jgcjh" Nov 03 02:51:46.172821 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:46.172794 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jgcjh" Nov 03 02:51:46.174342 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:46.174292 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jgcjh" Nov 03 02:51:46.850461 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:46.850431 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c82367ed-80d4-44de-9d37-450c1fca5281" path="/var/lib/kubelet/pods/c82367ed-80d4-44de-9d37-450c1fca5281/volumes" Nov 03 02:51:48.041401 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:48.041369 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-089846b2-prq2v"] Nov 03 02:51:48.041827 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:48.041813 5605 cpu_manager.go:414] "containerMap: removing container" podUID="9aecc649-014f-45ef-849d-27fad4f34933" containerName="istio-proxy" Nov 03 02:51:48.041872 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:48.041828 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9aecc649-014f-45ef-849d-27fad4f34933" containerName="istio-proxy" Nov 03 02:51:48.041872 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:48.041863 5605 cpu_manager.go:414] "containerMap: removing container" podUID="c82367ed-80d4-44de-9d37-450c1fca5281" containerName="payload-processing" Nov 03 02:51:48.041872 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:48.041869 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c82367ed-80d4-44de-9d37-450c1fca5281" containerName="payload-processing" Nov 03 02:51:48.041967 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:48.041949 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="9aecc649-014f-45ef-849d-27fad4f34933" containerName="istio-proxy" Nov 03 02:51:48.041967 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:48.041963 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="c82367ed-80d4-44de-9d37-450c1fca5281" containerName="payload-processing" Nov 03 02:51:48.046448 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:48.046427 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-089846b2-prq2v" Nov 03 02:51:48.094811 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:48.094778 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn7h8\" (UniqueName: \"kubernetes.io/projected/0c1f51a2-c981-409a-9af7-4d08d952385b-kube-api-access-fn7h8\") pod \"maas-api-revoke-sub-089846b2-prq2v\" (UID: \"0c1f51a2-c981-409a-9af7-4d08d952385b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-089846b2-prq2v" Nov 03 02:51:48.095089 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:48.095008 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c1f51a2-c981-409a-9af7-4d08d952385b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-089846b2-prq2v\" (UID: \"0c1f51a2-c981-409a-9af7-4d08d952385b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-089846b2-prq2v" Nov 03 02:51:48.095187 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:48.095153 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c1f51a2-c981-409a-9af7-4d08d952385b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-089846b2-prq2v\" (UID: \"0c1f51a2-c981-409a-9af7-4d08d952385b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-089846b2-prq2v" Nov 03 02:51:48.100150 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:48.100120 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:51:48.100337 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:48.100204 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:51:48.100337 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:48.100288 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:51:48.102153 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:48.102122 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-089846b2-prq2v"] Nov 03 02:51:48.145406 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:48.145380 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:51:48.196550 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:48.196518 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn7h8\" (UniqueName: \"kubernetes.io/projected/0c1f51a2-c981-409a-9af7-4d08d952385b-kube-api-access-fn7h8\") pod \"maas-api-revoke-sub-089846b2-prq2v\" (UID: \"0c1f51a2-c981-409a-9af7-4d08d952385b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-089846b2-prq2v" Nov 03 02:51:48.196703 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:48.196647 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c1f51a2-c981-409a-9af7-4d08d952385b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-089846b2-prq2v\" (UID: \"0c1f51a2-c981-409a-9af7-4d08d952385b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-089846b2-prq2v" Nov 03 02:51:48.196777 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:48.196705 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c1f51a2-c981-409a-9af7-4d08d952385b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-089846b2-prq2v\" (UID: \"0c1f51a2-c981-409a-9af7-4d08d952385b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-089846b2-prq2v" Nov 03 02:51:48.197283 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:48.197252 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c1f51a2-c981-409a-9af7-4d08d952385b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-089846b2-prq2v\" (UID: \"0c1f51a2-c981-409a-9af7-4d08d952385b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-089846b2-prq2v" Nov 03 02:51:48.198946 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:48.198926 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c1f51a2-c981-409a-9af7-4d08d952385b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-089846b2-prq2v\" (UID: \"0c1f51a2-c981-409a-9af7-4d08d952385b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-089846b2-prq2v" Nov 03 02:51:48.251555 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:48.251527 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn7h8\" (UniqueName: \"kubernetes.io/projected/0c1f51a2-c981-409a-9af7-4d08d952385b-kube-api-access-fn7h8\") pod \"maas-api-revoke-sub-089846b2-prq2v\" (UID: \"0c1f51a2-c981-409a-9af7-4d08d952385b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-089846b2-prq2v" Nov 03 02:51:48.359065 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:48.359035 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-089846b2-prq2v" Nov 03 02:51:48.727523 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:48.727444 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-089846b2-prq2v"] Nov 03 02:51:48.733493 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:51:48.733464 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c1f51a2_c981_409a_9af7_4d08d952385b.slice/crio-3d8751ea829354e4abace391c38d322e56b959ca90303921f3b04054644412c1 WatchSource:0}: Error finding container 3d8751ea829354e4abace391c38d322e56b959ca90303921f3b04054644412c1: Status 404 returned error can't find the container with id 3d8751ea829354e4abace391c38d322e56b959ca90303921f3b04054644412c1 Nov 03 02:51:48.993346 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:48.993277 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-089846b2-prq2v" event={"ID":"0c1f51a2-c981-409a-9af7-4d08d952385b","Type":"ContainerStarted","Data":"3d8751ea829354e4abace391c38d322e56b959ca90303921f3b04054644412c1"} Nov 03 02:51:49.999925 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:49.999890 5605 generic.go:356] "Generic (PLEG): container finished" podID="0c1f51a2-c981-409a-9af7-4d08d952385b" containerID="e741815a500582ca12f95cc4543d74c996ba67f32e87f6395531ceeecb8190be" exitCode=0 Nov 03 02:51:50.000306 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:49.999926 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-089846b2-prq2v" event={"ID":"0c1f51a2-c981-409a-9af7-4d08d952385b","Type":"ContainerDied","Data":"e741815a500582ca12f95cc4543d74c996ba67f32e87f6395531ceeecb8190be"} Nov 03 02:51:51.141348 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:51.141321 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-089846b2-prq2v" Nov 03 02:51:51.230292 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:51.230264 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c1f51a2-c981-409a-9af7-4d08d952385b-kube-api-access-fn7h8\" (UniqueName: \"kubernetes.io/projected/0c1f51a2-c981-409a-9af7-4d08d952385b-kube-api-access-fn7h8\") pod \"0c1f51a2-c981-409a-9af7-4d08d952385b\" (UID: \"0c1f51a2-c981-409a-9af7-4d08d952385b\") " Nov 03 02:51:51.230468 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:51.230345 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c1f51a2-c981-409a-9af7-4d08d952385b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c1f51a2-c981-409a-9af7-4d08d952385b-maas-api-xks-ca\") pod \"0c1f51a2-c981-409a-9af7-4d08d952385b\" (UID: \"0c1f51a2-c981-409a-9af7-4d08d952385b\") " Nov 03 02:51:51.230531 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:51.230485 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c1f51a2-c981-409a-9af7-4d08d952385b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c1f51a2-c981-409a-9af7-4d08d952385b-maas-api-openshift-ca\") pod \"0c1f51a2-c981-409a-9af7-4d08d952385b\" (UID: \"0c1f51a2-c981-409a-9af7-4d08d952385b\") " Nov 03 02:51:51.230844 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:51.230815 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c1f51a2-c981-409a-9af7-4d08d952385b-maas-api-openshift-ca" pod "0c1f51a2-c981-409a-9af7-4d08d952385b" (UID: "0c1f51a2-c981-409a-9af7-4d08d952385b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:51:51.232451 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:51.232425 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c1f51a2-c981-409a-9af7-4d08d952385b-maas-api-xks-ca" pod "0c1f51a2-c981-409a-9af7-4d08d952385b" (UID: "0c1f51a2-c981-409a-9af7-4d08d952385b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:51:51.232451 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:51.232435 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c1f51a2-c981-409a-9af7-4d08d952385b-kube-api-access-fn7h8" pod "0c1f51a2-c981-409a-9af7-4d08d952385b" (UID: "0c1f51a2-c981-409a-9af7-4d08d952385b"). InnerVolumeSpecName "kube-api-access-fn7h8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:51:51.331801 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:51.331772 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c1f51a2-c981-409a-9af7-4d08d952385b-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:51:51.331801 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:51.331799 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fn7h8\" (UniqueName: \"kubernetes.io/projected/0c1f51a2-c981-409a-9af7-4d08d952385b-kube-api-access-fn7h8\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:51:51.331980 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:51.331809 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c1f51a2-c981-409a-9af7-4d08d952385b-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:51:52.012501 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:52.012472 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-089846b2-prq2v" Nov 03 02:51:52.012670 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:52.012472 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-089846b2-prq2v" event={"ID":"0c1f51a2-c981-409a-9af7-4d08d952385b","Type":"ContainerDied","Data":"3d8751ea829354e4abace391c38d322e56b959ca90303921f3b04054644412c1"} Nov 03 02:51:52.012670 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:51:52.012594 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d8751ea829354e4abace391c38d322e56b959ca90303921f3b04054644412c1" Nov 03 02:51:53.913168 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:51:53.913112 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:52:01.378145 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:01.378083 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-54bbc97448-g4xp2"] Nov 03 02:52:01.379465 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:01.379429 5605 cpu_manager.go:414] "containerMap: removing container" podUID="0c1f51a2-c981-409a-9af7-4d08d952385b" containerName="revoke-keys" Nov 03 02:52:01.379465 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:01.379455 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c1f51a2-c981-409a-9af7-4d08d952385b" containerName="revoke-keys" Nov 03 02:52:01.379752 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:01.379554 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c1f51a2-c981-409a-9af7-4d08d952385b" containerName="revoke-keys" Nov 03 02:52:01.383516 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:01.383494 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-54bbc97448-g4xp2" Nov 03 02:52:01.435059 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:01.435025 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-54bbc97448-g4xp2"] Nov 03 02:52:01.535209 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:01.535171 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flg24\" (UniqueName: \"kubernetes.io/projected/718280fc-9786-4b1d-b0ac-0262c5ebf544-kube-api-access-flg24\") pod \"payload-processing-external-model-54bbc97448-g4xp2\" (UID: \"718280fc-9786-4b1d-b0ac-0262c5ebf544\") " pod="models-as-a-service/payload-processing-external-model-54bbc97448-g4xp2" Nov 03 02:52:01.535209 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:01.535213 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/718280fc-9786-4b1d-b0ac-0262c5ebf544-provider-credentials\") pod \"payload-processing-external-model-54bbc97448-g4xp2\" (UID: \"718280fc-9786-4b1d-b0ac-0262c5ebf544\") " pod="models-as-a-service/payload-processing-external-model-54bbc97448-g4xp2" Nov 03 02:52:01.535456 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:01.535350 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/718280fc-9786-4b1d-b0ac-0262c5ebf544-routing-overlay-volume\") pod \"payload-processing-external-model-54bbc97448-g4xp2\" (UID: \"718280fc-9786-4b1d-b0ac-0262c5ebf544\") " pod="models-as-a-service/payload-processing-external-model-54bbc97448-g4xp2" Nov 03 02:52:01.535495 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:01.535463 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/718280fc-9786-4b1d-b0ac-0262c5ebf544-plugins-config-volume\") pod \"payload-processing-external-model-54bbc97448-g4xp2\" (UID: \"718280fc-9786-4b1d-b0ac-0262c5ebf544\") " pod="models-as-a-service/payload-processing-external-model-54bbc97448-g4xp2" Nov 03 02:52:01.636157 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:01.636093 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flg24\" (UniqueName: \"kubernetes.io/projected/718280fc-9786-4b1d-b0ac-0262c5ebf544-kube-api-access-flg24\") pod \"payload-processing-external-model-54bbc97448-g4xp2\" (UID: \"718280fc-9786-4b1d-b0ac-0262c5ebf544\") " pod="models-as-a-service/payload-processing-external-model-54bbc97448-g4xp2" Nov 03 02:52:01.636157 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:01.636125 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/718280fc-9786-4b1d-b0ac-0262c5ebf544-provider-credentials\") pod \"payload-processing-external-model-54bbc97448-g4xp2\" (UID: \"718280fc-9786-4b1d-b0ac-0262c5ebf544\") " pod="models-as-a-service/payload-processing-external-model-54bbc97448-g4xp2" Nov 03 02:52:01.636320 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:01.636198 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/718280fc-9786-4b1d-b0ac-0262c5ebf544-routing-overlay-volume\") pod \"payload-processing-external-model-54bbc97448-g4xp2\" (UID: \"718280fc-9786-4b1d-b0ac-0262c5ebf544\") " pod="models-as-a-service/payload-processing-external-model-54bbc97448-g4xp2" Nov 03 02:52:01.636320 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:01.636237 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/718280fc-9786-4b1d-b0ac-0262c5ebf544-plugins-config-volume\") pod \"payload-processing-external-model-54bbc97448-g4xp2\" (UID: \"718280fc-9786-4b1d-b0ac-0262c5ebf544\") " pod="models-as-a-service/payload-processing-external-model-54bbc97448-g4xp2" Nov 03 02:52:01.637022 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:01.636994 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/718280fc-9786-4b1d-b0ac-0262c5ebf544-routing-overlay-volume\") pod \"payload-processing-external-model-54bbc97448-g4xp2\" (UID: \"718280fc-9786-4b1d-b0ac-0262c5ebf544\") " pod="models-as-a-service/payload-processing-external-model-54bbc97448-g4xp2" Nov 03 02:52:01.637160 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:01.637140 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/718280fc-9786-4b1d-b0ac-0262c5ebf544-plugins-config-volume\") pod \"payload-processing-external-model-54bbc97448-g4xp2\" (UID: \"718280fc-9786-4b1d-b0ac-0262c5ebf544\") " pod="models-as-a-service/payload-processing-external-model-54bbc97448-g4xp2" Nov 03 02:52:01.638611 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:01.638593 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/718280fc-9786-4b1d-b0ac-0262c5ebf544-provider-credentials\") pod \"payload-processing-external-model-54bbc97448-g4xp2\" (UID: \"718280fc-9786-4b1d-b0ac-0262c5ebf544\") " pod="models-as-a-service/payload-processing-external-model-54bbc97448-g4xp2" Nov 03 02:52:01.678823 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:01.678799 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:52:01.691658 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:01.691636 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flg24\" (UniqueName: \"kubernetes.io/projected/718280fc-9786-4b1d-b0ac-0262c5ebf544-kube-api-access-flg24\") pod \"payload-processing-external-model-54bbc97448-g4xp2\" (UID: \"718280fc-9786-4b1d-b0ac-0262c5ebf544\") " pod="models-as-a-service/payload-processing-external-model-54bbc97448-g4xp2" Nov 03 02:52:01.698471 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:01.698453 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-54bbc97448-g4xp2" Nov 03 02:52:02.144484 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:02.144447 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-54bbc97448-g4xp2"] Nov 03 02:52:02.150083 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:52:02.150056 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod718280fc_9786_4b1d_b0ac_0262c5ebf544.slice/crio-4ff2b0d495af319f3a5bbb94c3117f53e9e4b9d1ab124427a767f1fe7bd4728a WatchSource:0}: Error finding container 4ff2b0d495af319f3a5bbb94c3117f53e9e4b9d1ab124427a767f1fe7bd4728a: Status 404 returned error can't find the container with id 4ff2b0d495af319f3a5bbb94c3117f53e9e4b9d1ab124427a767f1fe7bd4728a Nov 03 02:52:03.087894 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:03.087858 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-54bbc97448-g4xp2" event={"ID":"718280fc-9786-4b1d-b0ac-0262c5ebf544","Type":"ContainerStarted","Data":"71776a3ce67cf1506dda1dcebc93b6778ee724c5f0778a98e6c41a3b65ce3b3e"} Nov 03 02:52:03.088403 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:03.087911 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-54bbc97448-g4xp2" event={"ID":"718280fc-9786-4b1d-b0ac-0262c5ebf544","Type":"ContainerStarted","Data":"4ff2b0d495af319f3a5bbb94c3117f53e9e4b9d1ab124427a767f1fe7bd4728a"} Nov 03 02:52:03.088403 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:03.088003 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-54bbc97448-g4xp2" Nov 03 02:52:03.196093 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:03.196038 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-54bbc97448-g4xp2" podStartSLOduration=2.196023913 podStartE2EDuration="2.196023913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:52:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:52:03.194501153 +0000 UTC m=+3600.453699307" watchObservedRunningTime="2026-11-03 02:52:03.196023913 +0000 UTC m=+3600.455222066" Nov 03 02:52:04.608929 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:04.608896 5605 scope.go:122] "RemoveContainer" containerID="30bf88dda1735ec7588e266213fcd43ac4b1fcb8a9d974b3a4477a071bc0c4c7" Nov 03 02:52:04.617961 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:04.617941 5605 scope.go:122] "RemoveContainer" containerID="d003b2e0b5f4ffed049635adc90a9b3dfe4aa4a5ef90505f79ed1b91c682b896" Nov 03 02:52:06.616205 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:06.616170 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-54bbc97448-g4xp2"] Nov 03 02:52:06.647073 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:06.647045 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-55bd96b468-gq57q"] Nov 03 02:52:06.652452 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:06.652433 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-55bd96b468-gq57q" Nov 03 02:52:06.701381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:06.701331 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-a0d055dfaf5d\"" Nov 03 02:52:06.720949 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:06.720924 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d627c58-fb06-4b30-b930-2bd32bb9191c-plugins-config-volume\") pod \"payload-processing-external-model-55bd96b468-gq57q\" (UID: \"4d627c58-fb06-4b30-b930-2bd32bb9191c\") " pod="models-as-a-service/payload-processing-external-model-55bd96b468-gq57q" Nov 03 02:52:06.721061 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:06.720962 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4d627c58-fb06-4b30-b930-2bd32bb9191c-routing-overlay-volume\") pod \"payload-processing-external-model-55bd96b468-gq57q\" (UID: \"4d627c58-fb06-4b30-b930-2bd32bb9191c\") " pod="models-as-a-service/payload-processing-external-model-55bd96b468-gq57q" Nov 03 02:52:06.721061 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:06.721055 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4d627c58-fb06-4b30-b930-2bd32bb9191c-provider-credentials\") pod \"payload-processing-external-model-55bd96b468-gq57q\" (UID: \"4d627c58-fb06-4b30-b930-2bd32bb9191c\") " pod="models-as-a-service/payload-processing-external-model-55bd96b468-gq57q" Nov 03 02:52:06.721187 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:06.721169 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfc9f\" (UniqueName: \"kubernetes.io/projected/4d627c58-fb06-4b30-b930-2bd32bb9191c-kube-api-access-mfc9f\") pod \"payload-processing-external-model-55bd96b468-gq57q\" (UID: \"4d627c58-fb06-4b30-b930-2bd32bb9191c\") " pod="models-as-a-service/payload-processing-external-model-55bd96b468-gq57q" Nov 03 02:52:06.724497 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:06.724434 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-55bd96b468-gq57q"] Nov 03 02:52:06.822564 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:06.822529 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d627c58-fb06-4b30-b930-2bd32bb9191c-plugins-config-volume\") pod \"payload-processing-external-model-55bd96b468-gq57q\" (UID: \"4d627c58-fb06-4b30-b930-2bd32bb9191c\") " pod="models-as-a-service/payload-processing-external-model-55bd96b468-gq57q" Nov 03 02:52:06.822564 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:06.822567 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4d627c58-fb06-4b30-b930-2bd32bb9191c-routing-overlay-volume\") pod \"payload-processing-external-model-55bd96b468-gq57q\" (UID: \"4d627c58-fb06-4b30-b930-2bd32bb9191c\") " pod="models-as-a-service/payload-processing-external-model-55bd96b468-gq57q" Nov 03 02:52:06.822751 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:06.822607 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4d627c58-fb06-4b30-b930-2bd32bb9191c-provider-credentials\") pod \"payload-processing-external-model-55bd96b468-gq57q\" (UID: \"4d627c58-fb06-4b30-b930-2bd32bb9191c\") " pod="models-as-a-service/payload-processing-external-model-55bd96b468-gq57q" Nov 03 02:52:06.822751 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:06.822693 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfc9f\" (UniqueName: \"kubernetes.io/projected/4d627c58-fb06-4b30-b930-2bd32bb9191c-kube-api-access-mfc9f\") pod \"payload-processing-external-model-55bd96b468-gq57q\" (UID: \"4d627c58-fb06-4b30-b930-2bd32bb9191c\") " pod="models-as-a-service/payload-processing-external-model-55bd96b468-gq57q" Nov 03 02:52:06.823285 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:06.823260 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4d627c58-fb06-4b30-b930-2bd32bb9191c-routing-overlay-volume\") pod \"payload-processing-external-model-55bd96b468-gq57q\" (UID: \"4d627c58-fb06-4b30-b930-2bd32bb9191c\") " pod="models-as-a-service/payload-processing-external-model-55bd96b468-gq57q" Nov 03 02:52:06.823423 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:06.823318 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d627c58-fb06-4b30-b930-2bd32bb9191c-plugins-config-volume\") pod \"payload-processing-external-model-55bd96b468-gq57q\" (UID: \"4d627c58-fb06-4b30-b930-2bd32bb9191c\") " pod="models-as-a-service/payload-processing-external-model-55bd96b468-gq57q" Nov 03 02:52:06.825251 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:06.825230 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4d627c58-fb06-4b30-b930-2bd32bb9191c-provider-credentials\") pod \"payload-processing-external-model-55bd96b468-gq57q\" (UID: \"4d627c58-fb06-4b30-b930-2bd32bb9191c\") " pod="models-as-a-service/payload-processing-external-model-55bd96b468-gq57q" Nov 03 02:52:06.880465 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:06.880395 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfc9f\" (UniqueName: \"kubernetes.io/projected/4d627c58-fb06-4b30-b930-2bd32bb9191c-kube-api-access-mfc9f\") pod \"payload-processing-external-model-55bd96b468-gq57q\" (UID: \"4d627c58-fb06-4b30-b930-2bd32bb9191c\") " pod="models-as-a-service/payload-processing-external-model-55bd96b468-gq57q" Nov 03 02:52:06.986915 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:06.986885 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-55bd96b468-gq57q" Nov 03 02:52:07.369386 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:07.368834 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-55bd96b468-gq57q"] Nov 03 02:52:07.385621 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:52:07.385591 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d627c58_fb06_4b30_b930_2bd32bb9191c.slice/crio-7f34aed31b395b091061bad88216d8f4c28c1b2a8ebb0324514ada8391a5a6ea WatchSource:0}: Error finding container 7f34aed31b395b091061bad88216d8f4c28c1b2a8ebb0324514ada8391a5a6ea: Status 404 returned error can't find the container with id 7f34aed31b395b091061bad88216d8f4c28c1b2a8ebb0324514ada8391a5a6ea Nov 03 02:52:08.117522 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:08.117482 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-55bd96b468-gq57q" event={"ID":"4d627c58-fb06-4b30-b930-2bd32bb9191c","Type":"ContainerStarted","Data":"805feb7a8cf3927f26a74069218c44baca2a4381d0c68a7d5f26727cdd35688e"} Nov 03 02:52:08.117923 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:08.117572 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-55bd96b468-gq57q" event={"ID":"4d627c58-fb06-4b30-b930-2bd32bb9191c","Type":"ContainerStarted","Data":"7f34aed31b395b091061bad88216d8f4c28c1b2a8ebb0324514ada8391a5a6ea"} Nov 03 02:52:08.117923 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:08.117673 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-55bd96b468-gq57q" Nov 03 02:52:08.228871 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:08.228816 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-55bd96b468-gq57q" podStartSLOduration=2.228804363 podStartE2EDuration="2.228804363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:52:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:52:08.227452591 +0000 UTC m=+3605.486650744" watchObservedRunningTime="2026-11-03 02:52:08.228804363 +0000 UTC m=+3605.488002515" Nov 03 02:52:11.136397 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:11.136329 5605 generic.go:356] "Generic (PLEG): container finished" podID="53640730-1899-46e3-95de-b5a502efbd89" containerID="0f00a946e3edd3ee50f79cac07a83974f97e755c7cd8aaaf9ac5f719a06c3964" exitCode=0 Nov 03 02:52:11.136773 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:11.136398 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r" event={"ID":"53640730-1899-46e3-95de-b5a502efbd89","Type":"ContainerDied","Data":"0f00a946e3edd3ee50f79cac07a83974f97e755c7cd8aaaf9ac5f719a06c3964"} Nov 03 02:52:11.136819 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:11.136792 5605 scope.go:122] "RemoveContainer" containerID="0f00a946e3edd3ee50f79cac07a83974f97e755c7cd8aaaf9ac5f719a06c3964" Nov 03 02:52:11.138226 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:11.138198 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:52:11.138277 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:11.138263 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:52:11.138314 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:11.138295 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:52:11.617406 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:11.617348 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-54bbc97448-g4xp2" podUID="718280fc-9786-4b1d-b0ac-0262c5ebf544" containerName="payload-processing" containerID="cri-o://71776a3ce67cf1506dda1dcebc93b6778ee724c5f0778a98e6c41a3b65ce3b3e" gracePeriod=25 Nov 03 02:52:11.780771 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:11.780744 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-55bd96b468-gq57q"] Nov 03 02:52:11.829436 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:11.829403 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-54bbc97448-ntmsd"] Nov 03 02:52:11.833938 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:11.833916 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-54bbc97448-ntmsd" Nov 03 02:52:11.879381 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:11.879326 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lscnh\" (UniqueName: \"kubernetes.io/projected/818446a4-a6e6-413c-b698-7cdcb207922c-kube-api-access-lscnh\") pod \"payload-processing-external-model-54bbc97448-ntmsd\" (UID: \"818446a4-a6e6-413c-b698-7cdcb207922c\") " pod="models-as-a-service/payload-processing-external-model-54bbc97448-ntmsd" Nov 03 02:52:11.879513 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:11.879391 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/818446a4-a6e6-413c-b698-7cdcb207922c-plugins-config-volume\") pod \"payload-processing-external-model-54bbc97448-ntmsd\" (UID: \"818446a4-a6e6-413c-b698-7cdcb207922c\") " pod="models-as-a-service/payload-processing-external-model-54bbc97448-ntmsd" Nov 03 02:52:11.879587 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:11.879556 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/818446a4-a6e6-413c-b698-7cdcb207922c-routing-overlay-volume\") pod \"payload-processing-external-model-54bbc97448-ntmsd\" (UID: \"818446a4-a6e6-413c-b698-7cdcb207922c\") " pod="models-as-a-service/payload-processing-external-model-54bbc97448-ntmsd" Nov 03 02:52:11.879651 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:11.879620 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/818446a4-a6e6-413c-b698-7cdcb207922c-provider-credentials\") pod \"payload-processing-external-model-54bbc97448-ntmsd\" (UID: \"818446a4-a6e6-413c-b698-7cdcb207922c\") " pod="models-as-a-service/payload-processing-external-model-54bbc97448-ntmsd" Nov 03 02:52:11.895280 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:11.894472 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-54bbc97448-ntmsd"] Nov 03 02:52:11.909780 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:11.909759 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-54bbc97448-g4xp2" Nov 03 02:52:11.981110 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:11.981088 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/718280fc-9786-4b1d-b0ac-0262c5ebf544-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/718280fc-9786-4b1d-b0ac-0262c5ebf544-plugins-config-volume\") pod \"718280fc-9786-4b1d-b0ac-0262c5ebf544\" (UID: \"718280fc-9786-4b1d-b0ac-0262c5ebf544\") " Nov 03 02:52:11.981249 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:11.981128 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/718280fc-9786-4b1d-b0ac-0262c5ebf544-provider-credentials\" (UniqueName: \"kubernetes.io/projected/718280fc-9786-4b1d-b0ac-0262c5ebf544-provider-credentials\") pod \"718280fc-9786-4b1d-b0ac-0262c5ebf544\" (UID: \"718280fc-9786-4b1d-b0ac-0262c5ebf544\") " Nov 03 02:52:11.981249 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:11.981163 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/718280fc-9786-4b1d-b0ac-0262c5ebf544-kube-api-access-flg24\" (UniqueName: \"kubernetes.io/projected/718280fc-9786-4b1d-b0ac-0262c5ebf544-kube-api-access-flg24\") pod \"718280fc-9786-4b1d-b0ac-0262c5ebf544\" (UID: \"718280fc-9786-4b1d-b0ac-0262c5ebf544\") " Nov 03 02:52:11.981249 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:11.981194 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/718280fc-9786-4b1d-b0ac-0262c5ebf544-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/718280fc-9786-4b1d-b0ac-0262c5ebf544-routing-overlay-volume\") pod \"718280fc-9786-4b1d-b0ac-0262c5ebf544\" (UID: \"718280fc-9786-4b1d-b0ac-0262c5ebf544\") " Nov 03 02:52:11.981455 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:11.981427 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/818446a4-a6e6-413c-b698-7cdcb207922c-routing-overlay-volume\") pod \"payload-processing-external-model-54bbc97448-ntmsd\" (UID: \"818446a4-a6e6-413c-b698-7cdcb207922c\") " pod="models-as-a-service/payload-processing-external-model-54bbc97448-ntmsd" Nov 03 02:52:11.981512 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:11.981477 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/818446a4-a6e6-413c-b698-7cdcb207922c-provider-credentials\") pod \"payload-processing-external-model-54bbc97448-ntmsd\" (UID: \"818446a4-a6e6-413c-b698-7cdcb207922c\") " pod="models-as-a-service/payload-processing-external-model-54bbc97448-ntmsd" Nov 03 02:52:11.981606 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:11.981587 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lscnh\" (UniqueName: \"kubernetes.io/projected/818446a4-a6e6-413c-b698-7cdcb207922c-kube-api-access-lscnh\") pod \"payload-processing-external-model-54bbc97448-ntmsd\" (UID: \"818446a4-a6e6-413c-b698-7cdcb207922c\") " pod="models-as-a-service/payload-processing-external-model-54bbc97448-ntmsd" Nov 03 02:52:11.981606 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:11.981594 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/718280fc-9786-4b1d-b0ac-0262c5ebf544-plugins-config-volume" pod "718280fc-9786-4b1d-b0ac-0262c5ebf544" (UID: "718280fc-9786-4b1d-b0ac-0262c5ebf544"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:52:11.981735 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:11.981667 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/818446a4-a6e6-413c-b698-7cdcb207922c-plugins-config-volume\") pod \"payload-processing-external-model-54bbc97448-ntmsd\" (UID: \"818446a4-a6e6-413c-b698-7cdcb207922c\") " pod="models-as-a-service/payload-processing-external-model-54bbc97448-ntmsd" Nov 03 02:52:11.981795 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:11.981745 5605 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/718280fc-9786-4b1d-b0ac-0262c5ebf544-plugins-config-volume\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:52:11.982092 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:11.982029 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/718280fc-9786-4b1d-b0ac-0262c5ebf544-routing-overlay-volume" pod "718280fc-9786-4b1d-b0ac-0262c5ebf544" (UID: "718280fc-9786-4b1d-b0ac-0262c5ebf544"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:52:11.982336 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:11.982315 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/818446a4-a6e6-413c-b698-7cdcb207922c-routing-overlay-volume\") pod \"payload-processing-external-model-54bbc97448-ntmsd\" (UID: \"818446a4-a6e6-413c-b698-7cdcb207922c\") " pod="models-as-a-service/payload-processing-external-model-54bbc97448-ntmsd" Nov 03 02:52:11.982588 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:11.982564 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/818446a4-a6e6-413c-b698-7cdcb207922c-plugins-config-volume\") pod \"payload-processing-external-model-54bbc97448-ntmsd\" (UID: \"818446a4-a6e6-413c-b698-7cdcb207922c\") " pod="models-as-a-service/payload-processing-external-model-54bbc97448-ntmsd" Nov 03 02:52:11.983403 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:11.983381 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/718280fc-9786-4b1d-b0ac-0262c5ebf544-kube-api-access-flg24" pod "718280fc-9786-4b1d-b0ac-0262c5ebf544" (UID: "718280fc-9786-4b1d-b0ac-0262c5ebf544"). InnerVolumeSpecName "kube-api-access-flg24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:52:11.983489 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:11.983470 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/718280fc-9786-4b1d-b0ac-0262c5ebf544-provider-credentials" pod "718280fc-9786-4b1d-b0ac-0262c5ebf544" (UID: "718280fc-9786-4b1d-b0ac-0262c5ebf544"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:52:11.984416 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:11.984399 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/818446a4-a6e6-413c-b698-7cdcb207922c-provider-credentials\") pod \"payload-processing-external-model-54bbc97448-ntmsd\" (UID: \"818446a4-a6e6-413c-b698-7cdcb207922c\") " pod="models-as-a-service/payload-processing-external-model-54bbc97448-ntmsd" Nov 03 02:52:12.036213 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:12.036190 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lscnh\" (UniqueName: \"kubernetes.io/projected/818446a4-a6e6-413c-b698-7cdcb207922c-kube-api-access-lscnh\") pod \"payload-processing-external-model-54bbc97448-ntmsd\" (UID: \"818446a4-a6e6-413c-b698-7cdcb207922c\") " pod="models-as-a-service/payload-processing-external-model-54bbc97448-ntmsd" Nov 03 02:52:12.082999 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:12.082966 5605 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/718280fc-9786-4b1d-b0ac-0262c5ebf544-provider-credentials\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:52:12.082999 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:12.082994 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flg24\" (UniqueName: \"kubernetes.io/projected/718280fc-9786-4b1d-b0ac-0262c5ebf544-kube-api-access-flg24\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:52:12.082999 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:12.083004 5605 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/718280fc-9786-4b1d-b0ac-0262c5ebf544-routing-overlay-volume\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:52:12.144010 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:12.143928 5605 generic.go:356] "Generic (PLEG): container finished" podID="718280fc-9786-4b1d-b0ac-0262c5ebf544" containerID="71776a3ce67cf1506dda1dcebc93b6778ee724c5f0778a98e6c41a3b65ce3b3e" exitCode=0 Nov 03 02:52:12.144442 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:12.144176 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-54bbc97448-g4xp2" event={"ID":"718280fc-9786-4b1d-b0ac-0262c5ebf544","Type":"ContainerDied","Data":"71776a3ce67cf1506dda1dcebc93b6778ee724c5f0778a98e6c41a3b65ce3b3e"} Nov 03 02:52:12.144442 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:12.144241 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-54bbc97448-g4xp2" event={"ID":"718280fc-9786-4b1d-b0ac-0262c5ebf544","Type":"ContainerDied","Data":"4ff2b0d495af319f3a5bbb94c3117f53e9e4b9d1ab124427a767f1fe7bd4728a"} Nov 03 02:52:12.144442 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:12.144270 5605 scope.go:122] "RemoveContainer" containerID="71776a3ce67cf1506dda1dcebc93b6778ee724c5f0778a98e6c41a3b65ce3b3e" Nov 03 02:52:12.144715 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:12.144686 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-54bbc97448-g4xp2" Nov 03 02:52:12.147277 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:12.147243 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r" event={"ID":"53640730-1899-46e3-95de-b5a502efbd89","Type":"ContainerStarted","Data":"47f167d90555678f6c66fe37734676c52a2dd9a982481e6ef50ec6ee13fee4f9"} Nov 03 02:52:12.148222 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:12.148200 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-54bbc97448-ntmsd" Nov 03 02:52:12.186391 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:12.186344 5605 scope.go:122] "RemoveContainer" containerID="71776a3ce67cf1506dda1dcebc93b6778ee724c5f0778a98e6c41a3b65ce3b3e" Nov 03 02:52:12.186833 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:52:12.186805 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71776a3ce67cf1506dda1dcebc93b6778ee724c5f0778a98e6c41a3b65ce3b3e\": container with ID starting with 71776a3ce67cf1506dda1dcebc93b6778ee724c5f0778a98e6c41a3b65ce3b3e not found: ID does not exist" containerID="71776a3ce67cf1506dda1dcebc93b6778ee724c5f0778a98e6c41a3b65ce3b3e" Nov 03 02:52:12.186958 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:12.186848 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71776a3ce67cf1506dda1dcebc93b6778ee724c5f0778a98e6c41a3b65ce3b3e"} err="failed to get container status \"71776a3ce67cf1506dda1dcebc93b6778ee724c5f0778a98e6c41a3b65ce3b3e\": rpc error: code = NotFound desc = could not find container \"71776a3ce67cf1506dda1dcebc93b6778ee724c5f0778a98e6c41a3b65ce3b3e\": container with ID starting with 71776a3ce67cf1506dda1dcebc93b6778ee724c5f0778a98e6c41a3b65ce3b3e not found: ID does not exist" Nov 03 02:52:12.313710 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:12.313680 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-54bbc97448-g4xp2"] Nov 03 02:52:12.318762 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:12.318735 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-54bbc97448-g4xp2"] Nov 03 02:52:12.580470 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:12.580424 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-54bbc97448-ntmsd"] Nov 03 02:52:12.581342 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:52:12.581314 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod818446a4_a6e6_413c_b698_7cdcb207922c.slice/crio-6ca36ce5aaaa8587477afed9bcc2962cec70a06e624a56ca44016262ff2f7217 WatchSource:0}: Error finding container 6ca36ce5aaaa8587477afed9bcc2962cec70a06e624a56ca44016262ff2f7217: Status 404 returned error can't find the container with id 6ca36ce5aaaa8587477afed9bcc2962cec70a06e624a56ca44016262ff2f7217 Nov 03 02:52:12.850757 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:12.850680 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="718280fc-9786-4b1d-b0ac-0262c5ebf544" path="/var/lib/kubelet/pods/718280fc-9786-4b1d-b0ac-0262c5ebf544/volumes" Nov 03 02:52:13.074442 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:13.074411 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r" Nov 03 02:52:13.075475 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:13.075445 5605 patch_prober.go:29] interesting pod/e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Nov 03 02:52:13.075679 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:13.075652 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r" podUID="53640730-1899-46e3-95de-b5a502efbd89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Nov 03 02:52:13.179493 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:13.177053 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-54bbc97448-ntmsd" event={"ID":"818446a4-a6e6-413c-b698-7cdcb207922c","Type":"ContainerStarted","Data":"1d64bf1d6c03faffea03cfd11506178654281ded7665ee0c268a9d41e1250826"} Nov 03 02:52:13.179493 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:13.177112 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-54bbc97448-ntmsd" event={"ID":"818446a4-a6e6-413c-b698-7cdcb207922c","Type":"ContainerStarted","Data":"6ca36ce5aaaa8587477afed9bcc2962cec70a06e624a56ca44016262ff2f7217"} Nov 03 02:52:13.179493 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:13.177163 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-54bbc97448-ntmsd" Nov 03 02:52:13.296223 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:13.296176 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-54bbc97448-ntmsd" podStartSLOduration=2.296163767 podStartE2EDuration="2.296163767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:52:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:52:13.292839975 +0000 UTC m=+3610.552038128" watchObservedRunningTime="2026-11-03 02:52:13.296163767 +0000 UTC m=+3610.555361920" Nov 03 02:52:13.483965 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:13.483884 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r" Nov 03 02:52:14.079506 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:14.079460 5605 patch_prober.go:29] interesting pod/e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Nov 03 02:52:14.079715 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:14.079523 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r" podUID="53640730-1899-46e3-95de-b5a502efbd89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Nov 03 02:52:14.430424 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:14.430327 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4"] Nov 03 02:52:14.430865 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:14.430837 5605 cpu_manager.go:414] "containerMap: removing container" podUID="718280fc-9786-4b1d-b0ac-0262c5ebf544" containerName="payload-processing" Nov 03 02:52:14.430865 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:14.430857 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="718280fc-9786-4b1d-b0ac-0262c5ebf544" containerName="payload-processing" Nov 03 02:52:14.431070 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:14.430978 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="718280fc-9786-4b1d-b0ac-0262c5ebf544" containerName="payload-processing" Nov 03 02:52:14.434717 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:14.434698 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4" Nov 03 02:52:14.527379 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:14.526922 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-95152a-openshift-default-dockercfg-kxjv6\"" Nov 03 02:52:14.614888 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:14.613879 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4"] Nov 03 02:52:14.630745 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:14.630702 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/264e26fa-8062-4f70-86c8-3ba19e1c4384-credential-socket\") pod \"e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4\" (UID: \"264e26fa-8062-4f70-86c8-3ba19e1c4384\") " pod="openshift-ingress/e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4" Nov 03 02:52:14.631017 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:14.630998 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/264e26fa-8062-4f70-86c8-3ba19e1c4384-workload-certs\") pod \"e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4\" (UID: \"264e26fa-8062-4f70-86c8-3ba19e1c4384\") " pod="openshift-ingress/e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4" Nov 03 02:52:14.631196 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:14.631162 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/264e26fa-8062-4f70-86c8-3ba19e1c4384-istio-envoy\") pod \"e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4\" (UID: \"264e26fa-8062-4f70-86c8-3ba19e1c4384\") " pod="openshift-ingress/e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4" Nov 03 02:52:14.631432 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:14.631413 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5xdz\" (UniqueName: \"kubernetes.io/projected/264e26fa-8062-4f70-86c8-3ba19e1c4384-kube-api-access-l5xdz\") pod \"e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4\" (UID: \"264e26fa-8062-4f70-86c8-3ba19e1c4384\") " pod="openshift-ingress/e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4" Nov 03 02:52:14.631619 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:14.631601 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/264e26fa-8062-4f70-86c8-3ba19e1c4384-istio-token\") pod \"e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4\" (UID: \"264e26fa-8062-4f70-86c8-3ba19e1c4384\") " pod="openshift-ingress/e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4" Nov 03 02:52:14.631769 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:14.631753 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/264e26fa-8062-4f70-86c8-3ba19e1c4384-istio-podinfo\") pod \"e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4\" (UID: \"264e26fa-8062-4f70-86c8-3ba19e1c4384\") " pod="openshift-ingress/e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4" Nov 03 02:52:14.631928 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:14.631912 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/264e26fa-8062-4f70-86c8-3ba19e1c4384-workload-socket\") pod \"e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4\" (UID: \"264e26fa-8062-4f70-86c8-3ba19e1c4384\") " pod="openshift-ingress/e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4" Nov 03 02:52:14.632103 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:14.632078 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/264e26fa-8062-4f70-86c8-3ba19e1c4384-istiod-ca-cert\") pod \"e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4\" (UID: \"264e26fa-8062-4f70-86c8-3ba19e1c4384\") " pod="openshift-ingress/e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4" Nov 03 02:52:14.632236 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:14.632219 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/264e26fa-8062-4f70-86c8-3ba19e1c4384-istio-data\") pod \"e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4\" (UID: \"264e26fa-8062-4f70-86c8-3ba19e1c4384\") " pod="openshift-ingress/e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4" Nov 03 02:52:14.733142 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:14.733064 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/264e26fa-8062-4f70-86c8-3ba19e1c4384-credential-socket\") pod \"e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4\" (UID: \"264e26fa-8062-4f70-86c8-3ba19e1c4384\") " pod="openshift-ingress/e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4" Nov 03 02:52:14.733142 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:14.733100 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/264e26fa-8062-4f70-86c8-3ba19e1c4384-workload-certs\") pod \"e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4\" (UID: \"264e26fa-8062-4f70-86c8-3ba19e1c4384\") " pod="openshift-ingress/e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4" Nov 03 02:52:14.733142 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:14.733123 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/264e26fa-8062-4f70-86c8-3ba19e1c4384-istio-envoy\") pod \"e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4\" (UID: \"264e26fa-8062-4f70-86c8-3ba19e1c4384\") " pod="openshift-ingress/e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4" Nov 03 02:52:14.733427 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:14.733177 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5xdz\" (UniqueName: \"kubernetes.io/projected/264e26fa-8062-4f70-86c8-3ba19e1c4384-kube-api-access-l5xdz\") pod \"e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4\" (UID: \"264e26fa-8062-4f70-86c8-3ba19e1c4384\") " pod="openshift-ingress/e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4" Nov 03 02:52:14.733427 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:14.733197 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/264e26fa-8062-4f70-86c8-3ba19e1c4384-istio-token\") pod \"e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4\" (UID: \"264e26fa-8062-4f70-86c8-3ba19e1c4384\") " pod="openshift-ingress/e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4" Nov 03 02:52:14.733427 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:14.733214 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/264e26fa-8062-4f70-86c8-3ba19e1c4384-istio-podinfo\") pod \"e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4\" (UID: \"264e26fa-8062-4f70-86c8-3ba19e1c4384\") " pod="openshift-ingress/e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4" Nov 03 02:52:14.733427 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:14.733239 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/264e26fa-8062-4f70-86c8-3ba19e1c4384-workload-socket\") pod \"e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4\" (UID: \"264e26fa-8062-4f70-86c8-3ba19e1c4384\") " pod="openshift-ingress/e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4" Nov 03 02:52:14.733427 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:14.733266 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/264e26fa-8062-4f70-86c8-3ba19e1c4384-istiod-ca-cert\") pod \"e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4\" (UID: \"264e26fa-8062-4f70-86c8-3ba19e1c4384\") " pod="openshift-ingress/e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4" Nov 03 02:52:14.733427 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:14.733283 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/264e26fa-8062-4f70-86c8-3ba19e1c4384-istio-data\") pod \"e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4\" (UID: \"264e26fa-8062-4f70-86c8-3ba19e1c4384\") " pod="openshift-ingress/e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4" Nov 03 02:52:14.733784 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:14.733518 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/264e26fa-8062-4f70-86c8-3ba19e1c4384-workload-certs\") pod \"e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4\" (UID: \"264e26fa-8062-4f70-86c8-3ba19e1c4384\") " pod="openshift-ingress/e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4" Nov 03 02:52:14.733784 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:14.733674 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/264e26fa-8062-4f70-86c8-3ba19e1c4384-istio-data\") pod \"e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4\" (UID: \"264e26fa-8062-4f70-86c8-3ba19e1c4384\") " pod="openshift-ingress/e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4" Nov 03 02:52:14.733784 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:14.733718 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/264e26fa-8062-4f70-86c8-3ba19e1c4384-credential-socket\") pod \"e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4\" (UID: \"264e26fa-8062-4f70-86c8-3ba19e1c4384\") " pod="openshift-ingress/e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4" Nov 03 02:52:14.733784 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:14.733758 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/264e26fa-8062-4f70-86c8-3ba19e1c4384-workload-socket\") pod \"e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4\" (UID: \"264e26fa-8062-4f70-86c8-3ba19e1c4384\") " pod="openshift-ingress/e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4" Nov 03 02:52:14.734107 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:14.734084 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/264e26fa-8062-4f70-86c8-3ba19e1c4384-istiod-ca-cert\") pod \"e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4\" (UID: \"264e26fa-8062-4f70-86c8-3ba19e1c4384\") " pod="openshift-ingress/e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4" Nov 03 02:52:14.735420 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:14.735399 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/264e26fa-8062-4f70-86c8-3ba19e1c4384-istio-envoy\") pod \"e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4\" (UID: \"264e26fa-8062-4f70-86c8-3ba19e1c4384\") " pod="openshift-ingress/e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4" Nov 03 02:52:14.735976 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:14.735956 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/264e26fa-8062-4f70-86c8-3ba19e1c4384-istio-podinfo\") pod \"e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4\" (UID: \"264e26fa-8062-4f70-86c8-3ba19e1c4384\") " pod="openshift-ingress/e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4" Nov 03 02:52:14.805089 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:14.805055 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5xdz\" (UniqueName: \"kubernetes.io/projected/264e26fa-8062-4f70-86c8-3ba19e1c4384-kube-api-access-l5xdz\") pod \"e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4\" (UID: \"264e26fa-8062-4f70-86c8-3ba19e1c4384\") " pod="openshift-ingress/e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4" Nov 03 02:52:14.817052 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:14.817024 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/264e26fa-8062-4f70-86c8-3ba19e1c4384-istio-token\") pod \"e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4\" (UID: \"264e26fa-8062-4f70-86c8-3ba19e1c4384\") " pod="openshift-ingress/e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4" Nov 03 02:52:15.047496 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:15.047418 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4" Nov 03 02:52:15.081382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:15.072645 5605 patch_prober.go:29] interesting pod/e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Nov 03 02:52:15.081382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:15.072705 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r" podUID="53640730-1899-46e3-95de-b5a502efbd89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Nov 03 02:52:15.501951 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:15.501916 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4"] Nov 03 02:52:15.506844 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:52:15.506815 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod264e26fa_8062_4f70_86c8_3ba19e1c4384.slice/crio-a32c50bca009c5c9429a9e847b562a780b81bd84e054cd36ce097e19dec96c92 WatchSource:0}: Error finding container a32c50bca009c5c9429a9e847b562a780b81bd84e054cd36ce097e19dec96c92: Status 404 returned error can't find the container with id a32c50bca009c5c9429a9e847b562a780b81bd84e054cd36ce097e19dec96c92 Nov 03 02:52:15.509760 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:15.509717 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:52:15.509838 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:15.509813 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:52:15.509875 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:15.509855 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:52:16.080383 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:16.076908 5605 patch_prober.go:29] interesting pod/e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Nov 03 02:52:16.080383 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:16.076970 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r" podUID="53640730-1899-46e3-95de-b5a502efbd89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Nov 03 02:52:16.199500 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:16.199467 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4" event={"ID":"264e26fa-8062-4f70-86c8-3ba19e1c4384","Type":"ContainerStarted","Data":"f48790376c3b6049b37350d92e3525239e699813ace92b7850f727d9e2a6259d"} Nov 03 02:52:16.199697 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:16.199515 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4" event={"ID":"264e26fa-8062-4f70-86c8-3ba19e1c4384","Type":"ContainerStarted","Data":"a32c50bca009c5c9429a9e847b562a780b81bd84e054cd36ce097e19dec96c92"} Nov 03 02:52:16.315651 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:16.315595 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4" podStartSLOduration=2.315581991 podStartE2EDuration="2.315581991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:52:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:52:16.313435688 +0000 UTC m=+3613.572633840" watchObservedRunningTime="2026-11-03 02:52:16.315581991 +0000 UTC m=+3613.574780142" Nov 03 02:52:16.782322 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:16.781569 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-55bd96b468-gq57q" podUID="4d627c58-fb06-4b30-b930-2bd32bb9191c" containerName="payload-processing" containerID="cri-o://805feb7a8cf3927f26a74069218c44baca2a4381d0c68a7d5f26727cdd35688e" gracePeriod=25 Nov 03 02:52:17.046776 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:17.046716 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-55bd96b468-gq57q" Nov 03 02:52:17.048365 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:17.048329 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4" Nov 03 02:52:17.053889 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:17.053868 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4" Nov 03 02:52:17.080194 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:17.080162 5605 patch_prober.go:29] interesting pod/e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Nov 03 02:52:17.080401 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:17.080226 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r" podUID="53640730-1899-46e3-95de-b5a502efbd89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Nov 03 02:52:17.084641 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:17.084532 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d627c58-fb06-4b30-b930-2bd32bb9191c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4d627c58-fb06-4b30-b930-2bd32bb9191c-routing-overlay-volume\") pod \"4d627c58-fb06-4b30-b930-2bd32bb9191c\" (UID: \"4d627c58-fb06-4b30-b930-2bd32bb9191c\") " Nov 03 02:52:17.084641 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:17.084630 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d627c58-fb06-4b30-b930-2bd32bb9191c-kube-api-access-mfc9f\" (UniqueName: \"kubernetes.io/projected/4d627c58-fb06-4b30-b930-2bd32bb9191c-kube-api-access-mfc9f\") pod \"4d627c58-fb06-4b30-b930-2bd32bb9191c\" (UID: \"4d627c58-fb06-4b30-b930-2bd32bb9191c\") " Nov 03 02:52:17.084768 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:17.084708 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d627c58-fb06-4b30-b930-2bd32bb9191c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4d627c58-fb06-4b30-b930-2bd32bb9191c-provider-credentials\") pod \"4d627c58-fb06-4b30-b930-2bd32bb9191c\" (UID: \"4d627c58-fb06-4b30-b930-2bd32bb9191c\") " Nov 03 02:52:17.084815 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:17.084777 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d627c58-fb06-4b30-b930-2bd32bb9191c-routing-overlay-volume" pod "4d627c58-fb06-4b30-b930-2bd32bb9191c" (UID: "4d627c58-fb06-4b30-b930-2bd32bb9191c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:52:17.084852 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:17.084828 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d627c58-fb06-4b30-b930-2bd32bb9191c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d627c58-fb06-4b30-b930-2bd32bb9191c-plugins-config-volume\") pod \"4d627c58-fb06-4b30-b930-2bd32bb9191c\" (UID: \"4d627c58-fb06-4b30-b930-2bd32bb9191c\") " Nov 03 02:52:17.085223 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:17.085135 5605 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4d627c58-fb06-4b30-b930-2bd32bb9191c-routing-overlay-volume\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:52:17.085223 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:17.085197 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d627c58-fb06-4b30-b930-2bd32bb9191c-plugins-config-volume" pod "4d627c58-fb06-4b30-b930-2bd32bb9191c" (UID: "4d627c58-fb06-4b30-b930-2bd32bb9191c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:52:17.087444 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:17.087393 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d627c58-fb06-4b30-b930-2bd32bb9191c-kube-api-access-mfc9f" pod "4d627c58-fb06-4b30-b930-2bd32bb9191c" (UID: "4d627c58-fb06-4b30-b930-2bd32bb9191c"). InnerVolumeSpecName "kube-api-access-mfc9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:52:17.087755 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:17.087736 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d627c58-fb06-4b30-b930-2bd32bb9191c-provider-credentials" pod "4d627c58-fb06-4b30-b930-2bd32bb9191c" (UID: "4d627c58-fb06-4b30-b930-2bd32bb9191c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:52:17.186566 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:17.186536 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfc9f\" (UniqueName: \"kubernetes.io/projected/4d627c58-fb06-4b30-b930-2bd32bb9191c-kube-api-access-mfc9f\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:52:17.186566 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:17.186566 5605 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4d627c58-fb06-4b30-b930-2bd32bb9191c-provider-credentials\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:52:17.186799 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:17.186578 5605 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d627c58-fb06-4b30-b930-2bd32bb9191c-plugins-config-volume\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:52:17.207039 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:17.207012 5605 generic.go:356] "Generic (PLEG): container finished" podID="4d627c58-fb06-4b30-b930-2bd32bb9191c" containerID="805feb7a8cf3927f26a74069218c44baca2a4381d0c68a7d5f26727cdd35688e" exitCode=0 Nov 03 02:52:17.207167 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:17.207054 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-55bd96b468-gq57q" Nov 03 02:52:17.207167 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:17.207093 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-55bd96b468-gq57q" event={"ID":"4d627c58-fb06-4b30-b930-2bd32bb9191c","Type":"ContainerDied","Data":"805feb7a8cf3927f26a74069218c44baca2a4381d0c68a7d5f26727cdd35688e"} Nov 03 02:52:17.207167 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:17.207143 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-55bd96b468-gq57q" event={"ID":"4d627c58-fb06-4b30-b930-2bd32bb9191c","Type":"ContainerDied","Data":"7f34aed31b395b091061bad88216d8f4c28c1b2a8ebb0324514ada8391a5a6ea"} Nov 03 02:52:17.207302 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:17.207174 5605 scope.go:122] "RemoveContainer" containerID="805feb7a8cf3927f26a74069218c44baca2a4381d0c68a7d5f26727cdd35688e" Nov 03 02:52:17.207558 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:17.207393 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4" Nov 03 02:52:17.208689 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:17.208629 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4" Nov 03 02:52:17.220446 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:17.220429 5605 scope.go:122] "RemoveContainer" containerID="805feb7a8cf3927f26a74069218c44baca2a4381d0c68a7d5f26727cdd35688e" Nov 03 02:52:17.221176 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:52:17.221143 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"805feb7a8cf3927f26a74069218c44baca2a4381d0c68a7d5f26727cdd35688e\": container with ID starting with 805feb7a8cf3927f26a74069218c44baca2a4381d0c68a7d5f26727cdd35688e not found: ID does not exist" containerID="805feb7a8cf3927f26a74069218c44baca2a4381d0c68a7d5f26727cdd35688e" Nov 03 02:52:17.221252 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:17.221177 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"805feb7a8cf3927f26a74069218c44baca2a4381d0c68a7d5f26727cdd35688e"} err="failed to get container status \"805feb7a8cf3927f26a74069218c44baca2a4381d0c68a7d5f26727cdd35688e\": rpc error: code = NotFound desc = could not find container \"805feb7a8cf3927f26a74069218c44baca2a4381d0c68a7d5f26727cdd35688e\": container with ID starting with 805feb7a8cf3927f26a74069218c44baca2a4381d0c68a7d5f26727cdd35688e not found: ID does not exist" Nov 03 02:52:17.509023 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:17.508975 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-55bd96b468-gq57q"] Nov 03 02:52:17.514793 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:17.512596 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-55bd96b468-gq57q"] Nov 03 02:52:18.074981 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:18.074954 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r" Nov 03 02:52:18.075795 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:18.075775 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r" Nov 03 02:52:18.850809 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:18.850776 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d627c58-fb06-4b30-b930-2bd32bb9191c" path="/var/lib/kubelet/pods/4d627c58-fb06-4b30-b930-2bd32bb9191c/volumes" Nov 03 02:52:22.910345 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:22.910303 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-95152a-5d959db47f-w6cgh"] Nov 03 02:52:22.911015 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:22.911000 5605 cpu_manager.go:414] "containerMap: removing container" podUID="4d627c58-fb06-4b30-b930-2bd32bb9191c" containerName="payload-processing" Nov 03 02:52:22.911064 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:22.911018 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d627c58-fb06-4b30-b930-2bd32bb9191c" containerName="payload-processing" Nov 03 02:52:22.911163 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:22.911152 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d627c58-fb06-4b30-b930-2bd32bb9191c" containerName="payload-processing" Nov 03 02:52:22.916170 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:22.916143 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-95152a-5d959db47f-w6cgh" Nov 03 02:52:22.947405 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:22.947378 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lscpv\" (UniqueName: \"kubernetes.io/projected/51beebde-3a29-4cc8-8319-a91a990d4755-kube-api-access-lscpv\") pod \"payload-processing-e2e-inf-a-95152a-5d959db47f-w6cgh\" (UID: \"51beebde-3a29-4cc8-8319-a91a990d4755\") " pod="openshift-ingress/payload-processing-e2e-inf-a-95152a-5d959db47f-w6cgh" Nov 03 02:52:22.947560 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:22.947446 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51beebde-3a29-4cc8-8319-a91a990d4755-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-95152a-5d959db47f-w6cgh\" (UID: \"51beebde-3a29-4cc8-8319-a91a990d4755\") " pod="openshift-ingress/payload-processing-e2e-inf-a-95152a-5d959db47f-w6cgh" Nov 03 02:52:22.965277 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:22.965242 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-95152a-5d959db47f-w6cgh"] Nov 03 02:52:23.009417 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:23.009394 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-95152a\"" Nov 03 02:52:23.009539 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:23.009468 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-95152a-dockercfg-vcvdx\"" Nov 03 02:52:23.049083 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:23.049055 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lscpv\" (UniqueName: \"kubernetes.io/projected/51beebde-3a29-4cc8-8319-a91a990d4755-kube-api-access-lscpv\") pod \"payload-processing-e2e-inf-a-95152a-5d959db47f-w6cgh\" (UID: \"51beebde-3a29-4cc8-8319-a91a990d4755\") " pod="openshift-ingress/payload-processing-e2e-inf-a-95152a-5d959db47f-w6cgh" Nov 03 02:52:23.049186 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:23.049098 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51beebde-3a29-4cc8-8319-a91a990d4755-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-95152a-5d959db47f-w6cgh\" (UID: \"51beebde-3a29-4cc8-8319-a91a990d4755\") " pod="openshift-ingress/payload-processing-e2e-inf-a-95152a-5d959db47f-w6cgh" Nov 03 02:52:23.049760 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:23.049741 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51beebde-3a29-4cc8-8319-a91a990d4755-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-95152a-5d959db47f-w6cgh\" (UID: \"51beebde-3a29-4cc8-8319-a91a990d4755\") " pod="openshift-ingress/payload-processing-e2e-inf-a-95152a-5d959db47f-w6cgh" Nov 03 02:52:23.110846 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:23.110819 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lscpv\" (UniqueName: \"kubernetes.io/projected/51beebde-3a29-4cc8-8319-a91a990d4755-kube-api-access-lscpv\") pod \"payload-processing-e2e-inf-a-95152a-5d959db47f-w6cgh\" (UID: \"51beebde-3a29-4cc8-8319-a91a990d4755\") " pod="openshift-ingress/payload-processing-e2e-inf-a-95152a-5d959db47f-w6cgh" Nov 03 02:52:23.229211 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:23.229155 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-95152a-5d959db47f-w6cgh" Nov 03 02:52:23.595855 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:23.595824 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-95152a-5d959db47f-w6cgh"] Nov 03 02:52:23.601412 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:52:23.601387 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51beebde_3a29_4cc8_8319_a91a990d4755.slice/crio-9f154c14d48266548e43df2aceab09e39366535d17e09caccb202888f4eeee93 WatchSource:0}: Error finding container 9f154c14d48266548e43df2aceab09e39366535d17e09caccb202888f4eeee93: Status 404 returned error can't find the container with id 9f154c14d48266548e43df2aceab09e39366535d17e09caccb202888f4eeee93 Nov 03 02:52:23.913472 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:52:23.913259 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:52:24.250712 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:24.250618 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-95152a-5d959db47f-w6cgh" event={"ID":"51beebde-3a29-4cc8-8319-a91a990d4755","Type":"ContainerStarted","Data":"f719709d8077c4c13bb83bc5ec51c47231685cb1b57b1d59d2b059184dae9dfe"} Nov 03 02:52:24.250712 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:24.250669 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-95152a-5d959db47f-w6cgh" event={"ID":"51beebde-3a29-4cc8-8319-a91a990d4755","Type":"ContainerStarted","Data":"9f154c14d48266548e43df2aceab09e39366535d17e09caccb202888f4eeee93"} Nov 03 02:52:24.250961 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:24.250775 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-95152a-5d959db47f-w6cgh" Nov 03 02:52:24.357997 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:24.357951 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-95152a-5d959db47f-w6cgh" podStartSLOduration=2.357937606 podStartE2EDuration="2.357937606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:52:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:52:24.357348967 +0000 UTC m=+3621.616547121" watchObservedRunningTime="2026-11-03 02:52:24.357937606 +0000 UTC m=+3621.617135749" Nov 03 02:52:25.194474 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:25.194439 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-54bbc97448-ntmsd" podUID="818446a4-a6e6-413c-b698-7cdcb207922c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:52:26.417296 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:26.417265 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c8ee6b-4hn2p"] Nov 03 02:52:26.463617 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:26.463594 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c8ee6b-4hn2p"] Nov 03 02:52:26.693778 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:26.693687 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx"] Nov 03 02:52:26.693975 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:26.693942 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx" podUID="3bf95767-b937-4d51-86ff-a41db2ffc3a4" containerName="istio-proxy" containerID="cri-o://e8ac827fbcdc946dd425af6281b7ece4a97995efe070da2be9906073783b32da" gracePeriod=30 Nov 03 02:52:26.765287 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:26.765256 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:52:26.861638 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:26.861601 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecaa74f8-112f-434d-9ebf-45e23e713cde" path="/var/lib/kubelet/pods/ecaa74f8-112f-434d-9ebf-45e23e713cde/volumes" Nov 03 02:52:31.962239 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:31.962179 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx" Nov 03 02:52:32.052567 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:32.052541 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bf95767-b937-4d51-86ff-a41db2ffc3a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bf95767-b937-4d51-86ff-a41db2ffc3a4-istio-data\") pod \"3bf95767-b937-4d51-86ff-a41db2ffc3a4\" (UID: \"3bf95767-b937-4d51-86ff-a41db2ffc3a4\") " Nov 03 02:52:32.052687 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:32.052615 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3bf95767-b937-4d51-86ff-a41db2ffc3a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bf95767-b937-4d51-86ff-a41db2ffc3a4-istio-podinfo\") pod \"3bf95767-b937-4d51-86ff-a41db2ffc3a4\" (UID: \"3bf95767-b937-4d51-86ff-a41db2ffc3a4\") " Nov 03 02:52:32.052687 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:32.052660 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bf95767-b937-4d51-86ff-a41db2ffc3a4-istio-token\" (UniqueName: \"kubernetes.io/projected/3bf95767-b937-4d51-86ff-a41db2ffc3a4-istio-token\") pod \"3bf95767-b937-4d51-86ff-a41db2ffc3a4\" (UID: \"3bf95767-b937-4d51-86ff-a41db2ffc3a4\") " Nov 03 02:52:32.052783 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:32.052709 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bf95767-b937-4d51-86ff-a41db2ffc3a4-kube-api-access-sqmpv\" (UniqueName: \"kubernetes.io/projected/3bf95767-b937-4d51-86ff-a41db2ffc3a4-kube-api-access-sqmpv\") pod \"3bf95767-b937-4d51-86ff-a41db2ffc3a4\" (UID: \"3bf95767-b937-4d51-86ff-a41db2ffc3a4\") " Nov 03 02:52:32.052783 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:32.052748 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bf95767-b937-4d51-86ff-a41db2ffc3a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf95767-b937-4d51-86ff-a41db2ffc3a4-credential-socket\") pod \"3bf95767-b937-4d51-86ff-a41db2ffc3a4\" (UID: \"3bf95767-b937-4d51-86ff-a41db2ffc3a4\") " Nov 03 02:52:32.052899 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:32.052786 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bf95767-b937-4d51-86ff-a41db2ffc3a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf95767-b937-4d51-86ff-a41db2ffc3a4-workload-socket\") pod \"3bf95767-b937-4d51-86ff-a41db2ffc3a4\" (UID: \"3bf95767-b937-4d51-86ff-a41db2ffc3a4\") " Nov 03 02:52:32.052899 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:32.052805 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bf95767-b937-4d51-86ff-a41db2ffc3a4-istio-data" pod "3bf95767-b937-4d51-86ff-a41db2ffc3a4" (UID: "3bf95767-b937-4d51-86ff-a41db2ffc3a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:52:32.052899 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:32.052814 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bf95767-b937-4d51-86ff-a41db2ffc3a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bf95767-b937-4d51-86ff-a41db2ffc3a4-istio-envoy\") pod \"3bf95767-b937-4d51-86ff-a41db2ffc3a4\" (UID: \"3bf95767-b937-4d51-86ff-a41db2ffc3a4\") " Nov 03 02:52:32.052899 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:32.052878 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bf95767-b937-4d51-86ff-a41db2ffc3a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bf95767-b937-4d51-86ff-a41db2ffc3a4-workload-certs\") pod \"3bf95767-b937-4d51-86ff-a41db2ffc3a4\" (UID: \"3bf95767-b937-4d51-86ff-a41db2ffc3a4\") " Nov 03 02:52:32.053118 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:32.052913 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bf95767-b937-4d51-86ff-a41db2ffc3a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bf95767-b937-4d51-86ff-a41db2ffc3a4-istiod-ca-cert\") pod \"3bf95767-b937-4d51-86ff-a41db2ffc3a4\" (UID: \"3bf95767-b937-4d51-86ff-a41db2ffc3a4\") " Nov 03 02:52:32.053118 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:32.053034 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bf95767-b937-4d51-86ff-a41db2ffc3a4-credential-socket" pod "3bf95767-b937-4d51-86ff-a41db2ffc3a4" (UID: "3bf95767-b937-4d51-86ff-a41db2ffc3a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:52:32.053118 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:32.053077 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bf95767-b937-4d51-86ff-a41db2ffc3a4-workload-socket" pod "3bf95767-b937-4d51-86ff-a41db2ffc3a4" (UID: "3bf95767-b937-4d51-86ff-a41db2ffc3a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:52:32.053225 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:32.053192 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bf95767-b937-4d51-86ff-a41db2ffc3a4-workload-certs" pod "3bf95767-b937-4d51-86ff-a41db2ffc3a4" (UID: "3bf95767-b937-4d51-86ff-a41db2ffc3a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:52:32.053453 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:32.053407 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bf95767-b937-4d51-86ff-a41db2ffc3a4-istiod-ca-cert" pod "3bf95767-b937-4d51-86ff-a41db2ffc3a4" (UID: "3bf95767-b937-4d51-86ff-a41db2ffc3a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:52:32.053550 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:32.053454 5605 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf95767-b937-4d51-86ff-a41db2ffc3a4-credential-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:52:32.053550 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:32.053478 5605 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf95767-b937-4d51-86ff-a41db2ffc3a4-workload-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:52:32.053550 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:32.053494 5605 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bf95767-b937-4d51-86ff-a41db2ffc3a4-workload-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:52:32.053550 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:32.053507 5605 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bf95767-b937-4d51-86ff-a41db2ffc3a4-istio-data\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:52:32.054840 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:32.054817 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bf95767-b937-4d51-86ff-a41db2ffc3a4-kube-api-access-sqmpv" pod "3bf95767-b937-4d51-86ff-a41db2ffc3a4" (UID: "3bf95767-b937-4d51-86ff-a41db2ffc3a4"). InnerVolumeSpecName "kube-api-access-sqmpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:52:32.054951 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:32.054841 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3bf95767-b937-4d51-86ff-a41db2ffc3a4-istio-podinfo" pod "3bf95767-b937-4d51-86ff-a41db2ffc3a4" (UID: "3bf95767-b937-4d51-86ff-a41db2ffc3a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:52:32.054951 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:32.054894 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bf95767-b937-4d51-86ff-a41db2ffc3a4-istio-envoy" pod "3bf95767-b937-4d51-86ff-a41db2ffc3a4" (UID: "3bf95767-b937-4d51-86ff-a41db2ffc3a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:52:32.055107 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:32.055094 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bf95767-b937-4d51-86ff-a41db2ffc3a4-istio-token" pod "3bf95767-b937-4d51-86ff-a41db2ffc3a4" (UID: "3bf95767-b937-4d51-86ff-a41db2ffc3a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:52:32.154486 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:32.154461 5605 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bf95767-b937-4d51-86ff-a41db2ffc3a4-istio-token\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:52:32.154486 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:32.154486 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqmpv\" (UniqueName: \"kubernetes.io/projected/3bf95767-b937-4d51-86ff-a41db2ffc3a4-kube-api-access-sqmpv\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:52:32.154639 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:32.154496 5605 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bf95767-b937-4d51-86ff-a41db2ffc3a4-istio-envoy\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:52:32.154639 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:32.154505 5605 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bf95767-b937-4d51-86ff-a41db2ffc3a4-istiod-ca-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:52:32.154639 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:32.154514 5605 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bf95767-b937-4d51-86ff-a41db2ffc3a4-istio-podinfo\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:52:32.322341 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:32.322313 5605 generic.go:356] "Generic (PLEG): container finished" podID="3bf95767-b937-4d51-86ff-a41db2ffc3a4" containerID="e8ac827fbcdc946dd425af6281b7ece4a97995efe070da2be9906073783b32da" exitCode=0 Nov 03 02:52:32.322498 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:32.322399 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx" Nov 03 02:52:32.322498 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:32.322395 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx" event={"ID":"3bf95767-b937-4d51-86ff-a41db2ffc3a4","Type":"ContainerDied","Data":"e8ac827fbcdc946dd425af6281b7ece4a97995efe070da2be9906073783b32da"} Nov 03 02:52:32.322637 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:32.322516 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx" event={"ID":"3bf95767-b937-4d51-86ff-a41db2ffc3a4","Type":"ContainerDied","Data":"80cf23efc0c8e687ce8fafeb4cf5ff98b202788e09371e63c421f13729185c58"} Nov 03 02:52:32.322637 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:32.322557 5605 scope.go:122] "RemoveContainer" containerID="e8ac827fbcdc946dd425af6281b7ece4a97995efe070da2be9906073783b32da" Nov 03 02:52:32.333583 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:32.333566 5605 scope.go:122] "RemoveContainer" containerID="158e577028ff4dce01c75829bfb06516735f50c8ef5e1ba438d0c6020dabe8b2" Nov 03 02:52:32.342503 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:32.342486 5605 scope.go:122] "RemoveContainer" containerID="e8ac827fbcdc946dd425af6281b7ece4a97995efe070da2be9906073783b32da" Nov 03 02:52:32.342738 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:52:32.342719 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8ac827fbcdc946dd425af6281b7ece4a97995efe070da2be9906073783b32da\": container with ID starting with e8ac827fbcdc946dd425af6281b7ece4a97995efe070da2be9906073783b32da not found: ID does not exist" containerID="e8ac827fbcdc946dd425af6281b7ece4a97995efe070da2be9906073783b32da" Nov 03 02:52:32.342790 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:32.342746 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8ac827fbcdc946dd425af6281b7ece4a97995efe070da2be9906073783b32da"} err="failed to get container status \"e8ac827fbcdc946dd425af6281b7ece4a97995efe070da2be9906073783b32da\": rpc error: code = NotFound desc = could not find container \"e8ac827fbcdc946dd425af6281b7ece4a97995efe070da2be9906073783b32da\": container with ID starting with e8ac827fbcdc946dd425af6281b7ece4a97995efe070da2be9906073783b32da not found: ID does not exist" Nov 03 02:52:32.342790 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:32.342763 5605 scope.go:122] "RemoveContainer" containerID="158e577028ff4dce01c75829bfb06516735f50c8ef5e1ba438d0c6020dabe8b2" Nov 03 02:52:32.342957 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:52:32.342941 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"158e577028ff4dce01c75829bfb06516735f50c8ef5e1ba438d0c6020dabe8b2\": container with ID starting with 158e577028ff4dce01c75829bfb06516735f50c8ef5e1ba438d0c6020dabe8b2 not found: ID does not exist" containerID="158e577028ff4dce01c75829bfb06516735f50c8ef5e1ba438d0c6020dabe8b2" Nov 03 02:52:32.342993 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:32.342964 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"158e577028ff4dce01c75829bfb06516735f50c8ef5e1ba438d0c6020dabe8b2"} err="failed to get container status \"158e577028ff4dce01c75829bfb06516735f50c8ef5e1ba438d0c6020dabe8b2\": rpc error: code = NotFound desc = could not find container \"158e577028ff4dce01c75829bfb06516735f50c8ef5e1ba438d0c6020dabe8b2\": container with ID starting with 158e577028ff4dce01c75829bfb06516735f50c8ef5e1ba438d0c6020dabe8b2 not found: ID does not exist" Nov 03 02:52:32.478632 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:32.478605 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx"] Nov 03 02:52:32.483170 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:32.483148 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-c8ee6b-openshift-default-797854c84f-pzznx"] Nov 03 02:52:32.850995 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:32.850966 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bf95767-b937-4d51-86ff-a41db2ffc3a4" path="/var/lib/kubelet/pods/3bf95767-b937-4d51-86ff-a41db2ffc3a4/volumes" Nov 03 02:52:35.193210 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:35.193167 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-54bbc97448-ntmsd" podUID="818446a4-a6e6-413c-b698-7cdcb207922c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:52:35.259709 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:35.259673 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-95152a-5d959db47f-w6cgh" Nov 03 02:52:36.980273 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:36.980237 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ecdbc7-557dbfdccc-7bfdq"] Nov 03 02:52:36.980846 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:36.980822 5605 cpu_manager.go:414] "containerMap: removing container" podUID="3bf95767-b937-4d51-86ff-a41db2ffc3a4" containerName="istio-proxy" Nov 03 02:52:36.980846 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:36.980844 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bf95767-b937-4d51-86ff-a41db2ffc3a4" containerName="istio-proxy" Nov 03 02:52:36.981008 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:36.980855 5605 cpu_manager.go:414] "containerMap: removing container" podUID="3bf95767-b937-4d51-86ff-a41db2ffc3a4" containerName="istio-proxy" Nov 03 02:52:36.981008 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:36.980861 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bf95767-b937-4d51-86ff-a41db2ffc3a4" containerName="istio-proxy" Nov 03 02:52:36.981008 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:36.980977 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bf95767-b937-4d51-86ff-a41db2ffc3a4" containerName="istio-proxy" Nov 03 02:52:36.981008 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:36.980985 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bf95767-b937-4d51-86ff-a41db2ffc3a4" containerName="istio-proxy" Nov 03 02:52:36.984294 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:36.984267 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ecdbc7-557dbfdccc-7bfdq" Nov 03 02:52:37.037442 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:37.037421 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ecdbc7-557dbfdccc-7bfdq"] Nov 03 02:52:37.081783 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:37.081762 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-ecdbc7\"" Nov 03 02:52:37.081783 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:37.081776 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-ecdbc7-dockercfg-xzt25\"" Nov 03 02:52:37.103840 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:37.103816 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wpk9\" (UniqueName: \"kubernetes.io/projected/072868fd-5f17-42fc-890b-e72ffec89324-kube-api-access-2wpk9\") pod \"payload-processing-e2e-inf-a-ecdbc7-557dbfdccc-7bfdq\" (UID: \"072868fd-5f17-42fc-890b-e72ffec89324\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ecdbc7-557dbfdccc-7bfdq" Nov 03 02:52:37.103931 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:37.103867 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/072868fd-5f17-42fc-890b-e72ffec89324-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ecdbc7-557dbfdccc-7bfdq\" (UID: \"072868fd-5f17-42fc-890b-e72ffec89324\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ecdbc7-557dbfdccc-7bfdq" Nov 03 02:52:37.204709 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:37.204682 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wpk9\" (UniqueName: \"kubernetes.io/projected/072868fd-5f17-42fc-890b-e72ffec89324-kube-api-access-2wpk9\") pod \"payload-processing-e2e-inf-a-ecdbc7-557dbfdccc-7bfdq\" (UID: \"072868fd-5f17-42fc-890b-e72ffec89324\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ecdbc7-557dbfdccc-7bfdq" Nov 03 02:52:37.204848 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:37.204832 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/072868fd-5f17-42fc-890b-e72ffec89324-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ecdbc7-557dbfdccc-7bfdq\" (UID: \"072868fd-5f17-42fc-890b-e72ffec89324\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ecdbc7-557dbfdccc-7bfdq" Nov 03 02:52:37.205473 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:37.205456 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/072868fd-5f17-42fc-890b-e72ffec89324-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ecdbc7-557dbfdccc-7bfdq\" (UID: \"072868fd-5f17-42fc-890b-e72ffec89324\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ecdbc7-557dbfdccc-7bfdq" Nov 03 02:52:37.259072 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:37.259030 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wpk9\" (UniqueName: \"kubernetes.io/projected/072868fd-5f17-42fc-890b-e72ffec89324-kube-api-access-2wpk9\") pod \"payload-processing-e2e-inf-a-ecdbc7-557dbfdccc-7bfdq\" (UID: \"072868fd-5f17-42fc-890b-e72ffec89324\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ecdbc7-557dbfdccc-7bfdq" Nov 03 02:52:37.299036 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:37.299017 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ecdbc7-557dbfdccc-7bfdq" Nov 03 02:52:37.667927 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:37.667898 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ecdbc7-557dbfdccc-7bfdq"] Nov 03 02:52:37.673809 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:52:37.673784 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod072868fd_5f17_42fc_890b_e72ffec89324.slice/crio-f311d5da50b4c05920f8c7bffc454e7527788b998157956d65abbc7c7955ee2e WatchSource:0}: Error finding container f311d5da50b4c05920f8c7bffc454e7527788b998157956d65abbc7c7955ee2e: Status 404 returned error can't find the container with id f311d5da50b4c05920f8c7bffc454e7527788b998157956d65abbc7c7955ee2e Nov 03 02:52:38.364851 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:38.362614 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ecdbc7-557dbfdccc-7bfdq" event={"ID":"072868fd-5f17-42fc-890b-e72ffec89324","Type":"ContainerStarted","Data":"2432f18e908d61f4ab0fdc4b692223d4da8317a9c048c4a91b30fd1ae92fc838"} Nov 03 02:52:38.364851 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:38.362677 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ecdbc7-557dbfdccc-7bfdq" event={"ID":"072868fd-5f17-42fc-890b-e72ffec89324","Type":"ContainerStarted","Data":"f311d5da50b4c05920f8c7bffc454e7527788b998157956d65abbc7c7955ee2e"} Nov 03 02:52:38.364851 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:38.362763 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ecdbc7-557dbfdccc-7bfdq" Nov 03 02:52:38.472438 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:38.472388 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-ecdbc7-557dbfdccc-7bfdq" podStartSLOduration=2.4723755 podStartE2EDuration="2.4723755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:52:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:52:38.470082341 +0000 UTC m=+3635.729280494" watchObservedRunningTime="2026-11-03 02:52:38.4723755 +0000 UTC m=+3635.731573649" Nov 03 02:52:45.193416 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:45.193353 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-54bbc97448-ntmsd" podUID="818446a4-a6e6-413c-b698-7cdcb207922c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:52:49.014554 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:52:49.014523 5605 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-50a3cab9906d: secret "agc-cred-50a3cab9906d" not found Nov 03 02:52:49.014554 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:52:49.014557 5605 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-68686f579-ntp6g: secret "agc-cred-50a3cab9906d" not found Nov 03 02:52:49.015020 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:52:49.014638 5605 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d89cb81e-1adf-4e6c-82ea-dde66712aab7-provider-credentials podName:d89cb81e-1adf-4e6c-82ea-dde66712aab7 nodeName:}" failed. No retries permitted until 2026-11-03 02:52:49.514619468 +0000 UTC m=+3646.773817622 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d89cb81e-1adf-4e6c-82ea-dde66712aab7-provider-credentials") pod "payload-processing-external-model-68686f579-ntp6g" (UID: "d89cb81e-1adf-4e6c-82ea-dde66712aab7") : secret "agc-cred-50a3cab9906d" not found Nov 03 02:52:49.370973 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:49.370947 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ecdbc7-557dbfdccc-7bfdq" Nov 03 02:52:49.518944 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:52:49.518910 5605 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-50a3cab9906d: secret "agc-cred-50a3cab9906d" not found Nov 03 02:52:49.518944 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:52:49.518945 5605 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-68686f579-ntp6g: secret "agc-cred-50a3cab9906d" not found Nov 03 02:52:49.519199 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:52:49.519014 5605 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d89cb81e-1adf-4e6c-82ea-dde66712aab7-provider-credentials podName:d89cb81e-1adf-4e6c-82ea-dde66712aab7 nodeName:}" failed. No retries permitted until 2026-11-03 02:52:50.51899642 +0000 UTC m=+3647.778194564 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d89cb81e-1adf-4e6c-82ea-dde66712aab7-provider-credentials") pod "payload-processing-external-model-68686f579-ntp6g" (UID: "d89cb81e-1adf-4e6c-82ea-dde66712aab7") : secret "agc-cred-50a3cab9906d" not found Nov 03 02:52:50.527879 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:52:50.527849 5605 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-50a3cab9906d: secret "agc-cred-50a3cab9906d" not found Nov 03 02:52:50.527879 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:52:50.527875 5605 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-68686f579-ntp6g: secret "agc-cred-50a3cab9906d" not found Nov 03 02:52:50.528317 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:52:50.527932 5605 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d89cb81e-1adf-4e6c-82ea-dde66712aab7-provider-credentials podName:d89cb81e-1adf-4e6c-82ea-dde66712aab7 nodeName:}" failed. No retries permitted until 2026-11-03 02:52:52.527919407 +0000 UTC m=+3649.787117539 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d89cb81e-1adf-4e6c-82ea-dde66712aab7-provider-credentials") pod "payload-processing-external-model-68686f579-ntp6g" (UID: "d89cb81e-1adf-4e6c-82ea-dde66712aab7") : secret "agc-cred-50a3cab9906d" not found Nov 03 02:52:51.967422 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:51.967393 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-644zf"] Nov 03 02:52:51.972297 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:51.972279 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-644zf" Nov 03 02:52:52.142104 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:52.142078 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b3468eb-1b4e-4727-910f-d975527c8644-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-644zf\" (UID: \"0b3468eb-1b4e-4727-910f-d975527c8644\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-644zf" Nov 03 02:52:52.142251 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:52.142112 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b3468eb-1b4e-4727-910f-d975527c8644-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-644zf\" (UID: \"0b3468eb-1b4e-4727-910f-d975527c8644\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-644zf" Nov 03 02:52:52.142251 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:52.142138 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b3468eb-1b4e-4727-910f-d975527c8644-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-644zf\" (UID: \"0b3468eb-1b4e-4727-910f-d975527c8644\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-644zf" Nov 03 02:52:52.142251 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:52.142245 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b3468eb-1b4e-4727-910f-d975527c8644-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-644zf\" (UID: \"0b3468eb-1b4e-4727-910f-d975527c8644\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-644zf" Nov 03 02:52:52.142455 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:52.142266 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b3468eb-1b4e-4727-910f-d975527c8644-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-644zf\" (UID: \"0b3468eb-1b4e-4727-910f-d975527c8644\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-644zf" Nov 03 02:52:52.142455 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:52.142443 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b3468eb-1b4e-4727-910f-d975527c8644-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-644zf\" (UID: \"0b3468eb-1b4e-4727-910f-d975527c8644\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-644zf" Nov 03 02:52:52.142573 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:52.142497 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b3468eb-1b4e-4727-910f-d975527c8644-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-644zf\" (UID: \"0b3468eb-1b4e-4727-910f-d975527c8644\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-644zf" Nov 03 02:52:52.142573 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:52.142563 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b3468eb-1b4e-4727-910f-d975527c8644-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-644zf\" (UID: \"0b3468eb-1b4e-4727-910f-d975527c8644\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-644zf" Nov 03 02:52:52.142658 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:52.142630 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcsz6\" (UniqueName: \"kubernetes.io/projected/0b3468eb-1b4e-4727-910f-d975527c8644-kube-api-access-mcsz6\") pod \"maas-default-gateway-openshift-default-5bc884dd87-644zf\" (UID: \"0b3468eb-1b4e-4727-910f-d975527c8644\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-644zf" Nov 03 02:52:52.167589 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:52.167567 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-644zf"] Nov 03 02:52:52.243945 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:52.243872 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b3468eb-1b4e-4727-910f-d975527c8644-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-644zf\" (UID: \"0b3468eb-1b4e-4727-910f-d975527c8644\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-644zf" Nov 03 02:52:52.243945 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:52.243937 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b3468eb-1b4e-4727-910f-d975527c8644-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-644zf\" (UID: \"0b3468eb-1b4e-4727-910f-d975527c8644\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-644zf" Nov 03 02:52:52.244143 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:52.244000 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcsz6\" (UniqueName: \"kubernetes.io/projected/0b3468eb-1b4e-4727-910f-d975527c8644-kube-api-access-mcsz6\") pod \"maas-default-gateway-openshift-default-5bc884dd87-644zf\" (UID: \"0b3468eb-1b4e-4727-910f-d975527c8644\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-644zf" Nov 03 02:52:52.244143 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:52.244047 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b3468eb-1b4e-4727-910f-d975527c8644-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-644zf\" (UID: \"0b3468eb-1b4e-4727-910f-d975527c8644\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-644zf" Nov 03 02:52:52.244143 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:52.244078 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b3468eb-1b4e-4727-910f-d975527c8644-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-644zf\" (UID: \"0b3468eb-1b4e-4727-910f-d975527c8644\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-644zf" Nov 03 02:52:52.244143 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:52.244108 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b3468eb-1b4e-4727-910f-d975527c8644-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-644zf\" (UID: \"0b3468eb-1b4e-4727-910f-d975527c8644\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-644zf" Nov 03 02:52:52.244396 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:52.244153 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b3468eb-1b4e-4727-910f-d975527c8644-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-644zf\" (UID: \"0b3468eb-1b4e-4727-910f-d975527c8644\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-644zf" Nov 03 02:52:52.244396 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:52.244176 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b3468eb-1b4e-4727-910f-d975527c8644-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-644zf\" (UID: \"0b3468eb-1b4e-4727-910f-d975527c8644\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-644zf" Nov 03 02:52:52.244396 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:52.244341 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b3468eb-1b4e-4727-910f-d975527c8644-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-644zf\" (UID: \"0b3468eb-1b4e-4727-910f-d975527c8644\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-644zf" Nov 03 02:52:52.244560 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:52.244412 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b3468eb-1b4e-4727-910f-d975527c8644-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-644zf\" (UID: \"0b3468eb-1b4e-4727-910f-d975527c8644\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-644zf" Nov 03 02:52:52.244957 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:52.244930 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b3468eb-1b4e-4727-910f-d975527c8644-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-644zf\" (UID: \"0b3468eb-1b4e-4727-910f-d975527c8644\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-644zf" Nov 03 02:52:52.245063 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:52.244946 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b3468eb-1b4e-4727-910f-d975527c8644-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-644zf\" (UID: \"0b3468eb-1b4e-4727-910f-d975527c8644\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-644zf" Nov 03 02:52:52.245110 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:52.245088 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b3468eb-1b4e-4727-910f-d975527c8644-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-644zf\" (UID: \"0b3468eb-1b4e-4727-910f-d975527c8644\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-644zf" Nov 03 02:52:52.245209 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:52.245194 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b3468eb-1b4e-4727-910f-d975527c8644-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-644zf\" (UID: \"0b3468eb-1b4e-4727-910f-d975527c8644\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-644zf" Nov 03 02:52:52.246209 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:52.246192 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b3468eb-1b4e-4727-910f-d975527c8644-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-644zf\" (UID: \"0b3468eb-1b4e-4727-910f-d975527c8644\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-644zf" Nov 03 02:52:52.246478 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:52.246459 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b3468eb-1b4e-4727-910f-d975527c8644-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-644zf\" (UID: \"0b3468eb-1b4e-4727-910f-d975527c8644\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-644zf" Nov 03 02:52:52.298817 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:52.298798 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b3468eb-1b4e-4727-910f-d975527c8644-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-644zf\" (UID: \"0b3468eb-1b4e-4727-910f-d975527c8644\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-644zf" Nov 03 02:52:52.299768 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:52.299743 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcsz6\" (UniqueName: \"kubernetes.io/projected/0b3468eb-1b4e-4727-910f-d975527c8644-kube-api-access-mcsz6\") pod \"maas-default-gateway-openshift-default-5bc884dd87-644zf\" (UID: \"0b3468eb-1b4e-4727-910f-d975527c8644\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-644zf" Nov 03 02:52:52.547505 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:52:52.547431 5605 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-50a3cab9906d: secret "agc-cred-50a3cab9906d" not found Nov 03 02:52:52.547505 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:52:52.547457 5605 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-68686f579-ntp6g: secret "agc-cred-50a3cab9906d" not found Nov 03 02:52:52.547704 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:52:52.547526 5605 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d89cb81e-1adf-4e6c-82ea-dde66712aab7-provider-credentials podName:d89cb81e-1adf-4e6c-82ea-dde66712aab7 nodeName:}" failed. No retries permitted until 2026-11-03 02:52:56.547509076 +0000 UTC m=+3653.806707216 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d89cb81e-1adf-4e6c-82ea-dde66712aab7-provider-credentials") pod "payload-processing-external-model-68686f579-ntp6g" (UID: "d89cb81e-1adf-4e6c-82ea-dde66712aab7") : secret "agc-cred-50a3cab9906d" not found Nov 03 02:52:52.584264 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:52.584241 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-644zf" Nov 03 02:52:53.005486 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:53.005453 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-644zf"] Nov 03 02:52:53.049445 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:52:53.049419 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b3468eb_1b4e_4727_910f_d975527c8644.slice/crio-e60ec527a6fa16fd68d585442528876833fccc4d3c32be54ed7c61ac7cc792ac WatchSource:0}: Error finding container e60ec527a6fa16fd68d585442528876833fccc4d3c32be54ed7c61ac7cc792ac: Status 404 returned error can't find the container with id e60ec527a6fa16fd68d585442528876833fccc4d3c32be54ed7c61ac7cc792ac Nov 03 02:52:53.051859 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:53.051827 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:52:53.051955 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:53.051907 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:52:53.052021 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:53.051954 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:52:53.446708 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:53.446666 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-644zf" event={"ID":"0b3468eb-1b4e-4727-910f-d975527c8644","Type":"ContainerStarted","Data":"8681bb515ff81651c2615f5062da8424e8fcd1ea05484a48046bf79e4fa44060"} Nov 03 02:52:53.446878 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:53.446730 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-644zf" event={"ID":"0b3468eb-1b4e-4727-910f-d975527c8644","Type":"ContainerStarted","Data":"e60ec527a6fa16fd68d585442528876833fccc4d3c32be54ed7c61ac7cc792ac"} Nov 03 02:52:53.585475 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:53.585432 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-644zf" Nov 03 02:52:53.652993 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:53.652937 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-644zf" podStartSLOduration=2.652921929 podStartE2EDuration="2.652921929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:52:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:52:53.605051905 +0000 UTC m=+3650.864250058" watchObservedRunningTime="2026-11-03 02:52:53.652921929 +0000 UTC m=+3650.912120081" Nov 03 02:52:53.916983 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:52:53.916946 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:52:54.585987 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:54.585954 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-644zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body= Nov 03 02:52:54.586470 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:54.586005 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-644zf" podUID="0b3468eb-1b4e-4727-910f-d975527c8644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" Nov 03 02:52:55.193704 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:55.193667 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-54bbc97448-ntmsd" podUID="818446a4-a6e6-413c-b698-7cdcb207922c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:52:55.585518 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:55.585487 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-644zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body= Nov 03 02:52:55.585698 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:55.585535 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-644zf" podUID="0b3468eb-1b4e-4727-910f-d975527c8644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" Nov 03 02:52:56.585589 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:56.585556 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-644zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body= Nov 03 02:52:56.586056 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:56.585622 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-644zf" podUID="0b3468eb-1b4e-4727-910f-d975527c8644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" Nov 03 02:52:56.605432 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:52:56.605405 5605 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-50a3cab9906d: secret "agc-cred-50a3cab9906d" not found Nov 03 02:52:56.605432 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:52:56.605428 5605 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-68686f579-ntp6g: secret "agc-cred-50a3cab9906d" not found Nov 03 02:52:56.605580 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:52:56.605485 5605 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d89cb81e-1adf-4e6c-82ea-dde66712aab7-provider-credentials podName:d89cb81e-1adf-4e6c-82ea-dde66712aab7 nodeName:}" failed. No retries permitted until 2026-11-03 02:53:04.605472276 +0000 UTC m=+3661.864670420 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d89cb81e-1adf-4e6c-82ea-dde66712aab7-provider-credentials") pod "payload-processing-external-model-68686f579-ntp6g" (UID: "d89cb81e-1adf-4e6c-82ea-dde66712aab7") : secret "agc-cred-50a3cab9906d" not found Nov 03 02:52:57.585035 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:57.585001 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-644zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body= Nov 03 02:52:57.585214 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:57.585053 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-644zf" podUID="0b3468eb-1b4e-4727-910f-d975527c8644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" Nov 03 02:52:58.584617 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:58.584575 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-644zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body= Nov 03 02:52:58.585003 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:58.584629 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-644zf" podUID="0b3468eb-1b4e-4727-910f-d975527c8644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" Nov 03 02:52:59.585382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:59.585326 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-644zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body= Nov 03 02:52:59.585876 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:52:59.585399 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-644zf" podUID="0b3468eb-1b4e-4727-910f-d975527c8644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" Nov 03 02:53:01.368199 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:01.368166 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-644zf" Nov 03 02:53:01.369387 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:01.369343 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-644zf" Nov 03 02:53:01.370220 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:01.370201 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-644zf" Nov 03 02:53:04.618382 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:53:04.610305 5605 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-50a3cab9906d: secret "agc-cred-50a3cab9906d" not found Nov 03 02:53:04.618382 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:53:04.610341 5605 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-68686f579-ntp6g: secret "agc-cred-50a3cab9906d" not found Nov 03 02:53:04.618382 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:53:04.610412 5605 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d89cb81e-1adf-4e6c-82ea-dde66712aab7-provider-credentials podName:d89cb81e-1adf-4e6c-82ea-dde66712aab7 nodeName:}" failed. No retries permitted until 2026-11-03 02:53:20.610394904 +0000 UTC m=+3677.869593043 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d89cb81e-1adf-4e6c-82ea-dde66712aab7-provider-credentials") pod "payload-processing-external-model-68686f579-ntp6g" (UID: "d89cb81e-1adf-4e6c-82ea-dde66712aab7") : secret "agc-cred-50a3cab9906d" not found Nov 03 02:53:05.193726 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:05.193692 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-54bbc97448-ntmsd" podUID="818446a4-a6e6-413c-b698-7cdcb207922c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:53:08.573450 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:08.573415 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:53:15.193459 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:15.193424 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-54bbc97448-ntmsd" podUID="818446a4-a6e6-413c-b698-7cdcb207922c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:53:19.109688 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:19.109655 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-dc14f6-85cdc965cf-7v5lt"] Nov 03 02:53:19.115028 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:19.115008 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dc14f6-85cdc965cf-7v5lt" Nov 03 02:53:19.166879 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:19.166855 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-dc14f6-85cdc965cf-7v5lt"] Nov 03 02:53:19.210620 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:19.210597 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-dc14f6\"" Nov 03 02:53:19.265197 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:19.265141 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pjqn\" (UniqueName: \"kubernetes.io/projected/874c955f-0655-4260-8110-6509aef377ca-kube-api-access-7pjqn\") pod \"payload-pre-processing-e2e-extmod-b-dc14f6-85cdc965cf-7v5lt\" (UID: \"874c955f-0655-4260-8110-6509aef377ca\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dc14f6-85cdc965cf-7v5lt" Nov 03 02:53:19.265437 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:19.265418 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/874c955f-0655-4260-8110-6509aef377ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-dc14f6-85cdc965cf-7v5lt\" (UID: \"874c955f-0655-4260-8110-6509aef377ca\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dc14f6-85cdc965cf-7v5lt" Nov 03 02:53:19.293693 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:19.293655 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-dc14f6-94dd86c89-bqhhz"] Nov 03 02:53:19.297823 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:19.297802 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-dc14f6-94dd86c89-bqhhz" Nov 03 02:53:19.322299 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:19.322269 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-dc14f6-94dd86c89-bqhhz"] Nov 03 02:53:19.346559 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:19.346532 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-dc14f6-dockercfg-d4qzr\"" Nov 03 02:53:19.373155 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:19.373089 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/874c955f-0655-4260-8110-6509aef377ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-dc14f6-85cdc965cf-7v5lt\" (UID: \"874c955f-0655-4260-8110-6509aef377ca\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dc14f6-85cdc965cf-7v5lt" Nov 03 02:53:19.373155 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:19.373155 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pjqn\" (UniqueName: \"kubernetes.io/projected/874c955f-0655-4260-8110-6509aef377ca-kube-api-access-7pjqn\") pod \"payload-pre-processing-e2e-extmod-b-dc14f6-85cdc965cf-7v5lt\" (UID: \"874c955f-0655-4260-8110-6509aef377ca\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dc14f6-85cdc965cf-7v5lt" Nov 03 02:53:19.373811 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:19.373792 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/874c955f-0655-4260-8110-6509aef377ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-dc14f6-85cdc965cf-7v5lt\" (UID: \"874c955f-0655-4260-8110-6509aef377ca\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dc14f6-85cdc965cf-7v5lt" Nov 03 02:53:19.427125 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:19.427100 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pjqn\" (UniqueName: \"kubernetes.io/projected/874c955f-0655-4260-8110-6509aef377ca-kube-api-access-7pjqn\") pod \"payload-pre-processing-e2e-extmod-b-dc14f6-85cdc965cf-7v5lt\" (UID: \"874c955f-0655-4260-8110-6509aef377ca\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dc14f6-85cdc965cf-7v5lt" Nov 03 02:53:19.427267 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:19.427245 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dc14f6-85cdc965cf-7v5lt" Nov 03 02:53:19.475106 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:19.475077 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29a7b266-20d5-4fe6-8a0e-c57ef087f0bb-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-dc14f6-94dd86c89-bqhhz\" (UID: \"29a7b266-20d5-4fe6-8a0e-c57ef087f0bb\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-dc14f6-94dd86c89-bqhhz" Nov 03 02:53:19.475232 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:19.475119 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z7w4\" (UniqueName: \"kubernetes.io/projected/29a7b266-20d5-4fe6-8a0e-c57ef087f0bb-kube-api-access-6z7w4\") pod \"payload-processing-e2e-extmod-b-dc14f6-94dd86c89-bqhhz\" (UID: \"29a7b266-20d5-4fe6-8a0e-c57ef087f0bb\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-dc14f6-94dd86c89-bqhhz" Nov 03 02:53:19.576475 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:19.576443 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29a7b266-20d5-4fe6-8a0e-c57ef087f0bb-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-dc14f6-94dd86c89-bqhhz\" (UID: \"29a7b266-20d5-4fe6-8a0e-c57ef087f0bb\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-dc14f6-94dd86c89-bqhhz" Nov 03 02:53:19.576622 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:19.576478 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z7w4\" (UniqueName: \"kubernetes.io/projected/29a7b266-20d5-4fe6-8a0e-c57ef087f0bb-kube-api-access-6z7w4\") pod \"payload-processing-e2e-extmod-b-dc14f6-94dd86c89-bqhhz\" (UID: \"29a7b266-20d5-4fe6-8a0e-c57ef087f0bb\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-dc14f6-94dd86c89-bqhhz" Nov 03 02:53:19.577185 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:19.577161 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29a7b266-20d5-4fe6-8a0e-c57ef087f0bb-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-dc14f6-94dd86c89-bqhhz\" (UID: \"29a7b266-20d5-4fe6-8a0e-c57ef087f0bb\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-dc14f6-94dd86c89-bqhhz" Nov 03 02:53:19.630101 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:19.630044 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z7w4\" (UniqueName: \"kubernetes.io/projected/29a7b266-20d5-4fe6-8a0e-c57ef087f0bb-kube-api-access-6z7w4\") pod \"payload-processing-e2e-extmod-b-dc14f6-94dd86c89-bqhhz\" (UID: \"29a7b266-20d5-4fe6-8a0e-c57ef087f0bb\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-dc14f6-94dd86c89-bqhhz" Nov 03 02:53:19.790675 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:19.790646 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-dc14f6-85cdc965cf-7v5lt"] Nov 03 02:53:19.800079 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:53:19.800039 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod874c955f_0655_4260_8110_6509aef377ca.slice/crio-c3199b7f844d13f2cdd7ad3e588158a43fbb50b5cd92322a0561c63fd7688712 WatchSource:0}: Error finding container c3199b7f844d13f2cdd7ad3e588158a43fbb50b5cd92322a0561c63fd7688712: Status 404 returned error can't find the container with id c3199b7f844d13f2cdd7ad3e588158a43fbb50b5cd92322a0561c63fd7688712 Nov 03 02:53:19.913278 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:19.913226 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-dc14f6-94dd86c89-bqhhz" Nov 03 02:53:20.277123 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:20.277044 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-dc14f6-94dd86c89-bqhhz"] Nov 03 02:53:20.283193 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:53:20.283159 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29a7b266_20d5_4fe6_8a0e_c57ef087f0bb.slice/crio-8b25e6eb2a9cc403a60877204df68c41587f33c91db2476686286f572b7d35a9 WatchSource:0}: Error finding container 8b25e6eb2a9cc403a60877204df68c41587f33c91db2476686286f572b7d35a9: Status 404 returned error can't find the container with id 8b25e6eb2a9cc403a60877204df68c41587f33c91db2476686286f572b7d35a9 Nov 03 02:53:20.625976 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:20.625931 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-dc14f6-94dd86c89-bqhhz" event={"ID":"29a7b266-20d5-4fe6-8a0e-c57ef087f0bb","Type":"ContainerStarted","Data":"c8a2fd39f03a922ab1ebaecabaa77ae3daaafe53feffb0a2efacbbcc2de324df"} Nov 03 02:53:20.626163 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:20.626032 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-dc14f6-94dd86c89-bqhhz" Nov 03 02:53:20.626163 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:20.626056 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-dc14f6-94dd86c89-bqhhz" event={"ID":"29a7b266-20d5-4fe6-8a0e-c57ef087f0bb","Type":"ContainerStarted","Data":"8b25e6eb2a9cc403a60877204df68c41587f33c91db2476686286f572b7d35a9"} Nov 03 02:53:20.627495 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:20.627471 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dc14f6-85cdc965cf-7v5lt" event={"ID":"874c955f-0655-4260-8110-6509aef377ca","Type":"ContainerStarted","Data":"e0dcdeaa117df17565d95bb3747973defbeda34263fad5b0ee4169c437aa8949"} Nov 03 02:53:20.627618 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:20.627514 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dc14f6-85cdc965cf-7v5lt" event={"ID":"874c955f-0655-4260-8110-6509aef377ca","Type":"ContainerStarted","Data":"c3199b7f844d13f2cdd7ad3e588158a43fbb50b5cd92322a0561c63fd7688712"} Nov 03 02:53:20.627618 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:20.627548 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dc14f6-85cdc965cf-7v5lt" Nov 03 02:53:20.686480 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:53:20.686457 5605 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-50a3cab9906d: secret "agc-cred-50a3cab9906d" not found Nov 03 02:53:20.686480 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:53:20.686477 5605 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-68686f579-ntp6g: secret "agc-cred-50a3cab9906d" not found Nov 03 02:53:20.686605 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:53:20.686525 5605 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d89cb81e-1adf-4e6c-82ea-dde66712aab7-provider-credentials podName:d89cb81e-1adf-4e6c-82ea-dde66712aab7 nodeName:}" failed. No retries permitted until 2026-11-03 02:53:52.686511565 +0000 UTC m=+3709.945709710 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d89cb81e-1adf-4e6c-82ea-dde66712aab7-provider-credentials") pod "payload-processing-external-model-68686f579-ntp6g" (UID: "d89cb81e-1adf-4e6c-82ea-dde66712aab7") : secret "agc-cred-50a3cab9906d" not found Nov 03 02:53:20.732906 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:20.732860 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-dc14f6-94dd86c89-bqhhz" podStartSLOduration=1.7328510750000001 podStartE2EDuration="1.732851075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:53:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:53:20.732576209 +0000 UTC m=+3677.991774363" watchObservedRunningTime="2026-11-03 02:53:20.732851075 +0000 UTC m=+3677.992049227" Nov 03 02:53:20.842162 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:20.842111 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dc14f6-85cdc965cf-7v5lt" podStartSLOduration=1.8420942249999999 podStartE2EDuration="1.842094225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:53:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:53:20.838261669 +0000 UTC m=+3678.097459821" watchObservedRunningTime="2026-11-03 02:53:20.842094225 +0000 UTC m=+3678.101292378" Nov 03 02:53:23.914317 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:53:23.914280 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:53:25.193711 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:25.193671 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-54bbc97448-ntmsd" podUID="818446a4-a6e6-413c-b698-7cdcb207922c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:53:30.844432 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:30.844398 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-54bbc97448-ntmsd" podUID="818446a4-a6e6-413c-b698-7cdcb207922c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:53:31.635406 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:31.635377 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-dc14f6-94dd86c89-bqhhz" Nov 03 02:53:31.635580 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:31.635564 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dc14f6-85cdc965cf-7v5lt" Nov 03 02:53:40.845364 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:40.845262 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-54bbc97448-ntmsd" podUID="818446a4-a6e6-413c-b698-7cdcb207922c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:53:49.010958 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:49.010926 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:53:50.844561 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:50.844524 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-54bbc97448-ntmsd" podUID="818446a4-a6e6-413c-b698-7cdcb207922c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:53:52.428054 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:52.428021 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-flj4q"] Nov 03 02:53:52.432976 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:52.432958 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-flj4q" Nov 03 02:53:52.523165 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:52.523137 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-flj4q"] Nov 03 02:53:52.541125 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:52.541093 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d76cc067-c62a-40e3-90d2-6435dc8cf655-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-flj4q\" (UID: \"d76cc067-c62a-40e3-90d2-6435dc8cf655\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-flj4q" Nov 03 02:53:52.541254 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:52.541139 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d76cc067-c62a-40e3-90d2-6435dc8cf655-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-flj4q\" (UID: \"d76cc067-c62a-40e3-90d2-6435dc8cf655\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-flj4q" Nov 03 02:53:52.541254 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:52.541217 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d76cc067-c62a-40e3-90d2-6435dc8cf655-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-flj4q\" (UID: \"d76cc067-c62a-40e3-90d2-6435dc8cf655\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-flj4q" Nov 03 02:53:52.541254 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:52.541245 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d76cc067-c62a-40e3-90d2-6435dc8cf655-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-flj4q\" (UID: \"d76cc067-c62a-40e3-90d2-6435dc8cf655\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-flj4q" Nov 03 02:53:52.541409 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:52.541281 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62qbl\" (UniqueName: \"kubernetes.io/projected/d76cc067-c62a-40e3-90d2-6435dc8cf655-kube-api-access-62qbl\") pod \"maas-default-gateway-openshift-default-5bc884dd87-flj4q\" (UID: \"d76cc067-c62a-40e3-90d2-6435dc8cf655\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-flj4q" Nov 03 02:53:52.541409 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:52.541384 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d76cc067-c62a-40e3-90d2-6435dc8cf655-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-flj4q\" (UID: \"d76cc067-c62a-40e3-90d2-6435dc8cf655\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-flj4q" Nov 03 02:53:52.541487 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:52.541409 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d76cc067-c62a-40e3-90d2-6435dc8cf655-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-flj4q\" (UID: \"d76cc067-c62a-40e3-90d2-6435dc8cf655\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-flj4q" Nov 03 02:53:52.541487 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:52.541440 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d76cc067-c62a-40e3-90d2-6435dc8cf655-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-flj4q\" (UID: \"d76cc067-c62a-40e3-90d2-6435dc8cf655\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-flj4q" Nov 03 02:53:52.541487 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:52.541455 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d76cc067-c62a-40e3-90d2-6435dc8cf655-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-flj4q\" (UID: \"d76cc067-c62a-40e3-90d2-6435dc8cf655\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-flj4q" Nov 03 02:53:52.642735 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:52.642698 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d76cc067-c62a-40e3-90d2-6435dc8cf655-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-flj4q\" (UID: \"d76cc067-c62a-40e3-90d2-6435dc8cf655\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-flj4q" Nov 03 02:53:52.642735 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:52.642734 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d76cc067-c62a-40e3-90d2-6435dc8cf655-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-flj4q\" (UID: \"d76cc067-c62a-40e3-90d2-6435dc8cf655\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-flj4q" Nov 03 02:53:52.642983 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:52.642776 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d76cc067-c62a-40e3-90d2-6435dc8cf655-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-flj4q\" (UID: \"d76cc067-c62a-40e3-90d2-6435dc8cf655\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-flj4q" Nov 03 02:53:52.642983 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:52.642817 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d76cc067-c62a-40e3-90d2-6435dc8cf655-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-flj4q\" (UID: \"d76cc067-c62a-40e3-90d2-6435dc8cf655\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-flj4q" Nov 03 02:53:52.642983 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:52.642845 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d76cc067-c62a-40e3-90d2-6435dc8cf655-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-flj4q\" (UID: \"d76cc067-c62a-40e3-90d2-6435dc8cf655\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-flj4q" Nov 03 02:53:52.642983 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:52.642866 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d76cc067-c62a-40e3-90d2-6435dc8cf655-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-flj4q\" (UID: \"d76cc067-c62a-40e3-90d2-6435dc8cf655\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-flj4q" Nov 03 02:53:52.642983 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:52.642920 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62qbl\" (UniqueName: \"kubernetes.io/projected/d76cc067-c62a-40e3-90d2-6435dc8cf655-kube-api-access-62qbl\") pod \"maas-default-gateway-openshift-default-5bc884dd87-flj4q\" (UID: \"d76cc067-c62a-40e3-90d2-6435dc8cf655\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-flj4q" Nov 03 02:53:52.643273 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:52.643054 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d76cc067-c62a-40e3-90d2-6435dc8cf655-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-flj4q\" (UID: \"d76cc067-c62a-40e3-90d2-6435dc8cf655\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-flj4q" Nov 03 02:53:52.643273 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:52.643083 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d76cc067-c62a-40e3-90d2-6435dc8cf655-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-flj4q\" (UID: \"d76cc067-c62a-40e3-90d2-6435dc8cf655\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-flj4q" Nov 03 02:53:52.643273 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:52.643169 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d76cc067-c62a-40e3-90d2-6435dc8cf655-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-flj4q\" (UID: \"d76cc067-c62a-40e3-90d2-6435dc8cf655\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-flj4q" Nov 03 02:53:52.643273 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:52.643203 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d76cc067-c62a-40e3-90d2-6435dc8cf655-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-flj4q\" (UID: \"d76cc067-c62a-40e3-90d2-6435dc8cf655\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-flj4q" Nov 03 02:53:52.643525 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:52.643426 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d76cc067-c62a-40e3-90d2-6435dc8cf655-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-flj4q\" (UID: \"d76cc067-c62a-40e3-90d2-6435dc8cf655\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-flj4q" Nov 03 02:53:52.643705 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:52.643685 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d76cc067-c62a-40e3-90d2-6435dc8cf655-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-flj4q\" (UID: \"d76cc067-c62a-40e3-90d2-6435dc8cf655\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-flj4q" Nov 03 02:53:52.643795 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:52.643777 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d76cc067-c62a-40e3-90d2-6435dc8cf655-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-flj4q\" (UID: \"d76cc067-c62a-40e3-90d2-6435dc8cf655\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-flj4q" Nov 03 02:53:52.645418 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:52.645393 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d76cc067-c62a-40e3-90d2-6435dc8cf655-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-flj4q\" (UID: \"d76cc067-c62a-40e3-90d2-6435dc8cf655\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-flj4q" Nov 03 02:53:52.645796 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:52.645777 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d76cc067-c62a-40e3-90d2-6435dc8cf655-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-flj4q\" (UID: \"d76cc067-c62a-40e3-90d2-6435dc8cf655\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-flj4q" Nov 03 02:53:52.696629 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:52.696561 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d76cc067-c62a-40e3-90d2-6435dc8cf655-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-flj4q\" (UID: \"d76cc067-c62a-40e3-90d2-6435dc8cf655\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-flj4q" Nov 03 02:53:52.696988 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:52.696963 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62qbl\" (UniqueName: \"kubernetes.io/projected/d76cc067-c62a-40e3-90d2-6435dc8cf655-kube-api-access-62qbl\") pod \"maas-default-gateway-openshift-default-5bc884dd87-flj4q\" (UID: \"d76cc067-c62a-40e3-90d2-6435dc8cf655\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-flj4q" Nov 03 02:53:52.744293 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:53:52.744267 5605 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-50a3cab9906d: secret "agc-cred-50a3cab9906d" not found Nov 03 02:53:52.744293 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:53:52.744292 5605 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-68686f579-ntp6g: secret "agc-cred-50a3cab9906d" not found Nov 03 02:53:52.744470 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:53:52.744332 5605 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d89cb81e-1adf-4e6c-82ea-dde66712aab7-provider-credentials podName:d89cb81e-1adf-4e6c-82ea-dde66712aab7 nodeName:}" failed. No retries permitted until 2026-11-03 02:54:56.744320168 +0000 UTC m=+3774.003518299 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d89cb81e-1adf-4e6c-82ea-dde66712aab7-provider-credentials") pod "payload-processing-external-model-68686f579-ntp6g" (UID: "d89cb81e-1adf-4e6c-82ea-dde66712aab7") : secret "agc-cred-50a3cab9906d" not found Nov 03 02:53:52.746850 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:52.746830 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-flj4q" Nov 03 02:53:53.195032 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:53.194989 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-flj4q"] Nov 03 02:53:53.200522 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:53:53.200492 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd76cc067_c62a_40e3_90d2_6435dc8cf655.slice/crio-4dbfa3d5a846c421d43c3dcb846d65ad91be41e36ec19fa05c6194074199e6b7 WatchSource:0}: Error finding container 4dbfa3d5a846c421d43c3dcb846d65ad91be41e36ec19fa05c6194074199e6b7: Status 404 returned error can't find the container with id 4dbfa3d5a846c421d43c3dcb846d65ad91be41e36ec19fa05c6194074199e6b7 Nov 03 02:53:53.203194 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:53.203154 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:53:53.203302 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:53.203240 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:53:53.203302 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:53.203288 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:53:53.848287 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:53.848248 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-flj4q" event={"ID":"d76cc067-c62a-40e3-90d2-6435dc8cf655","Type":"ContainerStarted","Data":"cb3e796b406e3abaeef2192c6926a294dec1b9b8d73476089aa9b133a1dd7a24"} Nov 03 02:53:53.848758 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:53.848312 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-flj4q" event={"ID":"d76cc067-c62a-40e3-90d2-6435dc8cf655","Type":"ContainerStarted","Data":"4dbfa3d5a846c421d43c3dcb846d65ad91be41e36ec19fa05c6194074199e6b7"} Nov 03 02:53:53.914307 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:53:53.914274 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:53:53.989193 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:53.989123 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-flj4q" podStartSLOduration=1.9891009099999999 podStartE2EDuration="1.98910091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:53:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:53:53.977952517 +0000 UTC m=+3711.237150671" watchObservedRunningTime="2026-11-03 02:53:53.98910091 +0000 UTC m=+3711.248299109" Nov 03 02:53:54.747978 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:54.747939 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-flj4q" Nov 03 02:53:54.749319 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:54.749290 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-flj4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Nov 03 02:53:54.749484 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:54.749333 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-flj4q" podUID="d76cc067-c62a-40e3-90d2-6435dc8cf655" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Nov 03 02:53:55.748070 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:55.748041 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-flj4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Nov 03 02:53:55.748483 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:55.748089 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-flj4q" podUID="d76cc067-c62a-40e3-90d2-6435dc8cf655" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Nov 03 02:53:56.747770 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:56.747726 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-flj4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Nov 03 02:53:56.747959 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:56.747801 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-flj4q" podUID="d76cc067-c62a-40e3-90d2-6435dc8cf655" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Nov 03 02:53:57.747613 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:57.747579 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-flj4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Nov 03 02:53:57.748001 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:57.747629 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-flj4q" podUID="d76cc067-c62a-40e3-90d2-6435dc8cf655" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Nov 03 02:53:58.747380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:58.747317 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-flj4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Nov 03 02:53:58.747594 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:58.747389 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-flj4q" podUID="d76cc067-c62a-40e3-90d2-6435dc8cf655" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Nov 03 02:53:59.747548 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:59.747516 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-flj4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Nov 03 02:53:59.748103 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:59.747562 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-flj4q" podUID="d76cc067-c62a-40e3-90d2-6435dc8cf655" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Nov 03 02:53:59.911940 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:59.911908 5605 generic.go:356] "Generic (PLEG): container finished" podID="264e26fa-8062-4f70-86c8-3ba19e1c4384" containerID="f48790376c3b6049b37350d92e3525239e699813ace92b7850f727d9e2a6259d" exitCode=0 Nov 03 02:53:59.912110 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:59.911965 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4" event={"ID":"264e26fa-8062-4f70-86c8-3ba19e1c4384","Type":"ContainerDied","Data":"f48790376c3b6049b37350d92e3525239e699813ace92b7850f727d9e2a6259d"} Nov 03 02:53:59.912380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:59.912334 5605 scope.go:122] "RemoveContainer" containerID="f48790376c3b6049b37350d92e3525239e699813ace92b7850f727d9e2a6259d" Nov 03 02:53:59.914017 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:59.913986 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:53:59.914115 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:59.914074 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:53:59.914170 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:53:59.914122 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:54:00.747714 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:00.747670 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-flj4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Nov 03 02:54:00.748093 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:00.747743 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-flj4q" podUID="d76cc067-c62a-40e3-90d2-6435dc8cf655" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Nov 03 02:54:00.844463 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:00.844423 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-54bbc97448-ntmsd" podUID="818446a4-a6e6-413c-b698-7cdcb207922c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:54:00.921234 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:00.921199 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4" event={"ID":"264e26fa-8062-4f70-86c8-3ba19e1c4384","Type":"ContainerStarted","Data":"f672c89f9062847b547331ea0de276aed365db2c44de1ac896507081b75e459f"} Nov 03 02:54:01.047877 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:01.047791 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4" Nov 03 02:54:01.049141 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:01.049116 5605 patch_prober.go:29] interesting pod/e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Nov 03 02:54:01.049258 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:01.049167 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4" podUID="264e26fa-8062-4f70-86c8-3ba19e1c4384" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Nov 03 02:54:02.048349 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:02.048311 5605 patch_prober.go:29] interesting pod/e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Nov 03 02:54:02.048880 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:02.048410 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4" podUID="264e26fa-8062-4f70-86c8-3ba19e1c4384" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Nov 03 02:54:02.122398 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:02.122339 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-flj4q" Nov 03 02:54:02.122897 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:02.122859 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-flj4q" Nov 03 02:54:02.123687 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:02.123667 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-flj4q" Nov 03 02:54:02.207887 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:02.207856 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4" Nov 03 02:54:03.047751 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:03.047716 5605 patch_prober.go:29] interesting pod/e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Nov 03 02:54:03.047930 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:03.047782 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4" podUID="264e26fa-8062-4f70-86c8-3ba19e1c4384" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Nov 03 02:54:04.048310 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:04.048281 5605 patch_prober.go:29] interesting pod/e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Nov 03 02:54:04.048718 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:04.048329 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4" podUID="264e26fa-8062-4f70-86c8-3ba19e1c4384" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Nov 03 02:54:05.048310 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:05.048274 5605 patch_prober.go:29] interesting pod/e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Nov 03 02:54:05.048503 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:05.048324 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4" podUID="264e26fa-8062-4f70-86c8-3ba19e1c4384" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Nov 03 02:54:06.048656 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:06.048625 5605 patch_prober.go:29] interesting pod/e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Nov 03 02:54:06.049040 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:06.048685 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4" podUID="264e26fa-8062-4f70-86c8-3ba19e1c4384" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Nov 03 02:54:07.052166 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:07.052138 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4" Nov 03 02:54:07.053201 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:07.053181 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4" Nov 03 02:54:10.844454 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:10.844419 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-54bbc97448-ntmsd" podUID="818446a4-a6e6-413c-b698-7cdcb207922c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:54:20.845256 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:20.845213 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-54bbc97448-ntmsd" podUID="818446a4-a6e6-413c-b698-7cdcb207922c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:54:21.326268 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:21.326237 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:54:24.145613 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:54:24.145572 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:54:24.915384 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:24.910440 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a0e72e/simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq" podUID="1901f57b-279b-4cbf-bf0c-054abc5786d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:54:24.915384 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:24.911425 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-521f1f/premium-w2-521f1f-kserve-7578fb7957-89wfb" podUID="3d988573-d8b1-43ba-a423-51a63ca12337" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:54:24.941279 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:24.940618 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a0e72e/distinct-2-w3-a0e72e-kserve-59f58c59bf-jz5p8" podUID="70bd98e0-d8ac-47c2-901e-83f301d59a57" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:54:24.944496 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:24.944460 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5" podUID="fffc3fbb-097a-4a5d-82c8-b1e221d0e710" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 02:54:24.956120 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:24.955437 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a0e72e/premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv" podUID="a9b9af44-ab6d-4caf-89af-c986ea25b4c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 02:54:25.181435 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:25.179172 5605 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-521f1f/premium-w2-521f1f-kserve-7578fb7957-89wfb" podUID="3d988573-d8b1-43ba-a423-51a63ca12337" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:54:25.320447 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:25.319453 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a0e72e/distinct-w3-a0e72e-kserve-fb499cc4-2dfxc" podUID="308bca3b-9bd5-4724-9b95-acdbc978aa10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 02:54:25.323045 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:25.322313 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr" podUID="8282371b-c42d-43ab-a5c9-e07f2464dcdc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:54:25.362569 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:25.362467 5605 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a0e72e/simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq" podUID="1901f57b-279b-4cbf-bf0c-054abc5786d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:54:27.778680 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:27.778647 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52f9d921-pgxg8"] Nov 03 02:54:27.782869 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:27.782849 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f9d921-pgxg8" Nov 03 02:54:27.878937 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:27.878904 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9brl\" (UniqueName: \"kubernetes.io/projected/03610b3e-5d7c-4938-88ac-dcec852ce7a1-kube-api-access-x9brl\") pod \"maas-api-revoke-sub-52f9d921-pgxg8\" (UID: \"03610b3e-5d7c-4938-88ac-dcec852ce7a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f9d921-pgxg8" Nov 03 02:54:27.879109 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:27.878978 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:54:27.879109 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:27.878998 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03610b3e-5d7c-4938-88ac-dcec852ce7a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52f9d921-pgxg8\" (UID: \"03610b3e-5d7c-4938-88ac-dcec852ce7a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f9d921-pgxg8" Nov 03 02:54:27.879109 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:27.879027 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:54:27.879109 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:27.879075 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:54:27.879109 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:27.879078 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:54:27.879313 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:27.879113 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03610b3e-5d7c-4938-88ac-dcec852ce7a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52f9d921-pgxg8\" (UID: \"03610b3e-5d7c-4938-88ac-dcec852ce7a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f9d921-pgxg8" Nov 03 02:54:27.881610 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:27.881590 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52f9d921-pgxg8"] Nov 03 02:54:27.980089 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:27.980061 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03610b3e-5d7c-4938-88ac-dcec852ce7a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52f9d921-pgxg8\" (UID: \"03610b3e-5d7c-4938-88ac-dcec852ce7a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f9d921-pgxg8" Nov 03 02:54:27.980216 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:27.980199 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9brl\" (UniqueName: \"kubernetes.io/projected/03610b3e-5d7c-4938-88ac-dcec852ce7a1-kube-api-access-x9brl\") pod \"maas-api-revoke-sub-52f9d921-pgxg8\" (UID: \"03610b3e-5d7c-4938-88ac-dcec852ce7a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f9d921-pgxg8" Nov 03 02:54:27.980296 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:27.980279 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03610b3e-5d7c-4938-88ac-dcec852ce7a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52f9d921-pgxg8\" (UID: \"03610b3e-5d7c-4938-88ac-dcec852ce7a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f9d921-pgxg8" Nov 03 02:54:27.980871 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:27.980848 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03610b3e-5d7c-4938-88ac-dcec852ce7a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52f9d921-pgxg8\" (UID: \"03610b3e-5d7c-4938-88ac-dcec852ce7a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f9d921-pgxg8" Nov 03 02:54:27.982403 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:27.982378 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03610b3e-5d7c-4938-88ac-dcec852ce7a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52f9d921-pgxg8\" (UID: \"03610b3e-5d7c-4938-88ac-dcec852ce7a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f9d921-pgxg8" Nov 03 02:54:28.033740 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:28.033694 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9brl\" (UniqueName: \"kubernetes.io/projected/03610b3e-5d7c-4938-88ac-dcec852ce7a1-kube-api-access-x9brl\") pod \"maas-api-revoke-sub-52f9d921-pgxg8\" (UID: \"03610b3e-5d7c-4938-88ac-dcec852ce7a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f9d921-pgxg8" Nov 03 02:54:28.095779 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:28.095758 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f9d921-pgxg8" Nov 03 02:54:28.468059 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:28.468027 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52f9d921-pgxg8"] Nov 03 02:54:28.473398 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:54:28.473350 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03610b3e_5d7c_4938_88ac_dcec852ce7a1.slice/crio-9370b387cd1886634bd048e5650a580d75b87ba991744ddbb5e3b649252bd97a WatchSource:0}: Error finding container 9370b387cd1886634bd048e5650a580d75b87ba991744ddbb5e3b649252bd97a: Status 404 returned error can't find the container with id 9370b387cd1886634bd048e5650a580d75b87ba991744ddbb5e3b649252bd97a Nov 03 02:54:29.136257 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:29.136221 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f9d921-pgxg8" event={"ID":"03610b3e-5d7c-4938-88ac-dcec852ce7a1","Type":"ContainerStarted","Data":"58cc5283bd85eb6fe7ab0cf66903ad7e288cdd38a1b997544e9dd4e0edb38593"} Nov 03 02:54:29.136652 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:29.136280 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f9d921-pgxg8" event={"ID":"03610b3e-5d7c-4938-88ac-dcec852ce7a1","Type":"ContainerStarted","Data":"9370b387cd1886634bd048e5650a580d75b87ba991744ddbb5e3b649252bd97a"} Nov 03 02:54:29.245271 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:29.245224 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f9d921-pgxg8" podStartSLOduration=2.24521118 podStartE2EDuration="2.24521118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:54:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:54:29.244086591 +0000 UTC m=+3746.503284756" watchObservedRunningTime="2026-11-03 02:54:29.24521118 +0000 UTC m=+3746.504409361" Nov 03 02:54:30.143838 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:30.143800 5605 generic.go:356] "Generic (PLEG): container finished" podID="03610b3e-5d7c-4938-88ac-dcec852ce7a1" containerID="58cc5283bd85eb6fe7ab0cf66903ad7e288cdd38a1b997544e9dd4e0edb38593" exitCode=0 Nov 03 02:54:30.144328 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:30.143880 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f9d921-pgxg8" event={"ID":"03610b3e-5d7c-4938-88ac-dcec852ce7a1","Type":"ContainerDied","Data":"58cc5283bd85eb6fe7ab0cf66903ad7e288cdd38a1b997544e9dd4e0edb38593"} Nov 03 02:54:30.844751 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:30.844718 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-54bbc97448-ntmsd" podUID="818446a4-a6e6-413c-b698-7cdcb207922c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:54:31.299597 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:31.299568 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f9d921-pgxg8" Nov 03 02:54:31.417961 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:31.417897 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03610b3e-5d7c-4938-88ac-dcec852ce7a1-kube-api-access-x9brl\" (UniqueName: \"kubernetes.io/projected/03610b3e-5d7c-4938-88ac-dcec852ce7a1-kube-api-access-x9brl\") pod \"03610b3e-5d7c-4938-88ac-dcec852ce7a1\" (UID: \"03610b3e-5d7c-4938-88ac-dcec852ce7a1\") " Nov 03 02:54:31.418099 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:31.418083 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03610b3e-5d7c-4938-88ac-dcec852ce7a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03610b3e-5d7c-4938-88ac-dcec852ce7a1-maas-api-openshift-ca\") pod \"03610b3e-5d7c-4938-88ac-dcec852ce7a1\" (UID: \"03610b3e-5d7c-4938-88ac-dcec852ce7a1\") " Nov 03 02:54:31.418191 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:31.418175 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03610b3e-5d7c-4938-88ac-dcec852ce7a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03610b3e-5d7c-4938-88ac-dcec852ce7a1-maas-api-xks-ca\") pod \"03610b3e-5d7c-4938-88ac-dcec852ce7a1\" (UID: \"03610b3e-5d7c-4938-88ac-dcec852ce7a1\") " Nov 03 02:54:31.418411 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:31.418388 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03610b3e-5d7c-4938-88ac-dcec852ce7a1-maas-api-openshift-ca" pod "03610b3e-5d7c-4938-88ac-dcec852ce7a1" (UID: "03610b3e-5d7c-4938-88ac-dcec852ce7a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:54:31.418672 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:31.418650 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03610b3e-5d7c-4938-88ac-dcec852ce7a1-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:54:31.419926 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:31.419903 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03610b3e-5d7c-4938-88ac-dcec852ce7a1-kube-api-access-x9brl" pod "03610b3e-5d7c-4938-88ac-dcec852ce7a1" (UID: "03610b3e-5d7c-4938-88ac-dcec852ce7a1"). InnerVolumeSpecName "kube-api-access-x9brl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:54:31.420164 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:31.420142 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03610b3e-5d7c-4938-88ac-dcec852ce7a1-maas-api-xks-ca" pod "03610b3e-5d7c-4938-88ac-dcec852ce7a1" (UID: "03610b3e-5d7c-4938-88ac-dcec852ce7a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:54:31.519713 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:31.519685 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9brl\" (UniqueName: \"kubernetes.io/projected/03610b3e-5d7c-4938-88ac-dcec852ce7a1-kube-api-access-x9brl\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:54:31.519713 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:31.519711 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03610b3e-5d7c-4938-88ac-dcec852ce7a1-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:54:32.158129 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:32.158093 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f9d921-pgxg8" event={"ID":"03610b3e-5d7c-4938-88ac-dcec852ce7a1","Type":"ContainerDied","Data":"9370b387cd1886634bd048e5650a580d75b87ba991744ddbb5e3b649252bd97a"} Nov 03 02:54:32.158129 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:32.158125 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f9d921-pgxg8" Nov 03 02:54:32.158344 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:32.158142 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9370b387cd1886634bd048e5650a580d75b87ba991744ddbb5e3b649252bd97a" Nov 03 02:54:40.844439 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:40.844403 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-54bbc97448-ntmsd" podUID="818446a4-a6e6-413c-b698-7cdcb207922c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:54:41.477548 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:41.477513 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-54bbc97448-ntmsd"] Nov 03 02:54:41.478645 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:41.478617 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-54bbc97448-ntmsd" podUID="818446a4-a6e6-413c-b698-7cdcb207922c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:54:41.509563 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:41.509539 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-747c9f9449-86gct"] Nov 03 02:54:41.510024 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:41.510012 5605 cpu_manager.go:414] "containerMap: removing container" podUID="03610b3e-5d7c-4938-88ac-dcec852ce7a1" containerName="revoke-keys" Nov 03 02:54:41.510070 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:41.510025 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03610b3e-5d7c-4938-88ac-dcec852ce7a1" containerName="revoke-keys" Nov 03 02:54:41.510139 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:41.510130 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="03610b3e-5d7c-4938-88ac-dcec852ce7a1" containerName="revoke-keys" Nov 03 02:54:41.513574 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:41.513557 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-747c9f9449-86gct" Nov 03 02:54:41.524016 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:41.523992 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/41245c80-d885-4a8c-9ce5-24e59ed83704-routing-overlay-volume\") pod \"payload-processing-external-model-747c9f9449-86gct\" (UID: \"41245c80-d885-4a8c-9ce5-24e59ed83704\") " pod="models-as-a-service/payload-processing-external-model-747c9f9449-86gct" Nov 03 02:54:41.524116 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:41.524046 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4x5f\" (UniqueName: \"kubernetes.io/projected/41245c80-d885-4a8c-9ce5-24e59ed83704-kube-api-access-w4x5f\") pod \"payload-processing-external-model-747c9f9449-86gct\" (UID: \"41245c80-d885-4a8c-9ce5-24e59ed83704\") " pod="models-as-a-service/payload-processing-external-model-747c9f9449-86gct" Nov 03 02:54:41.524176 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:41.524151 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/41245c80-d885-4a8c-9ce5-24e59ed83704-provider-credentials\") pod \"payload-processing-external-model-747c9f9449-86gct\" (UID: \"41245c80-d885-4a8c-9ce5-24e59ed83704\") " pod="models-as-a-service/payload-processing-external-model-747c9f9449-86gct" Nov 03 02:54:41.524258 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:41.524239 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41245c80-d885-4a8c-9ce5-24e59ed83704-plugins-config-volume\") pod \"payload-processing-external-model-747c9f9449-86gct\" (UID: \"41245c80-d885-4a8c-9ce5-24e59ed83704\") " pod="models-as-a-service/payload-processing-external-model-747c9f9449-86gct" Nov 03 02:54:41.582353 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:41.582329 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-747c9f9449-86gct"] Nov 03 02:54:41.625853 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:41.625821 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4x5f\" (UniqueName: \"kubernetes.io/projected/41245c80-d885-4a8c-9ce5-24e59ed83704-kube-api-access-w4x5f\") pod \"payload-processing-external-model-747c9f9449-86gct\" (UID: \"41245c80-d885-4a8c-9ce5-24e59ed83704\") " pod="models-as-a-service/payload-processing-external-model-747c9f9449-86gct" Nov 03 02:54:41.625982 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:41.625875 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/41245c80-d885-4a8c-9ce5-24e59ed83704-provider-credentials\") pod \"payload-processing-external-model-747c9f9449-86gct\" (UID: \"41245c80-d885-4a8c-9ce5-24e59ed83704\") " pod="models-as-a-service/payload-processing-external-model-747c9f9449-86gct" Nov 03 02:54:41.625982 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:41.625913 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41245c80-d885-4a8c-9ce5-24e59ed83704-plugins-config-volume\") pod \"payload-processing-external-model-747c9f9449-86gct\" (UID: \"41245c80-d885-4a8c-9ce5-24e59ed83704\") " pod="models-as-a-service/payload-processing-external-model-747c9f9449-86gct" Nov 03 02:54:41.626056 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:41.626014 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/41245c80-d885-4a8c-9ce5-24e59ed83704-routing-overlay-volume\") pod \"payload-processing-external-model-747c9f9449-86gct\" (UID: \"41245c80-d885-4a8c-9ce5-24e59ed83704\") " pod="models-as-a-service/payload-processing-external-model-747c9f9449-86gct" Nov 03 02:54:41.626818 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:41.626792 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/41245c80-d885-4a8c-9ce5-24e59ed83704-routing-overlay-volume\") pod \"payload-processing-external-model-747c9f9449-86gct\" (UID: \"41245c80-d885-4a8c-9ce5-24e59ed83704\") " pod="models-as-a-service/payload-processing-external-model-747c9f9449-86gct" Nov 03 02:54:41.626890 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:41.626876 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41245c80-d885-4a8c-9ce5-24e59ed83704-plugins-config-volume\") pod \"payload-processing-external-model-747c9f9449-86gct\" (UID: \"41245c80-d885-4a8c-9ce5-24e59ed83704\") " pod="models-as-a-service/payload-processing-external-model-747c9f9449-86gct" Nov 03 02:54:41.628197 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:41.628171 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/41245c80-d885-4a8c-9ce5-24e59ed83704-provider-credentials\") pod \"payload-processing-external-model-747c9f9449-86gct\" (UID: \"41245c80-d885-4a8c-9ce5-24e59ed83704\") " pod="models-as-a-service/payload-processing-external-model-747c9f9449-86gct" Nov 03 02:54:41.678886 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:41.678856 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4x5f\" (UniqueName: \"kubernetes.io/projected/41245c80-d885-4a8c-9ce5-24e59ed83704-kube-api-access-w4x5f\") pod \"payload-processing-external-model-747c9f9449-86gct\" (UID: \"41245c80-d885-4a8c-9ce5-24e59ed83704\") " pod="models-as-a-service/payload-processing-external-model-747c9f9449-86gct" Nov 03 02:54:41.827145 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:41.827119 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-747c9f9449-86gct" Nov 03 02:54:42.195489 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:42.195411 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-747c9f9449-86gct"] Nov 03 02:54:42.201047 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:54:42.201017 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41245c80_d885_4a8c_9ce5_24e59ed83704.slice/crio-e59137bf67508d80899d76a045786978fcd44b9ac5116873ba95e51b6c9906cb WatchSource:0}: Error finding container e59137bf67508d80899d76a045786978fcd44b9ac5116873ba95e51b6c9906cb: Status 404 returned error can't find the container with id e59137bf67508d80899d76a045786978fcd44b9ac5116873ba95e51b6c9906cb Nov 03 02:54:42.297275 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:42.297247 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-747c9f9449-86gct" event={"ID":"41245c80-d885-4a8c-9ce5-24e59ed83704","Type":"ContainerStarted","Data":"275cf0e3db22eb84d0e99f5dffc9ebcd9654d0fd4f7b2d7c1fd4f60808b312fa"} Nov 03 02:54:42.297432 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:42.297303 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-747c9f9449-86gct" event={"ID":"41245c80-d885-4a8c-9ce5-24e59ed83704","Type":"ContainerStarted","Data":"e59137bf67508d80899d76a045786978fcd44b9ac5116873ba95e51b6c9906cb"} Nov 03 02:54:42.297513 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:42.297457 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-747c9f9449-86gct" Nov 03 02:54:42.405403 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:42.405345 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-747c9f9449-86gct" podStartSLOduration=1.405332616 podStartE2EDuration="1.405332616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:54:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:54:42.40424345 +0000 UTC m=+3759.663441602" watchObservedRunningTime="2026-11-03 02:54:42.405332616 +0000 UTC m=+3759.664530769" Nov 03 02:54:43.131148 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:43.131107 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7"] Nov 03 02:54:43.135299 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:43.135272 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7" Nov 03 02:54:43.166778 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:43.166211 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f7abd33-f094-4091-b738-d77e72cfbf05-istio-podinfo\") pod \"e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7\" (UID: \"6f7abd33-f094-4091-b738-d77e72cfbf05\") " pod="openshift-ingress/e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7" Nov 03 02:54:43.166778 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:43.166278 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f7abd33-f094-4091-b738-d77e72cfbf05-workload-socket\") pod \"e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7\" (UID: \"6f7abd33-f094-4091-b738-d77e72cfbf05\") " pod="openshift-ingress/e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7" Nov 03 02:54:43.166778 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:43.166333 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f7abd33-f094-4091-b738-d77e72cfbf05-istio-data\") pod \"e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7\" (UID: \"6f7abd33-f094-4091-b738-d77e72cfbf05\") " pod="openshift-ingress/e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7" Nov 03 02:54:43.166778 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:43.166454 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f7abd33-f094-4091-b738-d77e72cfbf05-workload-certs\") pod \"e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7\" (UID: \"6f7abd33-f094-4091-b738-d77e72cfbf05\") " pod="openshift-ingress/e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7" Nov 03 02:54:43.166778 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:43.166488 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4br27\" (UniqueName: \"kubernetes.io/projected/6f7abd33-f094-4091-b738-d77e72cfbf05-kube-api-access-4br27\") pod \"e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7\" (UID: \"6f7abd33-f094-4091-b738-d77e72cfbf05\") " pod="openshift-ingress/e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7" Nov 03 02:54:43.166778 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:43.166561 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f7abd33-f094-4091-b738-d77e72cfbf05-istio-token\") pod \"e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7\" (UID: \"6f7abd33-f094-4091-b738-d77e72cfbf05\") " pod="openshift-ingress/e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7" Nov 03 02:54:43.166778 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:43.166591 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f7abd33-f094-4091-b738-d77e72cfbf05-istio-envoy\") pod \"e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7\" (UID: \"6f7abd33-f094-4091-b738-d77e72cfbf05\") " pod="openshift-ingress/e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7" Nov 03 02:54:43.166778 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:43.166619 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f7abd33-f094-4091-b738-d77e72cfbf05-istiod-ca-cert\") pod \"e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7\" (UID: \"6f7abd33-f094-4091-b738-d77e72cfbf05\") " pod="openshift-ingress/e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7" Nov 03 02:54:43.166778 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:43.166649 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f7abd33-f094-4091-b738-d77e72cfbf05-credential-socket\") pod \"e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7\" (UID: \"6f7abd33-f094-4091-b738-d77e72cfbf05\") " pod="openshift-ingress/e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7" Nov 03 02:54:43.218391 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:43.218351 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-1c43e9-openshift-default-dockercfg-8hmmh\"" Nov 03 02:54:43.234853 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:43.234826 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7"] Nov 03 02:54:43.267953 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:43.267922 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f7abd33-f094-4091-b738-d77e72cfbf05-workload-certs\") pod \"e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7\" (UID: \"6f7abd33-f094-4091-b738-d77e72cfbf05\") " pod="openshift-ingress/e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7" Nov 03 02:54:43.268101 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:43.268034 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4br27\" (UniqueName: \"kubernetes.io/projected/6f7abd33-f094-4091-b738-d77e72cfbf05-kube-api-access-4br27\") pod \"e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7\" (UID: \"6f7abd33-f094-4091-b738-d77e72cfbf05\") " pod="openshift-ingress/e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7" Nov 03 02:54:43.268176 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:43.268132 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f7abd33-f094-4091-b738-d77e72cfbf05-istio-token\") pod \"e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7\" (UID: \"6f7abd33-f094-4091-b738-d77e72cfbf05\") " pod="openshift-ingress/e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7" Nov 03 02:54:43.268248 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:43.268170 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f7abd33-f094-4091-b738-d77e72cfbf05-istio-envoy\") pod \"e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7\" (UID: \"6f7abd33-f094-4091-b738-d77e72cfbf05\") " pod="openshift-ingress/e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7" Nov 03 02:54:43.268248 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:43.268201 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f7abd33-f094-4091-b738-d77e72cfbf05-istiod-ca-cert\") pod \"e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7\" (UID: \"6f7abd33-f094-4091-b738-d77e72cfbf05\") " pod="openshift-ingress/e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7" Nov 03 02:54:43.268248 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:43.268236 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f7abd33-f094-4091-b738-d77e72cfbf05-credential-socket\") pod \"e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7\" (UID: \"6f7abd33-f094-4091-b738-d77e72cfbf05\") " pod="openshift-ingress/e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7" Nov 03 02:54:43.268438 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:43.268310 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f7abd33-f094-4091-b738-d77e72cfbf05-istio-podinfo\") pod \"e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7\" (UID: \"6f7abd33-f094-4091-b738-d77e72cfbf05\") " pod="openshift-ingress/e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7" Nov 03 02:54:43.268438 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:43.268320 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f7abd33-f094-4091-b738-d77e72cfbf05-workload-certs\") pod \"e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7\" (UID: \"6f7abd33-f094-4091-b738-d77e72cfbf05\") " pod="openshift-ingress/e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7" Nov 03 02:54:43.268438 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:43.268422 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f7abd33-f094-4091-b738-d77e72cfbf05-workload-socket\") pod \"e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7\" (UID: \"6f7abd33-f094-4091-b738-d77e72cfbf05\") " pod="openshift-ingress/e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7" Nov 03 02:54:43.268631 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:43.268495 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f7abd33-f094-4091-b738-d77e72cfbf05-istio-data\") pod \"e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7\" (UID: \"6f7abd33-f094-4091-b738-d77e72cfbf05\") " pod="openshift-ingress/e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7" Nov 03 02:54:43.268783 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:43.268755 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f7abd33-f094-4091-b738-d77e72cfbf05-workload-socket\") pod \"e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7\" (UID: \"6f7abd33-f094-4091-b738-d77e72cfbf05\") " pod="openshift-ingress/e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7" Nov 03 02:54:43.268977 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:43.268953 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f7abd33-f094-4091-b738-d77e72cfbf05-istio-data\") pod \"e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7\" (UID: \"6f7abd33-f094-4091-b738-d77e72cfbf05\") " pod="openshift-ingress/e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7" Nov 03 02:54:43.269155 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:43.269008 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f7abd33-f094-4091-b738-d77e72cfbf05-credential-socket\") pod \"e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7\" (UID: \"6f7abd33-f094-4091-b738-d77e72cfbf05\") " pod="openshift-ingress/e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7" Nov 03 02:54:43.269977 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:43.269900 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f7abd33-f094-4091-b738-d77e72cfbf05-istiod-ca-cert\") pod \"e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7\" (UID: \"6f7abd33-f094-4091-b738-d77e72cfbf05\") " pod="openshift-ingress/e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7" Nov 03 02:54:43.272246 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:43.271470 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f7abd33-f094-4091-b738-d77e72cfbf05-istio-podinfo\") pod \"e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7\" (UID: \"6f7abd33-f094-4091-b738-d77e72cfbf05\") " pod="openshift-ingress/e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7" Nov 03 02:54:43.272246 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:43.271681 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f7abd33-f094-4091-b738-d77e72cfbf05-istio-envoy\") pod \"e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7\" (UID: \"6f7abd33-f094-4091-b738-d77e72cfbf05\") " pod="openshift-ingress/e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7" Nov 03 02:54:43.353382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:43.343163 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f7abd33-f094-4091-b738-d77e72cfbf05-istio-token\") pod \"e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7\" (UID: \"6f7abd33-f094-4091-b738-d77e72cfbf05\") " pod="openshift-ingress/e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7" Nov 03 02:54:43.353382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:43.345924 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4br27\" (UniqueName: \"kubernetes.io/projected/6f7abd33-f094-4091-b738-d77e72cfbf05-kube-api-access-4br27\") pod \"e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7\" (UID: \"6f7abd33-f094-4091-b738-d77e72cfbf05\") " pod="openshift-ingress/e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7" Nov 03 02:54:43.448962 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:43.448865 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7" Nov 03 02:54:43.875639 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:43.875606 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7"] Nov 03 02:54:43.881289 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:54:43.881264 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f7abd33_f094_4091_b738_d77e72cfbf05.slice/crio-dae15775eb9d2ee8d71cc6c0d3e19335124938ed09718ef7b9aff3c131db83fc WatchSource:0}: Error finding container dae15775eb9d2ee8d71cc6c0d3e19335124938ed09718ef7b9aff3c131db83fc: Status 404 returned error can't find the container with id dae15775eb9d2ee8d71cc6c0d3e19335124938ed09718ef7b9aff3c131db83fc Nov 03 02:54:43.884179 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:43.884142 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:54:43.884284 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:43.884207 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:54:43.884284 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:43.884235 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:54:44.369444 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:44.369409 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7" event={"ID":"6f7abd33-f094-4091-b738-d77e72cfbf05","Type":"ContainerStarted","Data":"ce55801f30094425de06e8c392a6eb1aecd0adbe74b9ee022bbf05a76d7b5f3b"} Nov 03 02:54:44.369821 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:44.369464 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7" event={"ID":"6f7abd33-f094-4091-b738-d77e72cfbf05","Type":"ContainerStarted","Data":"dae15775eb9d2ee8d71cc6c0d3e19335124938ed09718ef7b9aff3c131db83fc"} Nov 03 02:54:44.449659 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:44.449633 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7" Nov 03 02:54:44.455224 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:44.455201 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7" Nov 03 02:54:44.483414 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:44.483372 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7" podStartSLOduration=1.483345701 podStartE2EDuration="1.483345701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:54:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:54:44.481015725 +0000 UTC m=+3761.740213879" watchObservedRunningTime="2026-11-03 02:54:44.483345701 +0000 UTC m=+3761.742543854" Nov 03 02:54:45.375494 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:45.375461 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7" Nov 03 02:54:45.376504 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:45.376479 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7" Nov 03 02:54:46.478596 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:46.478552 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-54bbc97448-ntmsd" podUID="818446a4-a6e6-413c-b698-7cdcb207922c" containerName="payload-processing" containerID="cri-o://1d64bf1d6c03faffea03cfd11506178654281ded7665ee0c268a9d41e1250826" gracePeriod=25 Nov 03 02:54:46.737438 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:46.737353 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-54bbc97448-ntmsd" Nov 03 02:54:46.813529 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:46.813502 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/818446a4-a6e6-413c-b698-7cdcb207922c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/818446a4-a6e6-413c-b698-7cdcb207922c-routing-overlay-volume\") pod \"818446a4-a6e6-413c-b698-7cdcb207922c\" (UID: \"818446a4-a6e6-413c-b698-7cdcb207922c\") " Nov 03 02:54:46.813683 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:46.813641 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/818446a4-a6e6-413c-b698-7cdcb207922c-kube-api-access-lscnh\" (UniqueName: \"kubernetes.io/projected/818446a4-a6e6-413c-b698-7cdcb207922c-kube-api-access-lscnh\") pod \"818446a4-a6e6-413c-b698-7cdcb207922c\" (UID: \"818446a4-a6e6-413c-b698-7cdcb207922c\") " Nov 03 02:54:46.813756 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:46.813741 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/818446a4-a6e6-413c-b698-7cdcb207922c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/818446a4-a6e6-413c-b698-7cdcb207922c-provider-credentials\") pod \"818446a4-a6e6-413c-b698-7cdcb207922c\" (UID: \"818446a4-a6e6-413c-b698-7cdcb207922c\") " Nov 03 02:54:46.813819 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:46.813804 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/818446a4-a6e6-413c-b698-7cdcb207922c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/818446a4-a6e6-413c-b698-7cdcb207922c-plugins-config-volume\") pod \"818446a4-a6e6-413c-b698-7cdcb207922c\" (UID: \"818446a4-a6e6-413c-b698-7cdcb207922c\") " Nov 03 02:54:46.813879 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:46.813817 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/818446a4-a6e6-413c-b698-7cdcb207922c-routing-overlay-volume" pod "818446a4-a6e6-413c-b698-7cdcb207922c" (UID: "818446a4-a6e6-413c-b698-7cdcb207922c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:54:46.814174 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:46.814151 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/818446a4-a6e6-413c-b698-7cdcb207922c-plugins-config-volume" pod "818446a4-a6e6-413c-b698-7cdcb207922c" (UID: "818446a4-a6e6-413c-b698-7cdcb207922c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:54:46.814322 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:46.814291 5605 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/818446a4-a6e6-413c-b698-7cdcb207922c-plugins-config-volume\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:54:46.814322 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:46.814309 5605 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/818446a4-a6e6-413c-b698-7cdcb207922c-routing-overlay-volume\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:54:46.815786 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:46.815765 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/818446a4-a6e6-413c-b698-7cdcb207922c-kube-api-access-lscnh" pod "818446a4-a6e6-413c-b698-7cdcb207922c" (UID: "818446a4-a6e6-413c-b698-7cdcb207922c"). InnerVolumeSpecName "kube-api-access-lscnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:54:46.815868 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:46.815786 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/818446a4-a6e6-413c-b698-7cdcb207922c-provider-credentials" pod "818446a4-a6e6-413c-b698-7cdcb207922c" (UID: "818446a4-a6e6-413c-b698-7cdcb207922c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:54:46.915599 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:46.915570 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lscnh\" (UniqueName: \"kubernetes.io/projected/818446a4-a6e6-413c-b698-7cdcb207922c-kube-api-access-lscnh\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:54:46.915599 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:46.915595 5605 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/818446a4-a6e6-413c-b698-7cdcb207922c-provider-credentials\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:54:47.388792 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:47.388755 5605 generic.go:356] "Generic (PLEG): container finished" podID="818446a4-a6e6-413c-b698-7cdcb207922c" containerID="1d64bf1d6c03faffea03cfd11506178654281ded7665ee0c268a9d41e1250826" exitCode=0 Nov 03 02:54:47.388991 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:47.388806 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-54bbc97448-ntmsd" Nov 03 02:54:47.388991 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:47.388829 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-54bbc97448-ntmsd" event={"ID":"818446a4-a6e6-413c-b698-7cdcb207922c","Type":"ContainerDied","Data":"1d64bf1d6c03faffea03cfd11506178654281ded7665ee0c268a9d41e1250826"} Nov 03 02:54:47.388991 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:47.388890 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-54bbc97448-ntmsd" event={"ID":"818446a4-a6e6-413c-b698-7cdcb207922c","Type":"ContainerDied","Data":"6ca36ce5aaaa8587477afed9bcc2962cec70a06e624a56ca44016262ff2f7217"} Nov 03 02:54:47.388991 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:47.388919 5605 scope.go:122] "RemoveContainer" containerID="1d64bf1d6c03faffea03cfd11506178654281ded7665ee0c268a9d41e1250826" Nov 03 02:54:47.398161 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:47.398144 5605 scope.go:122] "RemoveContainer" containerID="1d64bf1d6c03faffea03cfd11506178654281ded7665ee0c268a9d41e1250826" Nov 03 02:54:47.398496 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:54:47.398476 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d64bf1d6c03faffea03cfd11506178654281ded7665ee0c268a9d41e1250826\": container with ID starting with 1d64bf1d6c03faffea03cfd11506178654281ded7665ee0c268a9d41e1250826 not found: ID does not exist" containerID="1d64bf1d6c03faffea03cfd11506178654281ded7665ee0c268a9d41e1250826" Nov 03 02:54:47.398579 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:47.398506 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d64bf1d6c03faffea03cfd11506178654281ded7665ee0c268a9d41e1250826"} err="failed to get container status \"1d64bf1d6c03faffea03cfd11506178654281ded7665ee0c268a9d41e1250826\": rpc error: code = NotFound desc = could not find container \"1d64bf1d6c03faffea03cfd11506178654281ded7665ee0c268a9d41e1250826\": container with ID starting with 1d64bf1d6c03faffea03cfd11506178654281ded7665ee0c268a9d41e1250826 not found: ID does not exist" Nov 03 02:54:47.561980 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:47.561956 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-54bbc97448-ntmsd"] Nov 03 02:54:47.608542 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:47.608520 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-54bbc97448-ntmsd"] Nov 03 02:54:48.850322 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:48.850289 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="818446a4-a6e6-413c-b698-7cdcb207922c" path="/var/lib/kubelet/pods/818446a4-a6e6-413c-b698-7cdcb207922c/volumes" Nov 03 02:54:51.058906 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:51.058867 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1c43e9-58dfd86775-b7nrz"] Nov 03 02:54:51.059456 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:51.059441 5605 cpu_manager.go:414] "containerMap: removing container" podUID="818446a4-a6e6-413c-b698-7cdcb207922c" containerName="payload-processing" Nov 03 02:54:51.059507 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:51.059456 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="818446a4-a6e6-413c-b698-7cdcb207922c" containerName="payload-processing" Nov 03 02:54:51.059568 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:51.059559 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="818446a4-a6e6-413c-b698-7cdcb207922c" containerName="payload-processing" Nov 03 02:54:51.063655 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:51.063636 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c43e9-58dfd86775-b7nrz" Nov 03 02:54:51.116228 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:51.116202 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1c43e9-58dfd86775-b7nrz"] Nov 03 02:54:51.160657 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:51.160619 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-1c43e9-dockercfg-ktq8r\"" Nov 03 02:54:51.160966 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:51.160950 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-1c43e9\"" Nov 03 02:54:51.187380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:51.186370 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8sgl\" (UniqueName: \"kubernetes.io/projected/ad6cd805-ef8c-4c3a-a63d-52a044bfe9ce-kube-api-access-k8sgl\") pod \"payload-pre-processing-e2e-inf-a-1c43e9-58dfd86775-b7nrz\" (UID: \"ad6cd805-ef8c-4c3a-a63d-52a044bfe9ce\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c43e9-58dfd86775-b7nrz" Nov 03 02:54:51.187380 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:51.186476 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad6cd805-ef8c-4c3a-a63d-52a044bfe9ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1c43e9-58dfd86775-b7nrz\" (UID: \"ad6cd805-ef8c-4c3a-a63d-52a044bfe9ce\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c43e9-58dfd86775-b7nrz" Nov 03 02:54:51.211322 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:51.211262 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1c43e9-76f6b754d7-mr225"] Nov 03 02:54:51.223710 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:51.223687 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1c43e9-76f6b754d7-mr225" Nov 03 02:54:51.272500 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:51.272426 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1c43e9-76f6b754d7-mr225"] Nov 03 02:54:51.288206 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:51.288177 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e82f0187-0392-499b-ac69-0e7e03eebbf7-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1c43e9-76f6b754d7-mr225\" (UID: \"e82f0187-0392-499b-ac69-0e7e03eebbf7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1c43e9-76f6b754d7-mr225" Nov 03 02:54:51.288405 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:51.288269 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8sgl\" (UniqueName: \"kubernetes.io/projected/ad6cd805-ef8c-4c3a-a63d-52a044bfe9ce-kube-api-access-k8sgl\") pod \"payload-pre-processing-e2e-inf-a-1c43e9-58dfd86775-b7nrz\" (UID: \"ad6cd805-ef8c-4c3a-a63d-52a044bfe9ce\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c43e9-58dfd86775-b7nrz" Nov 03 02:54:51.288494 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:51.288423 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td284\" (UniqueName: \"kubernetes.io/projected/e82f0187-0392-499b-ac69-0e7e03eebbf7-kube-api-access-td284\") pod \"payload-processing-e2e-inf-a-1c43e9-76f6b754d7-mr225\" (UID: \"e82f0187-0392-499b-ac69-0e7e03eebbf7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1c43e9-76f6b754d7-mr225" Nov 03 02:54:51.288494 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:51.288461 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad6cd805-ef8c-4c3a-a63d-52a044bfe9ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1c43e9-58dfd86775-b7nrz\" (UID: \"ad6cd805-ef8c-4c3a-a63d-52a044bfe9ce\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c43e9-58dfd86775-b7nrz" Nov 03 02:54:51.289101 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:51.289081 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad6cd805-ef8c-4c3a-a63d-52a044bfe9ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1c43e9-58dfd86775-b7nrz\" (UID: \"ad6cd805-ef8c-4c3a-a63d-52a044bfe9ce\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c43e9-58dfd86775-b7nrz" Nov 03 02:54:51.341934 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:51.341874 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8sgl\" (UniqueName: \"kubernetes.io/projected/ad6cd805-ef8c-4c3a-a63d-52a044bfe9ce-kube-api-access-k8sgl\") pod \"payload-pre-processing-e2e-inf-a-1c43e9-58dfd86775-b7nrz\" (UID: \"ad6cd805-ef8c-4c3a-a63d-52a044bfe9ce\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c43e9-58dfd86775-b7nrz" Nov 03 02:54:51.376721 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:51.376698 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c43e9-58dfd86775-b7nrz" Nov 03 02:54:51.389998 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:51.389970 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e82f0187-0392-499b-ac69-0e7e03eebbf7-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1c43e9-76f6b754d7-mr225\" (UID: \"e82f0187-0392-499b-ac69-0e7e03eebbf7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1c43e9-76f6b754d7-mr225" Nov 03 02:54:51.391006 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:51.390983 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td284\" (UniqueName: \"kubernetes.io/projected/e82f0187-0392-499b-ac69-0e7e03eebbf7-kube-api-access-td284\") pod \"payload-processing-e2e-inf-a-1c43e9-76f6b754d7-mr225\" (UID: \"e82f0187-0392-499b-ac69-0e7e03eebbf7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1c43e9-76f6b754d7-mr225" Nov 03 02:54:51.391559 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:51.390849 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e82f0187-0392-499b-ac69-0e7e03eebbf7-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1c43e9-76f6b754d7-mr225\" (UID: \"e82f0187-0392-499b-ac69-0e7e03eebbf7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1c43e9-76f6b754d7-mr225" Nov 03 02:54:51.448224 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:51.448195 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-td284\" (UniqueName: \"kubernetes.io/projected/e82f0187-0392-499b-ac69-0e7e03eebbf7-kube-api-access-td284\") pod \"payload-processing-e2e-inf-a-1c43e9-76f6b754d7-mr225\" (UID: \"e82f0187-0392-499b-ac69-0e7e03eebbf7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1c43e9-76f6b754d7-mr225" Nov 03 02:54:51.542177 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:51.542149 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1c43e9-76f6b754d7-mr225" Nov 03 02:54:51.747314 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:51.747282 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1c43e9-58dfd86775-b7nrz"] Nov 03 02:54:51.752312 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:54:51.752284 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad6cd805_ef8c_4c3a_a63d_52a044bfe9ce.slice/crio-e9210919df869cca2c1796b79083dd2ae1b214f19c218139cebe17bd1ce6d659 WatchSource:0}: Error finding container e9210919df869cca2c1796b79083dd2ae1b214f19c218139cebe17bd1ce6d659: Status 404 returned error can't find the container with id e9210919df869cca2c1796b79083dd2ae1b214f19c218139cebe17bd1ce6d659 Nov 03 02:54:51.910328 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:51.910257 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1c43e9-76f6b754d7-mr225"] Nov 03 02:54:52.423881 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:52.423838 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c43e9-58dfd86775-b7nrz" event={"ID":"ad6cd805-ef8c-4c3a-a63d-52a044bfe9ce","Type":"ContainerStarted","Data":"70c3141a92fa09bbeac1804b8a0468e8cb0554545d43408eb8a5200e76ceef8b"} Nov 03 02:54:52.424323 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:52.423937 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c43e9-58dfd86775-b7nrz" Nov 03 02:54:52.424323 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:52.423962 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c43e9-58dfd86775-b7nrz" event={"ID":"ad6cd805-ef8c-4c3a-a63d-52a044bfe9ce","Type":"ContainerStarted","Data":"e9210919df869cca2c1796b79083dd2ae1b214f19c218139cebe17bd1ce6d659"} Nov 03 02:54:52.425530 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:52.425503 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1c43e9-76f6b754d7-mr225" event={"ID":"e82f0187-0392-499b-ac69-0e7e03eebbf7","Type":"ContainerStarted","Data":"22165f79452f73127f1795d2442ff14df962786c552c1161143b2e0d54e07906"} Nov 03 02:54:52.425636 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:52.425551 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1c43e9-76f6b754d7-mr225" event={"ID":"e82f0187-0392-499b-ac69-0e7e03eebbf7","Type":"ContainerStarted","Data":"fcd43d9ad6209f0c578854b759862366ab387192bfabaf498fd41a187714fcdd"} Nov 03 02:54:52.425636 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:52.425589 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1c43e9-76f6b754d7-mr225" Nov 03 02:54:52.532984 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:52.532937 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c43e9-58dfd86775-b7nrz" podStartSLOduration=1.532924953 podStartE2EDuration="1.532924953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:54:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:54:52.530442741 +0000 UTC m=+3769.789640894" watchObservedRunningTime="2026-11-03 02:54:52.532924953 +0000 UTC m=+3769.792123105" Nov 03 02:54:52.636021 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:52.635975 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-1c43e9-76f6b754d7-mr225" podStartSLOduration=1.635963855 podStartE2EDuration="1.635963855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:54:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:54:52.635222183 +0000 UTC m=+3769.894420336" watchObservedRunningTime="2026-11-03 02:54:52.635963855 +0000 UTC m=+3769.895162009" Nov 03 02:54:53.365980 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:54:53.365945 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-747c9f9449-86gct" podUID="41245c80-d885-4a8c-9ce5-24e59ed83704" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:54:53.914739 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:54:53.914698 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:54:56.750667 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:54:56.750636 5605 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-50a3cab9906d: secret "agc-cred-50a3cab9906d" not found Nov 03 02:54:56.750667 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:54:56.750666 5605 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-f20e06ca5b75: secret "agc-cred-f20e06ca5b75" not found Nov 03 02:54:56.751142 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:54:56.750679 5605 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-68686f579-ntp6g: [secret "agc-cred-50a3cab9906d" not found, secret "agc-cred-f20e06ca5b75" not found] Nov 03 02:54:56.751142 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:54:56.750731 5605 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d89cb81e-1adf-4e6c-82ea-dde66712aab7-provider-credentials podName:d89cb81e-1adf-4e6c-82ea-dde66712aab7 nodeName:}" failed. No retries permitted until 2026-11-03 02:56:58.750716389 +0000 UTC m=+3896.009914533 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d89cb81e-1adf-4e6c-82ea-dde66712aab7-provider-credentials") pod "payload-processing-external-model-68686f579-ntp6g" (UID: "d89cb81e-1adf-4e6c-82ea-dde66712aab7") : [secret "agc-cred-50a3cab9906d" not found, secret "agc-cred-f20e06ca5b75" not found] Nov 03 02:55:01.525732 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:01.525695 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:55:03.365283 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:03.365234 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-747c9f9449-86gct" podUID="41245c80-d885-4a8c-9ce5-24e59ed83704" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:55:03.464397 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:03.462100 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c43e9-58dfd86775-b7nrz" Nov 03 02:55:03.464397 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:03.462648 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1c43e9-76f6b754d7-mr225" Nov 03 02:55:05.219379 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:05.218455 5605 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-521f1f/premium-w2-521f1f-kserve-7578fb7957-89wfb" podUID="3d988573-d8b1-43ba-a423-51a63ca12337" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 02:55:05.274594 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:05.274532 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a0e72e/distinct-w3-a0e72e-kserve-fb499cc4-2dfxc" podUID="308bca3b-9bd5-4724-9b95-acdbc978aa10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Nov 03 02:55:09.292162 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:09.292122 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk"] Nov 03 02:55:09.296154 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:09.296128 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk" Nov 03 02:55:09.340063 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:09.340024 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4-istiod-ca-cert\") pod \"e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk\" (UID: \"c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4\") " pod="openshift-ingress/e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk" Nov 03 02:55:09.340218 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:09.340075 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vld9\" (UniqueName: \"kubernetes.io/projected/c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4-kube-api-access-5vld9\") pod \"e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk\" (UID: \"c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4\") " pod="openshift-ingress/e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk" Nov 03 02:55:09.340290 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:09.340239 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4-istio-token\") pod \"e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk\" (UID: \"c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4\") " pod="openshift-ingress/e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk" Nov 03 02:55:09.340337 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:09.340292 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4-istio-data\") pod \"e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk\" (UID: \"c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4\") " pod="openshift-ingress/e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk" Nov 03 02:55:09.340463 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:09.340438 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4-credential-socket\") pod \"e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk\" (UID: \"c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4\") " pod="openshift-ingress/e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk" Nov 03 02:55:09.340501 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:09.340479 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4-istio-envoy\") pod \"e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk\" (UID: \"c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4\") " pod="openshift-ingress/e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk" Nov 03 02:55:09.340574 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:09.340551 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4-istio-podinfo\") pod \"e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk\" (UID: \"c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4\") " pod="openshift-ingress/e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk" Nov 03 02:55:09.340646 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:09.340628 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4-workload-certs\") pod \"e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk\" (UID: \"c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4\") " pod="openshift-ingress/e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk" Nov 03 02:55:09.340705 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:09.340683 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4-workload-socket\") pod \"e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk\" (UID: \"c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4\") " pod="openshift-ingress/e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk" Nov 03 02:55:09.345893 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:09.345868 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-ddcbd0-openshift-default-dockercfg-c9x8q\"" Nov 03 02:55:09.395299 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:09.395269 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk"] Nov 03 02:55:09.442293 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:09.442253 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4-istiod-ca-cert\") pod \"e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk\" (UID: \"c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4\") " pod="openshift-ingress/e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk" Nov 03 02:55:09.442517 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:09.442302 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vld9\" (UniqueName: \"kubernetes.io/projected/c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4-kube-api-access-5vld9\") pod \"e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk\" (UID: \"c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4\") " pod="openshift-ingress/e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk" Nov 03 02:55:09.442517 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:09.442437 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4-istio-token\") pod \"e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk\" (UID: \"c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4\") " pod="openshift-ingress/e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk" Nov 03 02:55:09.442517 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:09.442488 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4-istio-data\") pod \"e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk\" (UID: \"c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4\") " pod="openshift-ingress/e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk" Nov 03 02:55:09.442792 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:09.442590 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4-credential-socket\") pod \"e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk\" (UID: \"c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4\") " pod="openshift-ingress/e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk" Nov 03 02:55:09.442792 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:09.442625 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4-istio-envoy\") pod \"e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk\" (UID: \"c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4\") " pod="openshift-ingress/e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk" Nov 03 02:55:09.442792 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:09.442656 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4-istio-podinfo\") pod \"e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk\" (UID: \"c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4\") " pod="openshift-ingress/e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk" Nov 03 02:55:09.442792 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:09.442696 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4-workload-certs\") pod \"e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk\" (UID: \"c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4\") " pod="openshift-ingress/e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk" Nov 03 02:55:09.442792 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:09.442749 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4-workload-socket\") pod \"e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk\" (UID: \"c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4\") " pod="openshift-ingress/e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk" Nov 03 02:55:09.443205 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:09.443179 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4-workload-socket\") pod \"e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk\" (UID: \"c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4\") " pod="openshift-ingress/e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk" Nov 03 02:55:09.443514 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:09.443466 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4-istiod-ca-cert\") pod \"e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk\" (UID: \"c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4\") " pod="openshift-ingress/e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk" Nov 03 02:55:09.444124 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:09.443951 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4-istio-data\") pod \"e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk\" (UID: \"c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4\") " pod="openshift-ingress/e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk" Nov 03 02:55:09.446744 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:09.446694 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4-workload-certs\") pod \"e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk\" (UID: \"c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4\") " pod="openshift-ingress/e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk" Nov 03 02:55:09.450972 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:09.450950 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4-istio-podinfo\") pod \"e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk\" (UID: \"c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4\") " pod="openshift-ingress/e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk" Nov 03 02:55:09.451787 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:09.451759 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4-credential-socket\") pod \"e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk\" (UID: \"c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4\") " pod="openshift-ingress/e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk" Nov 03 02:55:09.456600 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:09.456571 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4-istio-envoy\") pod \"e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk\" (UID: \"c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4\") " pod="openshift-ingress/e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk" Nov 03 02:55:09.499377 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:09.496754 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vld9\" (UniqueName: \"kubernetes.io/projected/c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4-kube-api-access-5vld9\") pod \"e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk\" (UID: \"c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4\") " pod="openshift-ingress/e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk" Nov 03 02:55:09.499377 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:09.497175 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4-istio-token\") pod \"e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk\" (UID: \"c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4\") " pod="openshift-ingress/e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk" Nov 03 02:55:09.610050 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:09.609771 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk" Nov 03 02:55:10.034090 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:10.033998 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk"] Nov 03 02:55:10.039732 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:55:10.039702 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5f75ae2_27a2_4f9c_a3a4_feddd4fec3d4.slice/crio-11046157793fe08bc4dbf94ebac1d34ba1be9c82b5829e83d414b5912a280a16 WatchSource:0}: Error finding container 11046157793fe08bc4dbf94ebac1d34ba1be9c82b5829e83d414b5912a280a16: Status 404 returned error can't find the container with id 11046157793fe08bc4dbf94ebac1d34ba1be9c82b5829e83d414b5912a280a16 Nov 03 02:55:10.042557 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:10.042519 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:55:10.042665 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:10.042587 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:55:10.042665 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:10.042613 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:55:10.578263 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:10.578228 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk" event={"ID":"c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4","Type":"ContainerStarted","Data":"5e3619944c4f5eccfe047d840a9ab222a57143857bc0e30dce9dbb5339137dee"} Nov 03 02:55:10.578685 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:10.578285 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk" event={"ID":"c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4","Type":"ContainerStarted","Data":"11046157793fe08bc4dbf94ebac1d34ba1be9c82b5829e83d414b5912a280a16"} Nov 03 02:55:10.610722 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:10.610695 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk" Nov 03 02:55:10.690644 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:10.690580 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk" podStartSLOduration=1.690557172 podStartE2EDuration="1.690557172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:55:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:55:10.687422288 +0000 UTC m=+3787.946620437" watchObservedRunningTime="2026-11-03 02:55:10.690557172 +0000 UTC m=+3787.949755326" Nov 03 02:55:11.616244 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:11.616217 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk" Nov 03 02:55:12.495537 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:12.495505 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh"] Nov 03 02:55:12.499544 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:12.499523 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh" Nov 03 02:55:12.548452 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:12.548432 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-dc14f6-openshift-default-dockercfg-m277s\"" Nov 03 02:55:12.577280 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:12.577254 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e99c38f1-3005-4582-86d8-71e93dd4c1b8-workload-certs\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh\" (UID: \"e99c38f1-3005-4582-86d8-71e93dd4c1b8\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh" Nov 03 02:55:12.577420 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:12.577314 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e99c38f1-3005-4582-86d8-71e93dd4c1b8-istio-token\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh\" (UID: \"e99c38f1-3005-4582-86d8-71e93dd4c1b8\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh" Nov 03 02:55:12.577477 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:12.577438 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e99c38f1-3005-4582-86d8-71e93dd4c1b8-istio-podinfo\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh\" (UID: \"e99c38f1-3005-4582-86d8-71e93dd4c1b8\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh" Nov 03 02:55:12.577477 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:12.577460 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e99c38f1-3005-4582-86d8-71e93dd4c1b8-credential-socket\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh\" (UID: \"e99c38f1-3005-4582-86d8-71e93dd4c1b8\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh" Nov 03 02:55:12.577556 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:12.577492 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfvnm\" (UniqueName: \"kubernetes.io/projected/e99c38f1-3005-4582-86d8-71e93dd4c1b8-kube-api-access-gfvnm\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh\" (UID: \"e99c38f1-3005-4582-86d8-71e93dd4c1b8\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh" Nov 03 02:55:12.577556 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:12.577551 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e99c38f1-3005-4582-86d8-71e93dd4c1b8-istiod-ca-cert\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh\" (UID: \"e99c38f1-3005-4582-86d8-71e93dd4c1b8\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh" Nov 03 02:55:12.577625 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:12.577586 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e99c38f1-3005-4582-86d8-71e93dd4c1b8-istio-data\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh\" (UID: \"e99c38f1-3005-4582-86d8-71e93dd4c1b8\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh" Nov 03 02:55:12.577625 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:12.577603 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e99c38f1-3005-4582-86d8-71e93dd4c1b8-istio-envoy\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh\" (UID: \"e99c38f1-3005-4582-86d8-71e93dd4c1b8\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh" Nov 03 02:55:12.577719 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:12.577629 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e99c38f1-3005-4582-86d8-71e93dd4c1b8-workload-socket\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh\" (UID: \"e99c38f1-3005-4582-86d8-71e93dd4c1b8\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh" Nov 03 02:55:12.591393 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:12.590824 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk" Nov 03 02:55:12.591833 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:12.591814 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk" Nov 03 02:55:12.598113 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:12.598095 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh"] Nov 03 02:55:12.678563 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:12.678539 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e99c38f1-3005-4582-86d8-71e93dd4c1b8-workload-socket\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh\" (UID: \"e99c38f1-3005-4582-86d8-71e93dd4c1b8\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh" Nov 03 02:55:12.678927 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:12.678671 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e99c38f1-3005-4582-86d8-71e93dd4c1b8-workload-certs\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh\" (UID: \"e99c38f1-3005-4582-86d8-71e93dd4c1b8\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh" Nov 03 02:55:12.678927 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:12.678776 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e99c38f1-3005-4582-86d8-71e93dd4c1b8-istio-token\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh\" (UID: \"e99c38f1-3005-4582-86d8-71e93dd4c1b8\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh" Nov 03 02:55:12.678927 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:12.678823 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e99c38f1-3005-4582-86d8-71e93dd4c1b8-istio-podinfo\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh\" (UID: \"e99c38f1-3005-4582-86d8-71e93dd4c1b8\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh" Nov 03 02:55:12.678927 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:12.678855 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e99c38f1-3005-4582-86d8-71e93dd4c1b8-credential-socket\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh\" (UID: \"e99c38f1-3005-4582-86d8-71e93dd4c1b8\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh" Nov 03 02:55:12.678927 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:12.678904 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfvnm\" (UniqueName: \"kubernetes.io/projected/e99c38f1-3005-4582-86d8-71e93dd4c1b8-kube-api-access-gfvnm\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh\" (UID: \"e99c38f1-3005-4582-86d8-71e93dd4c1b8\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh" Nov 03 02:55:12.679195 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:12.678964 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e99c38f1-3005-4582-86d8-71e93dd4c1b8-istiod-ca-cert\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh\" (UID: \"e99c38f1-3005-4582-86d8-71e93dd4c1b8\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh" Nov 03 02:55:12.679195 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:12.679006 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e99c38f1-3005-4582-86d8-71e93dd4c1b8-workload-socket\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh\" (UID: \"e99c38f1-3005-4582-86d8-71e93dd4c1b8\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh" Nov 03 02:55:12.679195 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:12.679023 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e99c38f1-3005-4582-86d8-71e93dd4c1b8-istio-data\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh\" (UID: \"e99c38f1-3005-4582-86d8-71e93dd4c1b8\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh" Nov 03 02:55:12.679195 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:12.679052 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e99c38f1-3005-4582-86d8-71e93dd4c1b8-istio-envoy\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh\" (UID: \"e99c38f1-3005-4582-86d8-71e93dd4c1b8\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh" Nov 03 02:55:12.679195 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:12.679053 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e99c38f1-3005-4582-86d8-71e93dd4c1b8-workload-certs\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh\" (UID: \"e99c38f1-3005-4582-86d8-71e93dd4c1b8\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh" Nov 03 02:55:12.679430 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:12.679213 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e99c38f1-3005-4582-86d8-71e93dd4c1b8-credential-socket\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh\" (UID: \"e99c38f1-3005-4582-86d8-71e93dd4c1b8\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh" Nov 03 02:55:12.679468 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:12.679448 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e99c38f1-3005-4582-86d8-71e93dd4c1b8-istio-data\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh\" (UID: \"e99c38f1-3005-4582-86d8-71e93dd4c1b8\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh" Nov 03 02:55:12.679658 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:12.679631 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e99c38f1-3005-4582-86d8-71e93dd4c1b8-istiod-ca-cert\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh\" (UID: \"e99c38f1-3005-4582-86d8-71e93dd4c1b8\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh" Nov 03 02:55:12.681153 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:12.681126 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e99c38f1-3005-4582-86d8-71e93dd4c1b8-istio-envoy\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh\" (UID: \"e99c38f1-3005-4582-86d8-71e93dd4c1b8\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh" Nov 03 02:55:12.681256 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:12.681239 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e99c38f1-3005-4582-86d8-71e93dd4c1b8-istio-podinfo\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh\" (UID: \"e99c38f1-3005-4582-86d8-71e93dd4c1b8\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh" Nov 03 02:55:12.734106 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:12.734075 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfvnm\" (UniqueName: \"kubernetes.io/projected/e99c38f1-3005-4582-86d8-71e93dd4c1b8-kube-api-access-gfvnm\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh\" (UID: \"e99c38f1-3005-4582-86d8-71e93dd4c1b8\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh" Nov 03 02:55:12.734935 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:12.734908 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e99c38f1-3005-4582-86d8-71e93dd4c1b8-istio-token\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh\" (UID: \"e99c38f1-3005-4582-86d8-71e93dd4c1b8\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh" Nov 03 02:55:12.812968 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:12.812940 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh" Nov 03 02:55:13.239431 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:13.239347 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh"] Nov 03 02:55:13.243540 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:55:13.243517 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode99c38f1_3005_4582_86d8_71e93dd4c1b8.slice/crio-dc002eef3cacc20fdaf287356e11106d918e5262dba4983b0238ce96db86bf73 WatchSource:0}: Error finding container dc002eef3cacc20fdaf287356e11106d918e5262dba4983b0238ce96db86bf73: Status 404 returned error can't find the container with id dc002eef3cacc20fdaf287356e11106d918e5262dba4983b0238ce96db86bf73 Nov 03 02:55:13.245892 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:13.245862 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:55:13.245950 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:13.245926 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:55:13.245996 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:13.245956 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:55:13.365366 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:13.365305 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-747c9f9449-86gct" podUID="41245c80-d885-4a8c-9ce5-24e59ed83704" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:55:13.599099 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:13.599065 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh" event={"ID":"e99c38f1-3005-4582-86d8-71e93dd4c1b8","Type":"ContainerStarted","Data":"3550b274544a13b74102ca60034b637e7b754477895e5f4adfe63a132bcefb48"} Nov 03 02:55:13.599280 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:13.599127 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh" event={"ID":"e99c38f1-3005-4582-86d8-71e93dd4c1b8","Type":"ContainerStarted","Data":"dc002eef3cacc20fdaf287356e11106d918e5262dba4983b0238ce96db86bf73"} Nov 03 02:55:13.761994 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:13.761926 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh" podStartSLOduration=1.76190852 podStartE2EDuration="1.76190852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:55:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:55:13.76167761 +0000 UTC m=+3791.020875764" watchObservedRunningTime="2026-11-03 02:55:13.76190852 +0000 UTC m=+3791.021106675" Nov 03 02:55:13.813626 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:13.813593 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh" Nov 03 02:55:14.815709 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:14.815678 5605 patch_prober.go:29] interesting pod/e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body= Nov 03 02:55:14.816108 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:14.815727 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh" podUID="e99c38f1-3005-4582-86d8-71e93dd4c1b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" Nov 03 02:55:15.813871 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:15.813839 5605 patch_prober.go:29] interesting pod/e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body= Nov 03 02:55:15.814050 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:15.813887 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh" podUID="e99c38f1-3005-4582-86d8-71e93dd4c1b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" Nov 03 02:55:16.814861 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:16.814820 5605 patch_prober.go:29] interesting pod/e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body= Nov 03 02:55:16.815270 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:16.814875 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh" podUID="e99c38f1-3005-4582-86d8-71e93dd4c1b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" Nov 03 02:55:17.814139 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:17.814107 5605 patch_prober.go:29] interesting pod/e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body= Nov 03 02:55:17.814312 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:17.814155 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh" podUID="e99c38f1-3005-4582-86d8-71e93dd4c1b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" Nov 03 02:55:18.813462 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:18.813423 5605 patch_prober.go:29] interesting pod/e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body= Nov 03 02:55:18.813942 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:18.813485 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh" podUID="e99c38f1-3005-4582-86d8-71e93dd4c1b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" Nov 03 02:55:19.818396 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:19.818348 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh" Nov 03 02:55:19.818864 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:19.818802 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh" Nov 03 02:55:19.819411 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:19.819394 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh" Nov 03 02:55:23.364507 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:23.364469 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-747c9f9449-86gct" podUID="41245c80-d885-4a8c-9ce5-24e59ed83704" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:55:23.912834 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:55:23.912799 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:55:33.365037 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:33.364998 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-747c9f9449-86gct" podUID="41245c80-d885-4a8c-9ce5-24e59ed83704" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:55:36.478526 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:36.478492 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9a5e31-5bc595cb95-5zvn4"] Nov 03 02:55:36.482574 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:36.482551 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9a5e31-5bc595cb95-5zvn4" Nov 03 02:55:36.526408 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:36.526344 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5fa99f5-77dc-443e-a11b-9035ca6baf5a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9a5e31-5bc595cb95-5zvn4\" (UID: \"e5fa99f5-77dc-443e-a11b-9035ca6baf5a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9a5e31-5bc595cb95-5zvn4" Nov 03 02:55:36.526572 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:36.526512 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6k4c\" (UniqueName: \"kubernetes.io/projected/e5fa99f5-77dc-443e-a11b-9035ca6baf5a-kube-api-access-r6k4c\") pod \"payload-pre-processing-e2e-inf-b-9a5e31-5bc595cb95-5zvn4\" (UID: \"e5fa99f5-77dc-443e-a11b-9035ca6baf5a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9a5e31-5bc595cb95-5zvn4" Nov 03 02:55:36.532044 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:36.532016 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-9a5e31\"" Nov 03 02:55:36.578942 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:36.578895 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-9a5e31-dockercfg-w7b8n\"" Nov 03 02:55:36.583957 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:36.583923 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9a5e31-5bc595cb95-5zvn4"] Nov 03 02:55:36.634620 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:36.634564 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5fa99f5-77dc-443e-a11b-9035ca6baf5a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9a5e31-5bc595cb95-5zvn4\" (UID: \"e5fa99f5-77dc-443e-a11b-9035ca6baf5a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9a5e31-5bc595cb95-5zvn4" Nov 03 02:55:36.635119 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:36.635091 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6k4c\" (UniqueName: \"kubernetes.io/projected/e5fa99f5-77dc-443e-a11b-9035ca6baf5a-kube-api-access-r6k4c\") pod \"payload-pre-processing-e2e-inf-b-9a5e31-5bc595cb95-5zvn4\" (UID: \"e5fa99f5-77dc-443e-a11b-9035ca6baf5a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9a5e31-5bc595cb95-5zvn4" Nov 03 02:55:36.637101 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:36.637075 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5fa99f5-77dc-443e-a11b-9035ca6baf5a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9a5e31-5bc595cb95-5zvn4\" (UID: \"e5fa99f5-77dc-443e-a11b-9035ca6baf5a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9a5e31-5bc595cb95-5zvn4" Nov 03 02:55:36.639992 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:36.639963 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9a5e31-558d4bf67-2n6dc"] Nov 03 02:55:36.652122 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:36.649727 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9a5e31-558d4bf67-2n6dc" Nov 03 02:55:36.687869 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:36.687836 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9a5e31-558d4bf67-2n6dc"] Nov 03 02:55:36.692921 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:36.692892 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6k4c\" (UniqueName: \"kubernetes.io/projected/e5fa99f5-77dc-443e-a11b-9035ca6baf5a-kube-api-access-r6k4c\") pod \"payload-pre-processing-e2e-inf-b-9a5e31-5bc595cb95-5zvn4\" (UID: \"e5fa99f5-77dc-443e-a11b-9035ca6baf5a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9a5e31-5bc595cb95-5zvn4" Nov 03 02:55:36.797181 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:36.797102 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9a5e31-5bc595cb95-5zvn4" Nov 03 02:55:36.837071 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:36.837039 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f68e7c9-3df8-4df4-a9d8-0544d285c291-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9a5e31-558d4bf67-2n6dc\" (UID: \"2f68e7c9-3df8-4df4-a9d8-0544d285c291\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9a5e31-558d4bf67-2n6dc" Nov 03 02:55:36.837295 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:36.837279 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wkqj\" (UniqueName: \"kubernetes.io/projected/2f68e7c9-3df8-4df4-a9d8-0544d285c291-kube-api-access-5wkqj\") pod \"payload-processing-e2e-inf-b-9a5e31-558d4bf67-2n6dc\" (UID: \"2f68e7c9-3df8-4df4-a9d8-0544d285c291\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9a5e31-558d4bf67-2n6dc" Nov 03 02:55:36.938810 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:36.938779 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f68e7c9-3df8-4df4-a9d8-0544d285c291-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9a5e31-558d4bf67-2n6dc\" (UID: \"2f68e7c9-3df8-4df4-a9d8-0544d285c291\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9a5e31-558d4bf67-2n6dc" Nov 03 02:55:36.939096 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:36.939073 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wkqj\" (UniqueName: \"kubernetes.io/projected/2f68e7c9-3df8-4df4-a9d8-0544d285c291-kube-api-access-5wkqj\") pod \"payload-processing-e2e-inf-b-9a5e31-558d4bf67-2n6dc\" (UID: \"2f68e7c9-3df8-4df4-a9d8-0544d285c291\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9a5e31-558d4bf67-2n6dc" Nov 03 02:55:36.939568 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:36.939552 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f68e7c9-3df8-4df4-a9d8-0544d285c291-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9a5e31-558d4bf67-2n6dc\" (UID: \"2f68e7c9-3df8-4df4-a9d8-0544d285c291\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9a5e31-558d4bf67-2n6dc" Nov 03 02:55:36.992438 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:36.992415 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wkqj\" (UniqueName: \"kubernetes.io/projected/2f68e7c9-3df8-4df4-a9d8-0544d285c291-kube-api-access-5wkqj\") pod \"payload-processing-e2e-inf-b-9a5e31-558d4bf67-2n6dc\" (UID: \"2f68e7c9-3df8-4df4-a9d8-0544d285c291\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9a5e31-558d4bf67-2n6dc" Nov 03 02:55:37.171654 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:37.171611 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9a5e31-5bc595cb95-5zvn4"] Nov 03 02:55:37.177119 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:55:37.177092 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5fa99f5_77dc_443e_a11b_9035ca6baf5a.slice/crio-71b5a9a81b6cd0dc1a597157df71cb1882bbc08657b2b3bbc75cf6e64a9ac4d1 WatchSource:0}: Error finding container 71b5a9a81b6cd0dc1a597157df71cb1882bbc08657b2b3bbc75cf6e64a9ac4d1: Status 404 returned error can't find the container with id 71b5a9a81b6cd0dc1a597157df71cb1882bbc08657b2b3bbc75cf6e64a9ac4d1 Nov 03 02:55:37.267956 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:37.267924 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9a5e31-558d4bf67-2n6dc" Nov 03 02:55:37.640689 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:37.640659 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9a5e31-558d4bf67-2n6dc"] Nov 03 02:55:37.645226 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:55:37.645198 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f68e7c9_3df8_4df4_a9d8_0544d285c291.slice/crio-0ef1497beec096ef42b5ee752924ee89f90d2ec842939acd38f5a89abc0817d4 WatchSource:0}: Error finding container 0ef1497beec096ef42b5ee752924ee89f90d2ec842939acd38f5a89abc0817d4: Status 404 returned error can't find the container with id 0ef1497beec096ef42b5ee752924ee89f90d2ec842939acd38f5a89abc0817d4 Nov 03 02:55:37.740723 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:37.740695 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9a5e31-5bc595cb95-5zvn4" event={"ID":"e5fa99f5-77dc-443e-a11b-9035ca6baf5a","Type":"ContainerStarted","Data":"91f37a4bb6a6c4de5620732aa6b432b477689e018c08ed267aaddf6c80a0534f"} Nov 03 02:55:37.740833 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:37.740746 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9a5e31-5bc595cb95-5zvn4" event={"ID":"e5fa99f5-77dc-443e-a11b-9035ca6baf5a","Type":"ContainerStarted","Data":"71b5a9a81b6cd0dc1a597157df71cb1882bbc08657b2b3bbc75cf6e64a9ac4d1"} Nov 03 02:55:37.740907 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:37.740864 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9a5e31-5bc595cb95-5zvn4" Nov 03 02:55:37.742493 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:37.742471 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9a5e31-558d4bf67-2n6dc" event={"ID":"2f68e7c9-3df8-4df4-a9d8-0544d285c291","Type":"ContainerStarted","Data":"6e835840b2e6f545be2b1801e262a1589f28a41d17cec2e41baf3ce866400b90"} Nov 03 02:55:37.742589 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:37.742507 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9a5e31-558d4bf67-2n6dc" event={"ID":"2f68e7c9-3df8-4df4-a9d8-0544d285c291","Type":"ContainerStarted","Data":"0ef1497beec096ef42b5ee752924ee89f90d2ec842939acd38f5a89abc0817d4"} Nov 03 02:55:37.742589 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:37.742543 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-9a5e31-558d4bf67-2n6dc" Nov 03 02:55:37.848279 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:37.848232 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9a5e31-5bc595cb95-5zvn4" podStartSLOduration=1.8482200789999998 podStartE2EDuration="1.848220079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:55:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:55:37.845961554 +0000 UTC m=+3815.105159707" watchObservedRunningTime="2026-11-03 02:55:37.848220079 +0000 UTC m=+3815.107418232" Nov 03 02:55:37.952793 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:37.952716 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-9a5e31-558d4bf67-2n6dc" podStartSLOduration=1.952703029 podStartE2EDuration="1.952703029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:55:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:55:37.95164312 +0000 UTC m=+3815.210841272" watchObservedRunningTime="2026-11-03 02:55:37.952703029 +0000 UTC m=+3815.211901181" Nov 03 02:55:42.734952 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:42.734919 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj"] Nov 03 02:55:42.739219 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:42.739203 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj" Nov 03 02:55:42.781757 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:42.781733 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk"] Nov 03 02:55:42.785540 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:42.785526 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk" Nov 03 02:55:42.810408 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:42.810382 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/115ad04c-ad71-4cf1-9f39-a22ed3224fca-istio-token\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj\" (UID: \"115ad04c-ad71-4cf1-9f39-a22ed3224fca\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj" Nov 03 02:55:42.810408 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:42.810412 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/115ad04c-ad71-4cf1-9f39-a22ed3224fca-workload-certs\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj\" (UID: \"115ad04c-ad71-4cf1-9f39-a22ed3224fca\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj" Nov 03 02:55:42.810583 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:42.810446 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/115ad04c-ad71-4cf1-9f39-a22ed3224fca-istio-podinfo\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj\" (UID: \"115ad04c-ad71-4cf1-9f39-a22ed3224fca\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj" Nov 03 02:55:42.810583 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:42.810535 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/115ad04c-ad71-4cf1-9f39-a22ed3224fca-istiod-ca-cert\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj\" (UID: \"115ad04c-ad71-4cf1-9f39-a22ed3224fca\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj" Nov 03 02:55:42.810583 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:42.810571 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/115ad04c-ad71-4cf1-9f39-a22ed3224fca-workload-socket\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj\" (UID: \"115ad04c-ad71-4cf1-9f39-a22ed3224fca\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj" Nov 03 02:55:42.810777 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:42.810755 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/115ad04c-ad71-4cf1-9f39-a22ed3224fca-istio-envoy\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj\" (UID: \"115ad04c-ad71-4cf1-9f39-a22ed3224fca\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj" Nov 03 02:55:42.810839 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:42.810805 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/115ad04c-ad71-4cf1-9f39-a22ed3224fca-credential-socket\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj\" (UID: \"115ad04c-ad71-4cf1-9f39-a22ed3224fca\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj" Nov 03 02:55:42.810880 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:42.810841 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/115ad04c-ad71-4cf1-9f39-a22ed3224fca-istio-data\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj\" (UID: \"115ad04c-ad71-4cf1-9f39-a22ed3224fca\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj" Nov 03 02:55:42.810920 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:42.810881 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n6sz\" (UniqueName: \"kubernetes.io/projected/115ad04c-ad71-4cf1-9f39-a22ed3224fca-kube-api-access-7n6sz\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj\" (UID: \"115ad04c-ad71-4cf1-9f39-a22ed3224fca\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj" Nov 03 02:55:42.834198 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:42.834155 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj"] Nov 03 02:55:42.885733 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:42.885698 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk"] Nov 03 02:55:42.911991 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:42.911956 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/115ad04c-ad71-4cf1-9f39-a22ed3224fca-istio-token\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj\" (UID: \"115ad04c-ad71-4cf1-9f39-a22ed3224fca\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj" Nov 03 02:55:42.912160 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:42.912004 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/115ad04c-ad71-4cf1-9f39-a22ed3224fca-workload-certs\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj\" (UID: \"115ad04c-ad71-4cf1-9f39-a22ed3224fca\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj" Nov 03 02:55:42.912160 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:42.912050 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/115ad04c-ad71-4cf1-9f39-a22ed3224fca-istio-podinfo\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj\" (UID: \"115ad04c-ad71-4cf1-9f39-a22ed3224fca\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj" Nov 03 02:55:42.912160 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:42.912076 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2905ac02-0ee5-4c4d-bf53-f51adabf9744-workload-certs\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk\" (UID: \"2905ac02-0ee5-4c4d-bf53-f51adabf9744\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk" Nov 03 02:55:42.912160 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:42.912115 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/115ad04c-ad71-4cf1-9f39-a22ed3224fca-istiod-ca-cert\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj\" (UID: \"115ad04c-ad71-4cf1-9f39-a22ed3224fca\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj" Nov 03 02:55:42.912160 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:42.912142 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2905ac02-0ee5-4c4d-bf53-f51adabf9744-workload-socket\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk\" (UID: \"2905ac02-0ee5-4c4d-bf53-f51adabf9744\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk" Nov 03 02:55:42.912564 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:42.912167 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2905ac02-0ee5-4c4d-bf53-f51adabf9744-istio-podinfo\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk\" (UID: \"2905ac02-0ee5-4c4d-bf53-f51adabf9744\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk" Nov 03 02:55:42.912564 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:42.912218 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/115ad04c-ad71-4cf1-9f39-a22ed3224fca-workload-socket\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj\" (UID: \"115ad04c-ad71-4cf1-9f39-a22ed3224fca\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj" Nov 03 02:55:42.912564 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:42.912255 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2905ac02-0ee5-4c4d-bf53-f51adabf9744-istio-envoy\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk\" (UID: \"2905ac02-0ee5-4c4d-bf53-f51adabf9744\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk" Nov 03 02:55:42.912564 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:42.912279 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2905ac02-0ee5-4c4d-bf53-f51adabf9744-istiod-ca-cert\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk\" (UID: \"2905ac02-0ee5-4c4d-bf53-f51adabf9744\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk" Nov 03 02:55:42.912564 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:42.912321 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps7s7\" (UniqueName: \"kubernetes.io/projected/2905ac02-0ee5-4c4d-bf53-f51adabf9744-kube-api-access-ps7s7\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk\" (UID: \"2905ac02-0ee5-4c4d-bf53-f51adabf9744\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk" Nov 03 02:55:42.912564 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:42.912404 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2905ac02-0ee5-4c4d-bf53-f51adabf9744-istio-token\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk\" (UID: \"2905ac02-0ee5-4c4d-bf53-f51adabf9744\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk" Nov 03 02:55:42.912564 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:42.912461 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2905ac02-0ee5-4c4d-bf53-f51adabf9744-credential-socket\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk\" (UID: \"2905ac02-0ee5-4c4d-bf53-f51adabf9744\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk" Nov 03 02:55:42.912564 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:42.912461 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/115ad04c-ad71-4cf1-9f39-a22ed3224fca-workload-certs\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj\" (UID: \"115ad04c-ad71-4cf1-9f39-a22ed3224fca\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj" Nov 03 02:55:42.912564 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:42.912504 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/115ad04c-ad71-4cf1-9f39-a22ed3224fca-istio-envoy\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj\" (UID: \"115ad04c-ad71-4cf1-9f39-a22ed3224fca\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj" Nov 03 02:55:42.913095 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:42.912752 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/115ad04c-ad71-4cf1-9f39-a22ed3224fca-workload-socket\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj\" (UID: \"115ad04c-ad71-4cf1-9f39-a22ed3224fca\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj" Nov 03 02:55:42.913540 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:42.913508 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2905ac02-0ee5-4c4d-bf53-f51adabf9744-istio-data\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk\" (UID: \"2905ac02-0ee5-4c4d-bf53-f51adabf9744\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk" Nov 03 02:55:42.913695 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:42.913633 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/115ad04c-ad71-4cf1-9f39-a22ed3224fca-istiod-ca-cert\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj\" (UID: \"115ad04c-ad71-4cf1-9f39-a22ed3224fca\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj" Nov 03 02:55:42.913818 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:42.913750 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/115ad04c-ad71-4cf1-9f39-a22ed3224fca-credential-socket\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj\" (UID: \"115ad04c-ad71-4cf1-9f39-a22ed3224fca\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj" Nov 03 02:55:42.913818 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:42.913797 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/115ad04c-ad71-4cf1-9f39-a22ed3224fca-istio-data\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj\" (UID: \"115ad04c-ad71-4cf1-9f39-a22ed3224fca\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj" Nov 03 02:55:42.914083 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:42.914059 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/115ad04c-ad71-4cf1-9f39-a22ed3224fca-credential-socket\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj\" (UID: \"115ad04c-ad71-4cf1-9f39-a22ed3224fca\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj" Nov 03 02:55:42.914155 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:42.914123 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/115ad04c-ad71-4cf1-9f39-a22ed3224fca-istio-data\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj\" (UID: \"115ad04c-ad71-4cf1-9f39-a22ed3224fca\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj" Nov 03 02:55:42.914219 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:42.914150 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n6sz\" (UniqueName: \"kubernetes.io/projected/115ad04c-ad71-4cf1-9f39-a22ed3224fca-kube-api-access-7n6sz\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj\" (UID: \"115ad04c-ad71-4cf1-9f39-a22ed3224fca\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj" Nov 03 02:55:42.915116 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:42.915094 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/115ad04c-ad71-4cf1-9f39-a22ed3224fca-istio-envoy\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj\" (UID: \"115ad04c-ad71-4cf1-9f39-a22ed3224fca\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj" Nov 03 02:55:42.916852 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:42.916833 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/115ad04c-ad71-4cf1-9f39-a22ed3224fca-istio-podinfo\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj\" (UID: \"115ad04c-ad71-4cf1-9f39-a22ed3224fca\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj" Nov 03 02:55:42.966074 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:42.966045 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/115ad04c-ad71-4cf1-9f39-a22ed3224fca-istio-token\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj\" (UID: \"115ad04c-ad71-4cf1-9f39-a22ed3224fca\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj" Nov 03 02:55:42.966535 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:42.966511 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n6sz\" (UniqueName: \"kubernetes.io/projected/115ad04c-ad71-4cf1-9f39-a22ed3224fca-kube-api-access-7n6sz\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj\" (UID: \"115ad04c-ad71-4cf1-9f39-a22ed3224fca\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj" Nov 03 02:55:43.015464 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:43.015409 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2905ac02-0ee5-4c4d-bf53-f51adabf9744-workload-certs\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk\" (UID: \"2905ac02-0ee5-4c4d-bf53-f51adabf9744\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk" Nov 03 02:55:43.015464 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:43.015449 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2905ac02-0ee5-4c4d-bf53-f51adabf9744-workload-socket\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk\" (UID: \"2905ac02-0ee5-4c4d-bf53-f51adabf9744\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk" Nov 03 02:55:43.015464 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:43.015465 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2905ac02-0ee5-4c4d-bf53-f51adabf9744-istio-podinfo\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk\" (UID: \"2905ac02-0ee5-4c4d-bf53-f51adabf9744\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk" Nov 03 02:55:43.015670 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:43.015517 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2905ac02-0ee5-4c4d-bf53-f51adabf9744-istio-envoy\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk\" (UID: \"2905ac02-0ee5-4c4d-bf53-f51adabf9744\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk" Nov 03 02:55:43.015670 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:43.015534 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2905ac02-0ee5-4c4d-bf53-f51adabf9744-istiod-ca-cert\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk\" (UID: \"2905ac02-0ee5-4c4d-bf53-f51adabf9744\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk" Nov 03 02:55:43.015670 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:43.015649 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps7s7\" (UniqueName: \"kubernetes.io/projected/2905ac02-0ee5-4c4d-bf53-f51adabf9744-kube-api-access-ps7s7\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk\" (UID: \"2905ac02-0ee5-4c4d-bf53-f51adabf9744\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk" Nov 03 02:55:43.015838 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:43.015724 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2905ac02-0ee5-4c4d-bf53-f51adabf9744-istio-token\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk\" (UID: \"2905ac02-0ee5-4c4d-bf53-f51adabf9744\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk" Nov 03 02:55:43.015838 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:43.015783 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2905ac02-0ee5-4c4d-bf53-f51adabf9744-workload-certs\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk\" (UID: \"2905ac02-0ee5-4c4d-bf53-f51adabf9744\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk" Nov 03 02:55:43.015838 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:43.015789 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2905ac02-0ee5-4c4d-bf53-f51adabf9744-credential-socket\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk\" (UID: \"2905ac02-0ee5-4c4d-bf53-f51adabf9744\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk" Nov 03 02:55:43.016017 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:43.015857 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2905ac02-0ee5-4c4d-bf53-f51adabf9744-istio-data\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk\" (UID: \"2905ac02-0ee5-4c4d-bf53-f51adabf9744\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk" Nov 03 02:55:43.016017 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:43.015894 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2905ac02-0ee5-4c4d-bf53-f51adabf9744-workload-socket\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk\" (UID: \"2905ac02-0ee5-4c4d-bf53-f51adabf9744\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk" Nov 03 02:55:43.016128 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:43.016076 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2905ac02-0ee5-4c4d-bf53-f51adabf9744-credential-socket\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk\" (UID: \"2905ac02-0ee5-4c4d-bf53-f51adabf9744\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk" Nov 03 02:55:43.016185 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:43.016129 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2905ac02-0ee5-4c4d-bf53-f51adabf9744-istio-data\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk\" (UID: \"2905ac02-0ee5-4c4d-bf53-f51adabf9744\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk" Nov 03 02:55:43.016415 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:43.016390 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2905ac02-0ee5-4c4d-bf53-f51adabf9744-istiod-ca-cert\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk\" (UID: \"2905ac02-0ee5-4c4d-bf53-f51adabf9744\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk" Nov 03 02:55:43.017637 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:43.017618 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2905ac02-0ee5-4c4d-bf53-f51adabf9744-istio-envoy\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk\" (UID: \"2905ac02-0ee5-4c4d-bf53-f51adabf9744\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk" Nov 03 02:55:43.017893 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:43.017873 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2905ac02-0ee5-4c4d-bf53-f51adabf9744-istio-podinfo\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk\" (UID: \"2905ac02-0ee5-4c4d-bf53-f51adabf9744\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk" Nov 03 02:55:43.051926 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:43.051902 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj" Nov 03 02:55:43.070877 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:43.070860 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps7s7\" (UniqueName: \"kubernetes.io/projected/2905ac02-0ee5-4c4d-bf53-f51adabf9744-kube-api-access-ps7s7\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk\" (UID: \"2905ac02-0ee5-4c4d-bf53-f51adabf9744\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk" Nov 03 02:55:43.071160 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:43.071141 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2905ac02-0ee5-4c4d-bf53-f51adabf9744-istio-token\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk\" (UID: \"2905ac02-0ee5-4c4d-bf53-f51adabf9744\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk" Nov 03 02:55:43.098914 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:43.098886 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk" Nov 03 02:55:43.365247 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:43.365210 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-747c9f9449-86gct" podUID="41245c80-d885-4a8c-9ce5-24e59ed83704" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:55:43.473117 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:43.473086 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj"] Nov 03 02:55:43.478287 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:55:43.478263 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod115ad04c_ad71_4cf1_9f39_a22ed3224fca.slice/crio-7e2269fb3643630ddab686abb963a2c985894fdd730acf80b8caad2c93b1280f WatchSource:0}: Error finding container 7e2269fb3643630ddab686abb963a2c985894fdd730acf80b8caad2c93b1280f: Status 404 returned error can't find the container with id 7e2269fb3643630ddab686abb963a2c985894fdd730acf80b8caad2c93b1280f Nov 03 02:55:43.480706 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:43.480676 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:55:43.480777 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:43.480739 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:55:43.480777 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:43.480768 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:55:43.527740 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:43.527718 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk"] Nov 03 02:55:43.533684 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:55:43.533659 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2905ac02_0ee5_4c4d_bf53_f51adabf9744.slice/crio-ef1071ccb8cdf538c8cee4429874be65195eaaa1b99670c2a3afcb30fcea52c2 WatchSource:0}: Error finding container ef1071ccb8cdf538c8cee4429874be65195eaaa1b99670c2a3afcb30fcea52c2: Status 404 returned error can't find the container with id ef1071ccb8cdf538c8cee4429874be65195eaaa1b99670c2a3afcb30fcea52c2 Nov 03 02:55:43.536170 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:43.536143 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:55:43.536249 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:43.536200 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:55:43.536249 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:43.536228 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:55:43.783129 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:43.783041 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk" event={"ID":"2905ac02-0ee5-4c4d-bf53-f51adabf9744","Type":"ContainerStarted","Data":"ee1909a1d7abc5ed8895fb454e848c16a18a5b6e51571d3e7a1c756e8abe102b"} Nov 03 02:55:43.783129 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:43.783096 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk" event={"ID":"2905ac02-0ee5-4c4d-bf53-f51adabf9744","Type":"ContainerStarted","Data":"ef1071ccb8cdf538c8cee4429874be65195eaaa1b99670c2a3afcb30fcea52c2"} Nov 03 02:55:43.784958 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:43.784932 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj" event={"ID":"115ad04c-ad71-4cf1-9f39-a22ed3224fca","Type":"ContainerStarted","Data":"91d87857cee40bfa59bdb43f6ef69bb2069040d7519390ecc2610c7608772c01"} Nov 03 02:55:43.785100 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:43.784969 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj" event={"ID":"115ad04c-ad71-4cf1-9f39-a22ed3224fca","Type":"ContainerStarted","Data":"7e2269fb3643630ddab686abb963a2c985894fdd730acf80b8caad2c93b1280f"} Nov 03 02:55:43.894774 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:43.894711 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk" podStartSLOduration=1.8946932749999998 podStartE2EDuration="1.894693275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:55:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:55:43.893742358 +0000 UTC m=+3821.152940522" watchObservedRunningTime="2026-11-03 02:55:43.894693275 +0000 UTC m=+3821.153891428" Nov 03 02:55:44.008352 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:44.007516 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj" podStartSLOduration=2.007495877 podStartE2EDuration="2.007495877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:55:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:55:44.004901407 +0000 UTC m=+3821.264099562" watchObservedRunningTime="2026-11-03 02:55:44.007495877 +0000 UTC m=+3821.266694032" Nov 03 02:55:44.052761 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:44.052731 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj" Nov 03 02:55:44.054080 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:44.054055 5605 patch_prober.go:29] interesting pod/e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Nov 03 02:55:44.054213 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:44.054099 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj" podUID="115ad04c-ad71-4cf1-9f39-a22ed3224fca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Nov 03 02:55:44.100909 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:44.100864 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk" Nov 03 02:55:44.102063 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:44.102032 5605 patch_prober.go:29] interesting pod/e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Nov 03 02:55:44.102348 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:44.102281 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk" podUID="2905ac02-0ee5-4c4d-bf53-f51adabf9744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Nov 03 02:55:45.052593 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:45.052563 5605 patch_prober.go:29] interesting pod/e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Nov 03 02:55:45.053056 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:45.052611 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj" podUID="115ad04c-ad71-4cf1-9f39-a22ed3224fca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Nov 03 02:55:45.103759 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:45.103720 5605 patch_prober.go:29] interesting pod/e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Nov 03 02:55:45.104466 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:45.104433 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk" podUID="2905ac02-0ee5-4c4d-bf53-f51adabf9744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Nov 03 02:55:46.053038 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:46.052990 5605 patch_prober.go:29] interesting pod/e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Nov 03 02:55:46.053453 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:46.053054 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj" podUID="115ad04c-ad71-4cf1-9f39-a22ed3224fca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Nov 03 02:55:46.103867 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:46.103826 5605 patch_prober.go:29] interesting pod/e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Nov 03 02:55:46.104160 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:46.104130 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk" podUID="2905ac02-0ee5-4c4d-bf53-f51adabf9744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Nov 03 02:55:47.052654 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:47.052614 5605 patch_prober.go:29] interesting pod/e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Nov 03 02:55:47.052891 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:47.052670 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj" podUID="115ad04c-ad71-4cf1-9f39-a22ed3224fca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Nov 03 02:55:47.101383 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:47.100842 5605 patch_prober.go:29] interesting pod/e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Nov 03 02:55:47.101383 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:47.100927 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk" podUID="2905ac02-0ee5-4c4d-bf53-f51adabf9744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Nov 03 02:55:48.052666 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:48.052627 5605 patch_prober.go:29] interesting pod/e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Nov 03 02:55:48.052882 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:48.052688 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj" podUID="115ad04c-ad71-4cf1-9f39-a22ed3224fca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Nov 03 02:55:48.104816 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:48.104767 5605 patch_prober.go:29] interesting pod/e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Nov 03 02:55:48.105314 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:48.104829 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk" podUID="2905ac02-0ee5-4c4d-bf53-f51adabf9744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Nov 03 02:55:48.751542 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:48.751510 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-9a5e31-558d4bf67-2n6dc" Nov 03 02:55:48.751729 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:48.751579 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9a5e31-5bc595cb95-5zvn4" Nov 03 02:55:49.052528 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:49.052494 5605 patch_prober.go:29] interesting pod/e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Nov 03 02:55:49.052730 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:49.052561 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj" podUID="115ad04c-ad71-4cf1-9f39-a22ed3224fca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Nov 03 02:55:49.105339 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:49.105303 5605 patch_prober.go:29] interesting pod/e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Nov 03 02:55:49.108523 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:49.108470 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk" podUID="2905ac02-0ee5-4c4d-bf53-f51adabf9744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Nov 03 02:55:50.057575 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:50.057540 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj" Nov 03 02:55:50.057961 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:50.057939 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj" Nov 03 02:55:50.058657 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:50.058639 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj" Nov 03 02:55:50.104070 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:50.104039 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk" Nov 03 02:55:50.104333 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:50.104312 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk" Nov 03 02:55:50.104946 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:50.104927 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk" Nov 03 02:55:53.364820 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:55:53.364786 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-747c9f9449-86gct" podUID="41245c80-d885-4a8c-9ce5-24e59ed83704" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:55:53.912944 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:55:53.912906 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:56:00.844821 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:00.844782 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-747c9f9449-86gct" podUID="41245c80-d885-4a8c-9ce5-24e59ed83704" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:56:04.796300 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:04.796267 5605 scope.go:122] "RemoveContainer" containerID="f8a3f0d4807fb3d320385417ad1b7024a03109c01b21af494473392b8f78bbfe" Nov 03 02:56:10.845228 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:10.845177 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-747c9f9449-86gct" podUID="41245c80-d885-4a8c-9ce5-24e59ed83704" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:56:20.844933 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:20.844889 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-747c9f9449-86gct" podUID="41245c80-d885-4a8c-9ce5-24e59ed83704" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:56:21.869970 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:21.869935 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e039cb-6f47f9bfd7-9zqfn"] Nov 03 02:56:21.972509 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:21.972484 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e039cb-5bbb449d87-t8dnw"] Nov 03 02:56:23.913343 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:56:23.913305 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:56:26.871232 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:26.871190 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-e039cb-6f47f9bfd7-9zqfn" podUID="080ef559-4fcc-46ed-83d2-d9fa1e168792" containerName="payload-processing" containerID="cri-o://052fb46f0b7a12f70c595a298bb2633a1fba76400d37da8cb2f21b00374d6e82" gracePeriod=25 Nov 03 02:56:26.973429 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:26.973387 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e039cb-5bbb449d87-t8dnw" podUID="cec8117c-77b5-4cf8-8c6c-f3e5bffbf72b" containerName="payload-pre-processing" containerID="cri-o://936cbadc551e0d220087f8d1d86bffd3ac2ae1e68c249f684c25217c5f604707" gracePeriod=25 Nov 03 02:56:27.085605 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:27.085574 5605 generic.go:356] "Generic (PLEG): container finished" podID="080ef559-4fcc-46ed-83d2-d9fa1e168792" containerID="052fb46f0b7a12f70c595a298bb2633a1fba76400d37da8cb2f21b00374d6e82" exitCode=0 Nov 03 02:56:27.085798 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:27.085651 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-e039cb-6f47f9bfd7-9zqfn" event={"ID":"080ef559-4fcc-46ed-83d2-d9fa1e168792","Type":"ContainerDied","Data":"052fb46f0b7a12f70c595a298bb2633a1fba76400d37da8cb2f21b00374d6e82"} Nov 03 02:56:27.129566 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:27.129502 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-e039cb-6f47f9bfd7-9zqfn" Nov 03 02:56:27.242716 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:27.242405 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/080ef559-4fcc-46ed-83d2-d9fa1e168792-kube-api-access-2vxv8\" (UniqueName: \"kubernetes.io/projected/080ef559-4fcc-46ed-83d2-d9fa1e168792-kube-api-access-2vxv8\") pod \"080ef559-4fcc-46ed-83d2-d9fa1e168792\" (UID: \"080ef559-4fcc-46ed-83d2-d9fa1e168792\") " Nov 03 02:56:27.242716 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:27.242510 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/080ef559-4fcc-46ed-83d2-d9fa1e168792-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/080ef559-4fcc-46ed-83d2-d9fa1e168792-plugins-config-volume\") pod \"080ef559-4fcc-46ed-83d2-d9fa1e168792\" (UID: \"080ef559-4fcc-46ed-83d2-d9fa1e168792\") " Nov 03 02:56:27.243065 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:27.243039 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/080ef559-4fcc-46ed-83d2-d9fa1e168792-plugins-config-volume" pod "080ef559-4fcc-46ed-83d2-d9fa1e168792" (UID: "080ef559-4fcc-46ed-83d2-d9fa1e168792"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:56:27.244803 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:27.244776 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/080ef559-4fcc-46ed-83d2-d9fa1e168792-kube-api-access-2vxv8" pod "080ef559-4fcc-46ed-83d2-d9fa1e168792" (UID: "080ef559-4fcc-46ed-83d2-d9fa1e168792"). InnerVolumeSpecName "kube-api-access-2vxv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:56:27.246786 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:27.246762 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e039cb-5bbb449d87-t8dnw" Nov 03 02:56:27.344525 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:27.344482 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cec8117c-77b5-4cf8-8c6c-f3e5bffbf72b-kube-api-access-fpkqr\" (UniqueName: \"kubernetes.io/projected/cec8117c-77b5-4cf8-8c6c-f3e5bffbf72b-kube-api-access-fpkqr\") pod \"cec8117c-77b5-4cf8-8c6c-f3e5bffbf72b\" (UID: \"cec8117c-77b5-4cf8-8c6c-f3e5bffbf72b\") " Nov 03 02:56:27.344709 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:27.344697 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cec8117c-77b5-4cf8-8c6c-f3e5bffbf72b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cec8117c-77b5-4cf8-8c6c-f3e5bffbf72b-plugins-config-volume\") pod \"cec8117c-77b5-4cf8-8c6c-f3e5bffbf72b\" (UID: \"cec8117c-77b5-4cf8-8c6c-f3e5bffbf72b\") " Nov 03 02:56:27.345046 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:27.345025 5605 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/080ef559-4fcc-46ed-83d2-d9fa1e168792-plugins-config-volume\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:56:27.345046 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:27.345044 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vxv8\" (UniqueName: \"kubernetes.io/projected/080ef559-4fcc-46ed-83d2-d9fa1e168792-kube-api-access-2vxv8\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:56:27.345046 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:27.345036 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cec8117c-77b5-4cf8-8c6c-f3e5bffbf72b-plugins-config-volume" pod "cec8117c-77b5-4cf8-8c6c-f3e5bffbf72b" (UID: "cec8117c-77b5-4cf8-8c6c-f3e5bffbf72b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:56:27.346483 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:27.346463 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cec8117c-77b5-4cf8-8c6c-f3e5bffbf72b-kube-api-access-fpkqr" pod "cec8117c-77b5-4cf8-8c6c-f3e5bffbf72b" (UID: "cec8117c-77b5-4cf8-8c6c-f3e5bffbf72b"). InnerVolumeSpecName "kube-api-access-fpkqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:56:27.446310 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:27.446235 5605 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cec8117c-77b5-4cf8-8c6c-f3e5bffbf72b-plugins-config-volume\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:56:27.446310 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:27.446262 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpkqr\" (UniqueName: \"kubernetes.io/projected/cec8117c-77b5-4cf8-8c6c-f3e5bffbf72b-kube-api-access-fpkqr\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:56:28.093962 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:28.093933 5605 generic.go:356] "Generic (PLEG): container finished" podID="cec8117c-77b5-4cf8-8c6c-f3e5bffbf72b" containerID="936cbadc551e0d220087f8d1d86bffd3ac2ae1e68c249f684c25217c5f604707" exitCode=0 Nov 03 02:56:28.094430 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:28.093990 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e039cb-5bbb449d87-t8dnw" Nov 03 02:56:28.094430 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:28.094012 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e039cb-5bbb449d87-t8dnw" event={"ID":"cec8117c-77b5-4cf8-8c6c-f3e5bffbf72b","Type":"ContainerDied","Data":"936cbadc551e0d220087f8d1d86bffd3ac2ae1e68c249f684c25217c5f604707"} Nov 03 02:56:28.094430 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:28.094069 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e039cb-5bbb449d87-t8dnw" event={"ID":"cec8117c-77b5-4cf8-8c6c-f3e5bffbf72b","Type":"ContainerDied","Data":"f8f5e394fe2a88544b719275c192a163e9546cbec1556a7423eac04dc6bd5d6a"} Nov 03 02:56:28.094430 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:28.094100 5605 scope.go:122] "RemoveContainer" containerID="936cbadc551e0d220087f8d1d86bffd3ac2ae1e68c249f684c25217c5f604707" Nov 03 02:56:28.095785 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:28.095767 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-e039cb-6f47f9bfd7-9zqfn" Nov 03 02:56:28.096007 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:28.095774 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-e039cb-6f47f9bfd7-9zqfn" event={"ID":"080ef559-4fcc-46ed-83d2-d9fa1e168792","Type":"ContainerDied","Data":"04564c6a8d3d80e3f7075ea61a1c7d1e6aa6d2eb5cd0e804d1712050e5be8c68"} Nov 03 02:56:28.105661 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:28.105639 5605 scope.go:122] "RemoveContainer" containerID="936cbadc551e0d220087f8d1d86bffd3ac2ae1e68c249f684c25217c5f604707" Nov 03 02:56:28.105998 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:56:28.105977 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"936cbadc551e0d220087f8d1d86bffd3ac2ae1e68c249f684c25217c5f604707\": container with ID starting with 936cbadc551e0d220087f8d1d86bffd3ac2ae1e68c249f684c25217c5f604707 not found: ID does not exist" containerID="936cbadc551e0d220087f8d1d86bffd3ac2ae1e68c249f684c25217c5f604707" Nov 03 02:56:28.106059 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:28.106005 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"936cbadc551e0d220087f8d1d86bffd3ac2ae1e68c249f684c25217c5f604707"} err="failed to get container status \"936cbadc551e0d220087f8d1d86bffd3ac2ae1e68c249f684c25217c5f604707\": rpc error: code = NotFound desc = could not find container \"936cbadc551e0d220087f8d1d86bffd3ac2ae1e68c249f684c25217c5f604707\": container with ID starting with 936cbadc551e0d220087f8d1d86bffd3ac2ae1e68c249f684c25217c5f604707 not found: ID does not exist" Nov 03 02:56:28.106059 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:28.106021 5605 scope.go:122] "RemoveContainer" containerID="052fb46f0b7a12f70c595a298bb2633a1fba76400d37da8cb2f21b00374d6e82" Nov 03 02:56:28.248637 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:28.248577 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e039cb-5bbb449d87-t8dnw"] Nov 03 02:56:28.295070 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:28.295049 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e039cb-5bbb449d87-t8dnw"] Nov 03 02:56:28.396541 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:28.396512 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e039cb-6f47f9bfd7-9zqfn"] Nov 03 02:56:28.400191 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:28.400170 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e039cb-6f47f9bfd7-9zqfn"] Nov 03 02:56:28.850828 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:28.850800 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="080ef559-4fcc-46ed-83d2-d9fa1e168792" path="/var/lib/kubelet/pods/080ef559-4fcc-46ed-83d2-d9fa1e168792/volumes" Nov 03 02:56:28.851194 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:28.851180 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cec8117c-77b5-4cf8-8c6c-f3e5bffbf72b" path="/var/lib/kubelet/pods/cec8117c-77b5-4cf8-8c6c-f3e5bffbf72b/volumes" Nov 03 02:56:30.845038 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:30.844999 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-747c9f9449-86gct" podUID="41245c80-d885-4a8c-9ce5-24e59ed83704" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:56:34.924423 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:34.924387 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:56:40.845071 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:40.844995 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-747c9f9449-86gct" podUID="41245c80-d885-4a8c-9ce5-24e59ed83704" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:56:45.230486 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:45.230455 5605 generic.go:356] "Generic (PLEG): container finished" podID="6f7abd33-f094-4091-b738-d77e72cfbf05" containerID="ce55801f30094425de06e8c392a6eb1aecd0adbe74b9ee022bbf05a76d7b5f3b" exitCode=0 Nov 03 02:56:45.230950 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:45.230528 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7" event={"ID":"6f7abd33-f094-4091-b738-d77e72cfbf05","Type":"ContainerDied","Data":"ce55801f30094425de06e8c392a6eb1aecd0adbe74b9ee022bbf05a76d7b5f3b"} Nov 03 02:56:45.230950 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:45.230911 5605 scope.go:122] "RemoveContainer" containerID="ce55801f30094425de06e8c392a6eb1aecd0adbe74b9ee022bbf05a76d7b5f3b" Nov 03 02:56:45.232487 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:45.232460 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:56:45.232588 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:45.232545 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:56:45.232588 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:45.232581 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:56:45.375594 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:45.375490 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7" Nov 03 02:56:46.238446 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:46.238409 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7" event={"ID":"6f7abd33-f094-4091-b738-d77e72cfbf05","Type":"ContainerStarted","Data":"4e58ab496ccc37268cfa0e6186642c487313a315eb113d311fa56000561fd062"} Nov 03 02:56:46.449766 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:46.449726 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7" Nov 03 02:56:46.450788 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:46.450755 5605 patch_prober.go:29] interesting pod/e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body= Nov 03 02:56:46.450952 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:46.450816 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7" podUID="6f7abd33-f094-4091-b738-d77e72cfbf05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" Nov 03 02:56:47.449438 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:47.449406 5605 patch_prober.go:29] interesting pod/e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body= Nov 03 02:56:47.449894 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:47.449462 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7" podUID="6f7abd33-f094-4091-b738-d77e72cfbf05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" Nov 03 02:56:48.449182 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:48.449149 5605 patch_prober.go:29] interesting pod/e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body= Nov 03 02:56:48.449392 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:48.449207 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7" podUID="6f7abd33-f094-4091-b738-d77e72cfbf05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" Nov 03 02:56:49.449563 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:49.449532 5605 patch_prober.go:29] interesting pod/e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body= Nov 03 02:56:49.449965 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:49.449580 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7" podUID="6f7abd33-f094-4091-b738-d77e72cfbf05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" Nov 03 02:56:50.449377 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:50.449321 5605 patch_prober.go:29] interesting pod/e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body= Nov 03 02:56:50.449558 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:50.449399 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7" podUID="6f7abd33-f094-4091-b738-d77e72cfbf05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" Nov 03 02:56:50.844976 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:50.844935 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-747c9f9449-86gct" podUID="41245c80-d885-4a8c-9ce5-24e59ed83704" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:56:51.453985 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:51.453942 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7" Nov 03 02:56:51.454470 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:51.454449 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7" Nov 03 02:56:51.455130 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:51.455109 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7" Nov 03 02:56:52.045211 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:52.045180 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-089846b2-prq2v"] Nov 03 02:56:52.047648 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:52.047622 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-089846b2-prq2v"] Nov 03 02:56:52.850648 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:56:52.850615 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c1f51a2-c981-409a-9af7-4d08d952385b" path="/var/lib/kubelet/pods/0c1f51a2-c981-409a-9af7-4d08d952385b/volumes" Nov 03 02:56:53.913271 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:56:53.913233 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:56:58.773183 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:56:58.773154 5605 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-50a3cab9906d: secret "agc-cred-50a3cab9906d" not found Nov 03 02:56:58.773183 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:56:58.773177 5605 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-f20e06ca5b75: secret "agc-cred-f20e06ca5b75" not found Nov 03 02:56:58.773183 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:56:58.773186 5605 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-68686f579-ntp6g: [secret "agc-cred-50a3cab9906d" not found, secret "agc-cred-f20e06ca5b75" not found] Nov 03 02:56:58.773713 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:56:58.773240 5605 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d89cb81e-1adf-4e6c-82ea-dde66712aab7-provider-credentials podName:d89cb81e-1adf-4e6c-82ea-dde66712aab7 nodeName:}" failed. No retries permitted until 2026-11-03 02:59:00.773227882 +0000 UTC m=+4018.032426033 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d89cb81e-1adf-4e6c-82ea-dde66712aab7-provider-credentials") pod "payload-processing-external-model-68686f579-ntp6g" (UID: "d89cb81e-1adf-4e6c-82ea-dde66712aab7") : [secret "agc-cred-50a3cab9906d" not found, secret "agc-cred-f20e06ca5b75" not found] Nov 03 02:57:00.844835 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:00.844801 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-747c9f9449-86gct" podUID="41245c80-d885-4a8c-9ce5-24e59ed83704" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:57:08.890607 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:08.890571 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:57:10.844990 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:10.844950 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-747c9f9449-86gct" podUID="41245c80-d885-4a8c-9ce5-24e59ed83704" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:57:20.487487 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:20.487456 5605 generic.go:356] "Generic (PLEG): container finished" podID="c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4" containerID="5e3619944c4f5eccfe047d840a9ab222a57143857bc0e30dce9dbb5339137dee" exitCode=0 Nov 03 02:57:20.487915 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:20.487501 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk" event={"ID":"c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4","Type":"ContainerDied","Data":"5e3619944c4f5eccfe047d840a9ab222a57143857bc0e30dce9dbb5339137dee"} Nov 03 02:57:20.487915 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:20.487823 5605 scope.go:122] "RemoveContainer" containerID="5e3619944c4f5eccfe047d840a9ab222a57143857bc0e30dce9dbb5339137dee" Nov 03 02:57:20.489352 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:20.489323 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:57:20.489428 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:20.489408 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:57:20.489464 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:20.489441 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:57:20.844461 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:20.844424 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-747c9f9449-86gct" podUID="41245c80-d885-4a8c-9ce5-24e59ed83704" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:57:21.495678 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:21.495645 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk" event={"ID":"c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4","Type":"ContainerStarted","Data":"b6753043ee27ccb709b8133d79d7736d3beaac9a305d6a08af2bb2db384619c5"} Nov 03 02:57:21.610591 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:21.610553 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk" Nov 03 02:57:21.610816 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:21.610795 5605 patch_prober.go:29] interesting pod/e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Nov 03 02:57:21.610915 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:21.610844 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk" podUID="c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Nov 03 02:57:22.610868 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:22.610833 5605 patch_prober.go:29] interesting pod/e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Nov 03 02:57:22.611347 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:22.610890 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk" podUID="c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Nov 03 02:57:23.611046 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:23.611007 5605 patch_prober.go:29] interesting pod/e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Nov 03 02:57:23.611605 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:23.611082 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk" podUID="c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Nov 03 02:57:23.914673 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:57:23.914583 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:57:24.611657 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:24.611625 5605 patch_prober.go:29] interesting pod/e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Nov 03 02:57:24.612061 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:24.611686 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk" podUID="c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Nov 03 02:57:25.611591 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:25.611556 5605 patch_prober.go:29] interesting pod/e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Nov 03 02:57:25.611770 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:25.611604 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk" podUID="c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Nov 03 02:57:26.611227 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:26.611185 5605 patch_prober.go:29] interesting pod/e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Nov 03 02:57:26.611467 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:26.611251 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk" podUID="c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Nov 03 02:57:27.591447 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:27.591417 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk" Nov 03 02:57:27.616765 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:27.616736 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk" Nov 03 02:57:28.539247 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:28.539217 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk" Nov 03 02:57:30.844313 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:30.844277 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-747c9f9449-86gct" podUID="41245c80-d885-4a8c-9ce5-24e59ed83704" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:57:40.844731 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:40.844689 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-747c9f9449-86gct" podUID="41245c80-d885-4a8c-9ce5-24e59ed83704" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:57:42.818449 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:42.818410 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r"] Nov 03 02:57:42.818886 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:42.818609 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r" podUID="53640730-1899-46e3-95de-b5a502efbd89" containerName="istio-proxy" containerID="cri-o://47f167d90555678f6c66fe37734676c52a2dd9a982481e6ef50ec6ee13fee4f9" gracePeriod=30 Nov 03 02:57:44.611828 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:44.611798 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:57:48.080412 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:48.080332 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r" Nov 03 02:57:48.135880 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:48.135853 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53640730-1899-46e3-95de-b5a502efbd89-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53640730-1899-46e3-95de-b5a502efbd89-credential-socket\") pod \"53640730-1899-46e3-95de-b5a502efbd89\" (UID: \"53640730-1899-46e3-95de-b5a502efbd89\") " Nov 03 02:57:48.136014 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:48.135901 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53640730-1899-46e3-95de-b5a502efbd89-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53640730-1899-46e3-95de-b5a502efbd89-workload-socket\") pod \"53640730-1899-46e3-95de-b5a502efbd89\" (UID: \"53640730-1899-46e3-95de-b5a502efbd89\") " Nov 03 02:57:48.136014 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:48.135937 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53640730-1899-46e3-95de-b5a502efbd89-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53640730-1899-46e3-95de-b5a502efbd89-istio-envoy\") pod \"53640730-1899-46e3-95de-b5a502efbd89\" (UID: \"53640730-1899-46e3-95de-b5a502efbd89\") " Nov 03 02:57:48.136014 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:48.135964 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53640730-1899-46e3-95de-b5a502efbd89-kube-api-access-6xmhj\" (UniqueName: \"kubernetes.io/projected/53640730-1899-46e3-95de-b5a502efbd89-kube-api-access-6xmhj\") pod \"53640730-1899-46e3-95de-b5a502efbd89\" (UID: \"53640730-1899-46e3-95de-b5a502efbd89\") " Nov 03 02:57:48.136190 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:48.136086 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53640730-1899-46e3-95de-b5a502efbd89-credential-socket" pod "53640730-1899-46e3-95de-b5a502efbd89" (UID: "53640730-1899-46e3-95de-b5a502efbd89"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:57:48.136190 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:48.136132 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/53640730-1899-46e3-95de-b5a502efbd89-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53640730-1899-46e3-95de-b5a502efbd89-istio-podinfo\") pod \"53640730-1899-46e3-95de-b5a502efbd89\" (UID: \"53640730-1899-46e3-95de-b5a502efbd89\") " Nov 03 02:57:48.136190 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:48.136178 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53640730-1899-46e3-95de-b5a502efbd89-istio-token\" (UniqueName: \"kubernetes.io/projected/53640730-1899-46e3-95de-b5a502efbd89-istio-token\") pod \"53640730-1899-46e3-95de-b5a502efbd89\" (UID: \"53640730-1899-46e3-95de-b5a502efbd89\") " Nov 03 02:57:48.136407 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:48.136198 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53640730-1899-46e3-95de-b5a502efbd89-workload-socket" pod "53640730-1899-46e3-95de-b5a502efbd89" (UID: "53640730-1899-46e3-95de-b5a502efbd89"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:57:48.136407 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:48.136223 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53640730-1899-46e3-95de-b5a502efbd89-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53640730-1899-46e3-95de-b5a502efbd89-istiod-ca-cert\") pod \"53640730-1899-46e3-95de-b5a502efbd89\" (UID: \"53640730-1899-46e3-95de-b5a502efbd89\") " Nov 03 02:57:48.136407 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:48.136252 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53640730-1899-46e3-95de-b5a502efbd89-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53640730-1899-46e3-95de-b5a502efbd89-workload-certs\") pod \"53640730-1899-46e3-95de-b5a502efbd89\" (UID: \"53640730-1899-46e3-95de-b5a502efbd89\") " Nov 03 02:57:48.136581 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:48.136413 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53640730-1899-46e3-95de-b5a502efbd89-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53640730-1899-46e3-95de-b5a502efbd89-istio-data\") pod \"53640730-1899-46e3-95de-b5a502efbd89\" (UID: \"53640730-1899-46e3-95de-b5a502efbd89\") " Nov 03 02:57:48.136670 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:48.136614 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53640730-1899-46e3-95de-b5a502efbd89-workload-certs" pod "53640730-1899-46e3-95de-b5a502efbd89" (UID: "53640730-1899-46e3-95de-b5a502efbd89"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:57:48.136731 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:48.136680 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53640730-1899-46e3-95de-b5a502efbd89-istio-data" pod "53640730-1899-46e3-95de-b5a502efbd89" (UID: "53640730-1899-46e3-95de-b5a502efbd89"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:57:48.136731 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:48.136694 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53640730-1899-46e3-95de-b5a502efbd89-istiod-ca-cert" pod "53640730-1899-46e3-95de-b5a502efbd89" (UID: "53640730-1899-46e3-95de-b5a502efbd89"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:57:48.137059 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:48.137033 5605 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53640730-1899-46e3-95de-b5a502efbd89-istiod-ca-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:57:48.137146 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:48.137062 5605 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53640730-1899-46e3-95de-b5a502efbd89-workload-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:57:48.137146 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:48.137078 5605 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53640730-1899-46e3-95de-b5a502efbd89-istio-data\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:57:48.137146 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:48.137093 5605 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53640730-1899-46e3-95de-b5a502efbd89-credential-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:57:48.137146 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:48.137108 5605 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53640730-1899-46e3-95de-b5a502efbd89-workload-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:57:48.138216 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:48.138192 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53640730-1899-46e3-95de-b5a502efbd89-istio-envoy" pod "53640730-1899-46e3-95de-b5a502efbd89" (UID: "53640730-1899-46e3-95de-b5a502efbd89"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:57:48.138325 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:48.138216 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53640730-1899-46e3-95de-b5a502efbd89-istio-token" pod "53640730-1899-46e3-95de-b5a502efbd89" (UID: "53640730-1899-46e3-95de-b5a502efbd89"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:57:48.138325 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:48.138249 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53640730-1899-46e3-95de-b5a502efbd89-kube-api-access-6xmhj" pod "53640730-1899-46e3-95de-b5a502efbd89" (UID: "53640730-1899-46e3-95de-b5a502efbd89"). InnerVolumeSpecName "kube-api-access-6xmhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:57:48.138434 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:48.138347 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/53640730-1899-46e3-95de-b5a502efbd89-istio-podinfo" pod "53640730-1899-46e3-95de-b5a502efbd89" (UID: "53640730-1899-46e3-95de-b5a502efbd89"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:57:48.238673 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:48.238645 5605 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53640730-1899-46e3-95de-b5a502efbd89-istio-podinfo\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:57:48.238673 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:48.238668 5605 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53640730-1899-46e3-95de-b5a502efbd89-istio-token\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:57:48.238673 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:48.238679 5605 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53640730-1899-46e3-95de-b5a502efbd89-istio-envoy\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:57:48.238946 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:48.238689 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xmhj\" (UniqueName: \"kubernetes.io/projected/53640730-1899-46e3-95de-b5a502efbd89-kube-api-access-6xmhj\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:57:48.666533 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:48.666499 5605 generic.go:356] "Generic (PLEG): container finished" podID="53640730-1899-46e3-95de-b5a502efbd89" containerID="47f167d90555678f6c66fe37734676c52a2dd9a982481e6ef50ec6ee13fee4f9" exitCode=0 Nov 03 02:57:48.666717 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:48.666543 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r" event={"ID":"53640730-1899-46e3-95de-b5a502efbd89","Type":"ContainerDied","Data":"47f167d90555678f6c66fe37734676c52a2dd9a982481e6ef50ec6ee13fee4f9"} Nov 03 02:57:48.666717 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:48.666546 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r" Nov 03 02:57:48.666717 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:48.666586 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r" event={"ID":"53640730-1899-46e3-95de-b5a502efbd89","Type":"ContainerDied","Data":"6952053cc83242f3c820ec180b75df0848429b4da7a6b93b7e4b898d72a3658f"} Nov 03 02:57:48.666717 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:48.666609 5605 scope.go:122] "RemoveContainer" containerID="47f167d90555678f6c66fe37734676c52a2dd9a982481e6ef50ec6ee13fee4f9" Nov 03 02:57:48.677408 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:48.677382 5605 scope.go:122] "RemoveContainer" containerID="0f00a946e3edd3ee50f79cac07a83974f97e755c7cd8aaaf9ac5f719a06c3964" Nov 03 02:57:48.689617 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:48.687110 5605 scope.go:122] "RemoveContainer" containerID="47f167d90555678f6c66fe37734676c52a2dd9a982481e6ef50ec6ee13fee4f9" Nov 03 02:57:48.689617 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:57:48.688574 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47f167d90555678f6c66fe37734676c52a2dd9a982481e6ef50ec6ee13fee4f9\": container with ID starting with 47f167d90555678f6c66fe37734676c52a2dd9a982481e6ef50ec6ee13fee4f9 not found: ID does not exist" containerID="47f167d90555678f6c66fe37734676c52a2dd9a982481e6ef50ec6ee13fee4f9" Nov 03 02:57:48.689617 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:48.688611 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47f167d90555678f6c66fe37734676c52a2dd9a982481e6ef50ec6ee13fee4f9"} err="failed to get container status \"47f167d90555678f6c66fe37734676c52a2dd9a982481e6ef50ec6ee13fee4f9\": rpc error: code = NotFound desc = could not find container \"47f167d90555678f6c66fe37734676c52a2dd9a982481e6ef50ec6ee13fee4f9\": container with ID starting with 47f167d90555678f6c66fe37734676c52a2dd9a982481e6ef50ec6ee13fee4f9 not found: ID does not exist" Nov 03 02:57:48.689617 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:48.688635 5605 scope.go:122] "RemoveContainer" containerID="0f00a946e3edd3ee50f79cac07a83974f97e755c7cd8aaaf9ac5f719a06c3964" Nov 03 02:57:48.689617 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:57:48.689600 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f00a946e3edd3ee50f79cac07a83974f97e755c7cd8aaaf9ac5f719a06c3964\": container with ID starting with 0f00a946e3edd3ee50f79cac07a83974f97e755c7cd8aaaf9ac5f719a06c3964 not found: ID does not exist" containerID="0f00a946e3edd3ee50f79cac07a83974f97e755c7cd8aaaf9ac5f719a06c3964" Nov 03 02:57:48.689860 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:48.689633 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f00a946e3edd3ee50f79cac07a83974f97e755c7cd8aaaf9ac5f719a06c3964"} err="failed to get container status \"0f00a946e3edd3ee50f79cac07a83974f97e755c7cd8aaaf9ac5f719a06c3964\": rpc error: code = NotFound desc = could not find container \"0f00a946e3edd3ee50f79cac07a83974f97e755c7cd8aaaf9ac5f719a06c3964\": container with ID starting with 0f00a946e3edd3ee50f79cac07a83974f97e755c7cd8aaaf9ac5f719a06c3964 not found: ID does not exist" Nov 03 02:57:48.822901 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:48.822876 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r"] Nov 03 02:57:48.825336 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:48.825315 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r"] Nov 03 02:57:48.850853 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:48.850830 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53640730-1899-46e3-95de-b5a502efbd89" path="/var/lib/kubelet/pods/53640730-1899-46e3-95de-b5a502efbd89/volumes" Nov 03 02:57:49.076294 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:49.076266 5605 patch_prober.go:29] interesting pod/e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:57:49.076464 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:49.076332 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-e039cb-openshift-default-85656c5b69-j4d2r" podUID="53640730-1899-46e3-95de-b5a502efbd89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:57:50.558775 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:50.558742 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-dc14f6-94dd86c89-bqhhz"] Nov 03 02:57:50.663611 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:50.663583 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-dc14f6-85cdc965cf-7v5lt"] Nov 03 02:57:50.845159 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:50.845074 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-747c9f9449-86gct" podUID="41245c80-d885-4a8c-9ce5-24e59ed83704" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:57:53.010966 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:53.010926 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1c43e9-76f6b754d7-mr225"] Nov 03 02:57:53.134867 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:53.134831 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1c43e9-58dfd86775-b7nrz"] Nov 03 02:57:53.915658 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:57:53.915385 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:57:55.559949 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:55.559914 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-dc14f6-94dd86c89-bqhhz" podUID="29a7b266-20d5-4fe6-8a0e-c57ef087f0bb" containerName="payload-processing" containerID="cri-o://c8a2fd39f03a922ab1ebaecabaa77ae3daaafe53feffb0a2efacbbcc2de324df" gracePeriod=25 Nov 03 02:57:55.667815 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:55.667765 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dc14f6-85cdc965cf-7v5lt" podUID="874c955f-0655-4260-8110-6509aef377ca" containerName="payload-pre-processing" containerID="cri-o://e0dcdeaa117df17565d95bb3747973defbeda34263fad5b0ee4169c437aa8949" gracePeriod=25 Nov 03 02:57:55.711298 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:55.711263 5605 generic.go:356] "Generic (PLEG): container finished" podID="29a7b266-20d5-4fe6-8a0e-c57ef087f0bb" containerID="c8a2fd39f03a922ab1ebaecabaa77ae3daaafe53feffb0a2efacbbcc2de324df" exitCode=0 Nov 03 02:57:55.711298 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:55.711285 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-dc14f6-94dd86c89-bqhhz" event={"ID":"29a7b266-20d5-4fe6-8a0e-c57ef087f0bb","Type":"ContainerDied","Data":"c8a2fd39f03a922ab1ebaecabaa77ae3daaafe53feffb0a2efacbbcc2de324df"} Nov 03 02:57:55.827979 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:55.827902 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-dc14f6-94dd86c89-bqhhz" Nov 03 02:57:55.925870 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:55.925848 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dc14f6-85cdc965cf-7v5lt" Nov 03 02:57:55.934130 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:55.934112 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29a7b266-20d5-4fe6-8a0e-c57ef087f0bb-kube-api-access-6z7w4\" (UniqueName: \"kubernetes.io/projected/29a7b266-20d5-4fe6-8a0e-c57ef087f0bb-kube-api-access-6z7w4\") pod \"29a7b266-20d5-4fe6-8a0e-c57ef087f0bb\" (UID: \"29a7b266-20d5-4fe6-8a0e-c57ef087f0bb\") " Nov 03 02:57:55.934213 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:55.934197 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29a7b266-20d5-4fe6-8a0e-c57ef087f0bb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29a7b266-20d5-4fe6-8a0e-c57ef087f0bb-plugins-config-volume\") pod \"29a7b266-20d5-4fe6-8a0e-c57ef087f0bb\" (UID: \"29a7b266-20d5-4fe6-8a0e-c57ef087f0bb\") " Nov 03 02:57:55.934550 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:55.934527 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29a7b266-20d5-4fe6-8a0e-c57ef087f0bb-plugins-config-volume" pod "29a7b266-20d5-4fe6-8a0e-c57ef087f0bb" (UID: "29a7b266-20d5-4fe6-8a0e-c57ef087f0bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:57:55.934794 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:55.934776 5605 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29a7b266-20d5-4fe6-8a0e-c57ef087f0bb-plugins-config-volume\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:57:55.936424 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:55.936400 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29a7b266-20d5-4fe6-8a0e-c57ef087f0bb-kube-api-access-6z7w4" pod "29a7b266-20d5-4fe6-8a0e-c57ef087f0bb" (UID: "29a7b266-20d5-4fe6-8a0e-c57ef087f0bb"). InnerVolumeSpecName "kube-api-access-6z7w4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:57:56.035625 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:56.035593 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/874c955f-0655-4260-8110-6509aef377ca-kube-api-access-7pjqn\" (UniqueName: \"kubernetes.io/projected/874c955f-0655-4260-8110-6509aef377ca-kube-api-access-7pjqn\") pod \"874c955f-0655-4260-8110-6509aef377ca\" (UID: \"874c955f-0655-4260-8110-6509aef377ca\") " Nov 03 02:57:56.035791 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:56.035776 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/874c955f-0655-4260-8110-6509aef377ca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/874c955f-0655-4260-8110-6509aef377ca-plugins-config-volume\") pod \"874c955f-0655-4260-8110-6509aef377ca\" (UID: \"874c955f-0655-4260-8110-6509aef377ca\") " Nov 03 02:57:56.036118 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:56.036087 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/874c955f-0655-4260-8110-6509aef377ca-plugins-config-volume" pod "874c955f-0655-4260-8110-6509aef377ca" (UID: "874c955f-0655-4260-8110-6509aef377ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:57:56.036236 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:56.036221 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z7w4\" (UniqueName: \"kubernetes.io/projected/29a7b266-20d5-4fe6-8a0e-c57ef087f0bb-kube-api-access-6z7w4\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:57:56.036236 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:56.036233 5605 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/874c955f-0655-4260-8110-6509aef377ca-plugins-config-volume\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:57:56.037745 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:56.037726 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/874c955f-0655-4260-8110-6509aef377ca-kube-api-access-7pjqn" pod "874c955f-0655-4260-8110-6509aef377ca" (UID: "874c955f-0655-4260-8110-6509aef377ca"). InnerVolumeSpecName "kube-api-access-7pjqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:57:56.137455 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:56.137351 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pjqn\" (UniqueName: \"kubernetes.io/projected/874c955f-0655-4260-8110-6509aef377ca-kube-api-access-7pjqn\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:57:56.719137 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:56.719108 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-dc14f6-94dd86c89-bqhhz" Nov 03 02:57:56.719623 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:56.719144 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-dc14f6-94dd86c89-bqhhz" event={"ID":"29a7b266-20d5-4fe6-8a0e-c57ef087f0bb","Type":"ContainerDied","Data":"8b25e6eb2a9cc403a60877204df68c41587f33c91db2476686286f572b7d35a9"} Nov 03 02:57:56.719623 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:56.719199 5605 scope.go:122] "RemoveContainer" containerID="c8a2fd39f03a922ab1ebaecabaa77ae3daaafe53feffb0a2efacbbcc2de324df" Nov 03 02:57:56.720535 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:56.720510 5605 generic.go:356] "Generic (PLEG): container finished" podID="874c955f-0655-4260-8110-6509aef377ca" containerID="e0dcdeaa117df17565d95bb3747973defbeda34263fad5b0ee4169c437aa8949" exitCode=0 Nov 03 02:57:56.720672 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:56.720540 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dc14f6-85cdc965cf-7v5lt" event={"ID":"874c955f-0655-4260-8110-6509aef377ca","Type":"ContainerDied","Data":"e0dcdeaa117df17565d95bb3747973defbeda34263fad5b0ee4169c437aa8949"} Nov 03 02:57:56.720672 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:56.720567 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dc14f6-85cdc965cf-7v5lt" Nov 03 02:57:56.720672 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:56.720584 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dc14f6-85cdc965cf-7v5lt" event={"ID":"874c955f-0655-4260-8110-6509aef377ca","Type":"ContainerDied","Data":"c3199b7f844d13f2cdd7ad3e588158a43fbb50b5cd92322a0561c63fd7688712"} Nov 03 02:57:56.730121 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:56.730100 5605 scope.go:122] "RemoveContainer" containerID="e0dcdeaa117df17565d95bb3747973defbeda34263fad5b0ee4169c437aa8949" Nov 03 02:57:56.738974 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:56.738953 5605 scope.go:122] "RemoveContainer" containerID="e0dcdeaa117df17565d95bb3747973defbeda34263fad5b0ee4169c437aa8949" Nov 03 02:57:56.739235 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:57:56.739215 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0dcdeaa117df17565d95bb3747973defbeda34263fad5b0ee4169c437aa8949\": container with ID starting with e0dcdeaa117df17565d95bb3747973defbeda34263fad5b0ee4169c437aa8949 not found: ID does not exist" containerID="e0dcdeaa117df17565d95bb3747973defbeda34263fad5b0ee4169c437aa8949" Nov 03 02:57:56.739293 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:56.739245 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0dcdeaa117df17565d95bb3747973defbeda34263fad5b0ee4169c437aa8949"} err="failed to get container status \"e0dcdeaa117df17565d95bb3747973defbeda34263fad5b0ee4169c437aa8949\": rpc error: code = NotFound desc = could not find container \"e0dcdeaa117df17565d95bb3747973defbeda34263fad5b0ee4169c437aa8949\": container with ID starting with e0dcdeaa117df17565d95bb3747973defbeda34263fad5b0ee4169c437aa8949 not found: ID does not exist" Nov 03 02:57:56.872916 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:56.872844 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-dc14f6-94dd86c89-bqhhz"] Nov 03 02:57:56.916801 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:56.916778 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-dc14f6-94dd86c89-bqhhz"] Nov 03 02:57:57.019566 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:57.019540 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-dc14f6-85cdc965cf-7v5lt"] Nov 03 02:57:57.022545 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:57.022524 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-dc14f6-85cdc965cf-7v5lt"] Nov 03 02:57:58.012053 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:58.012016 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-1c43e9-76f6b754d7-mr225" podUID="e82f0187-0392-499b-ac69-0e7e03eebbf7" containerName="payload-processing" containerID="cri-o://22165f79452f73127f1795d2442ff14df962786c552c1161143b2e0d54e07906" gracePeriod=25 Nov 03 02:57:58.135316 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:58.135269 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c43e9-58dfd86775-b7nrz" podUID="ad6cd805-ef8c-4c3a-a63d-52a044bfe9ce" containerName="payload-pre-processing" containerID="cri-o://70c3141a92fa09bbeac1804b8a0468e8cb0554545d43408eb8a5200e76ceef8b" gracePeriod=25 Nov 03 02:57:58.274582 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:58.274514 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1c43e9-76f6b754d7-mr225" Nov 03 02:57:58.361194 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:58.361165 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e82f0187-0392-499b-ac69-0e7e03eebbf7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e82f0187-0392-499b-ac69-0e7e03eebbf7-plugins-config-volume\") pod \"e82f0187-0392-499b-ac69-0e7e03eebbf7\" (UID: \"e82f0187-0392-499b-ac69-0e7e03eebbf7\") " Nov 03 02:57:58.361415 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:58.361399 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e82f0187-0392-499b-ac69-0e7e03eebbf7-kube-api-access-td284\" (UniqueName: \"kubernetes.io/projected/e82f0187-0392-499b-ac69-0e7e03eebbf7-kube-api-access-td284\") pod \"e82f0187-0392-499b-ac69-0e7e03eebbf7\" (UID: \"e82f0187-0392-499b-ac69-0e7e03eebbf7\") " Nov 03 02:57:58.361620 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:58.361593 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e82f0187-0392-499b-ac69-0e7e03eebbf7-plugins-config-volume" pod "e82f0187-0392-499b-ac69-0e7e03eebbf7" (UID: "e82f0187-0392-499b-ac69-0e7e03eebbf7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:57:58.361878 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:58.361858 5605 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e82f0187-0392-499b-ac69-0e7e03eebbf7-plugins-config-volume\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:57:58.364010 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:58.363980 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e82f0187-0392-499b-ac69-0e7e03eebbf7-kube-api-access-td284" pod "e82f0187-0392-499b-ac69-0e7e03eebbf7" (UID: "e82f0187-0392-499b-ac69-0e7e03eebbf7"). InnerVolumeSpecName "kube-api-access-td284". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:57:58.382774 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:58.382750 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c43e9-58dfd86775-b7nrz" Nov 03 02:57:58.462684 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:58.462658 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad6cd805-ef8c-4c3a-a63d-52a044bfe9ce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad6cd805-ef8c-4c3a-a63d-52a044bfe9ce-plugins-config-volume\") pod \"ad6cd805-ef8c-4c3a-a63d-52a044bfe9ce\" (UID: \"ad6cd805-ef8c-4c3a-a63d-52a044bfe9ce\") " Nov 03 02:57:58.462798 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:58.462705 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad6cd805-ef8c-4c3a-a63d-52a044bfe9ce-kube-api-access-k8sgl\" (UniqueName: \"kubernetes.io/projected/ad6cd805-ef8c-4c3a-a63d-52a044bfe9ce-kube-api-access-k8sgl\") pod \"ad6cd805-ef8c-4c3a-a63d-52a044bfe9ce\" (UID: \"ad6cd805-ef8c-4c3a-a63d-52a044bfe9ce\") " Nov 03 02:57:58.462996 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:58.462975 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad6cd805-ef8c-4c3a-a63d-52a044bfe9ce-plugins-config-volume" pod "ad6cd805-ef8c-4c3a-a63d-52a044bfe9ce" (UID: "ad6cd805-ef8c-4c3a-a63d-52a044bfe9ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:57:58.463307 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:58.463291 5605 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad6cd805-ef8c-4c3a-a63d-52a044bfe9ce-plugins-config-volume\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:57:58.463343 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:58.463311 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td284\" (UniqueName: \"kubernetes.io/projected/e82f0187-0392-499b-ac69-0e7e03eebbf7-kube-api-access-td284\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:57:58.464769 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:58.464752 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad6cd805-ef8c-4c3a-a63d-52a044bfe9ce-kube-api-access-k8sgl" pod "ad6cd805-ef8c-4c3a-a63d-52a044bfe9ce" (UID: "ad6cd805-ef8c-4c3a-a63d-52a044bfe9ce"). InnerVolumeSpecName "kube-api-access-k8sgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:57:58.564813 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:58.564787 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8sgl\" (UniqueName: \"kubernetes.io/projected/ad6cd805-ef8c-4c3a-a63d-52a044bfe9ce-kube-api-access-k8sgl\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:57:58.735464 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:58.735439 5605 generic.go:356] "Generic (PLEG): container finished" podID="ad6cd805-ef8c-4c3a-a63d-52a044bfe9ce" containerID="70c3141a92fa09bbeac1804b8a0468e8cb0554545d43408eb8a5200e76ceef8b" exitCode=0 Nov 03 02:57:58.735597 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:58.735484 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c43e9-58dfd86775-b7nrz" Nov 03 02:57:58.735597 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:58.735528 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c43e9-58dfd86775-b7nrz" event={"ID":"ad6cd805-ef8c-4c3a-a63d-52a044bfe9ce","Type":"ContainerDied","Data":"70c3141a92fa09bbeac1804b8a0468e8cb0554545d43408eb8a5200e76ceef8b"} Nov 03 02:57:58.735678 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:58.735619 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c43e9-58dfd86775-b7nrz" event={"ID":"ad6cd805-ef8c-4c3a-a63d-52a044bfe9ce","Type":"ContainerDied","Data":"e9210919df869cca2c1796b79083dd2ae1b214f19c218139cebe17bd1ce6d659"} Nov 03 02:57:58.735678 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:58.735655 5605 scope.go:122] "RemoveContainer" containerID="70c3141a92fa09bbeac1804b8a0468e8cb0554545d43408eb8a5200e76ceef8b" Nov 03 02:57:58.736929 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:58.736906 5605 generic.go:356] "Generic (PLEG): container finished" podID="e82f0187-0392-499b-ac69-0e7e03eebbf7" containerID="22165f79452f73127f1795d2442ff14df962786c552c1161143b2e0d54e07906" exitCode=0 Nov 03 02:57:58.737062 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:58.736951 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1c43e9-76f6b754d7-mr225" Nov 03 02:57:58.737062 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:58.736968 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1c43e9-76f6b754d7-mr225" event={"ID":"e82f0187-0392-499b-ac69-0e7e03eebbf7","Type":"ContainerDied","Data":"22165f79452f73127f1795d2442ff14df962786c552c1161143b2e0d54e07906"} Nov 03 02:57:58.737062 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:58.737000 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1c43e9-76f6b754d7-mr225" event={"ID":"e82f0187-0392-499b-ac69-0e7e03eebbf7","Type":"ContainerDied","Data":"fcd43d9ad6209f0c578854b759862366ab387192bfabaf498fd41a187714fcdd"} Nov 03 02:57:58.745281 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:58.745264 5605 scope.go:122] "RemoveContainer" containerID="70c3141a92fa09bbeac1804b8a0468e8cb0554545d43408eb8a5200e76ceef8b" Nov 03 02:57:58.745584 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:57:58.745562 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70c3141a92fa09bbeac1804b8a0468e8cb0554545d43408eb8a5200e76ceef8b\": container with ID starting with 70c3141a92fa09bbeac1804b8a0468e8cb0554545d43408eb8a5200e76ceef8b not found: ID does not exist" containerID="70c3141a92fa09bbeac1804b8a0468e8cb0554545d43408eb8a5200e76ceef8b" Nov 03 02:57:58.745654 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:58.745594 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70c3141a92fa09bbeac1804b8a0468e8cb0554545d43408eb8a5200e76ceef8b"} err="failed to get container status \"70c3141a92fa09bbeac1804b8a0468e8cb0554545d43408eb8a5200e76ceef8b\": rpc error: code = NotFound desc = could not find container \"70c3141a92fa09bbeac1804b8a0468e8cb0554545d43408eb8a5200e76ceef8b\": container with ID starting with 70c3141a92fa09bbeac1804b8a0468e8cb0554545d43408eb8a5200e76ceef8b not found: ID does not exist" Nov 03 02:57:58.745654 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:58.745610 5605 scope.go:122] "RemoveContainer" containerID="22165f79452f73127f1795d2442ff14df962786c552c1161143b2e0d54e07906" Nov 03 02:57:58.754190 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:58.754173 5605 scope.go:122] "RemoveContainer" containerID="22165f79452f73127f1795d2442ff14df962786c552c1161143b2e0d54e07906" Nov 03 02:57:58.754487 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:57:58.754469 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22165f79452f73127f1795d2442ff14df962786c552c1161143b2e0d54e07906\": container with ID starting with 22165f79452f73127f1795d2442ff14df962786c552c1161143b2e0d54e07906 not found: ID does not exist" containerID="22165f79452f73127f1795d2442ff14df962786c552c1161143b2e0d54e07906" Nov 03 02:57:58.754551 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:58.754492 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22165f79452f73127f1795d2442ff14df962786c552c1161143b2e0d54e07906"} err="failed to get container status \"22165f79452f73127f1795d2442ff14df962786c552c1161143b2e0d54e07906\": rpc error: code = NotFound desc = could not find container \"22165f79452f73127f1795d2442ff14df962786c552c1161143b2e0d54e07906\": container with ID starting with 22165f79452f73127f1795d2442ff14df962786c552c1161143b2e0d54e07906 not found: ID does not exist" Nov 03 02:57:58.850552 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:58.850482 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29a7b266-20d5-4fe6-8a0e-c57ef087f0bb" path="/var/lib/kubelet/pods/29a7b266-20d5-4fe6-8a0e-c57ef087f0bb/volumes" Nov 03 02:57:58.850884 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:58.850862 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="874c955f-0655-4260-8110-6509aef377ca" path="/var/lib/kubelet/pods/874c955f-0655-4260-8110-6509aef377ca/volumes" Nov 03 02:57:58.889083 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:58.889058 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1c43e9-58dfd86775-b7nrz"] Nov 03 02:57:58.891382 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:58.891345 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1c43e9-58dfd86775-b7nrz"] Nov 03 02:57:58.992729 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:58.992701 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1c43e9-76f6b754d7-mr225"] Nov 03 02:57:58.997343 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:57:58.997314 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1c43e9-76f6b754d7-mr225"] Nov 03 02:58:00.225477 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:00.225447 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7"] Nov 03 02:58:00.225950 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:00.225636 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7" podUID="6f7abd33-f094-4091-b738-d77e72cfbf05" containerName="istio-proxy" containerID="cri-o://4e58ab496ccc37268cfa0e6186642c487313a315eb113d311fa56000561fd062" gracePeriod=30 Nov 03 02:58:00.844634 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:00.844595 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-747c9f9449-86gct" podUID="41245c80-d885-4a8c-9ce5-24e59ed83704" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:58:00.850304 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:00.850265 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad6cd805-ef8c-4c3a-a63d-52a044bfe9ce" path="/var/lib/kubelet/pods/ad6cd805-ef8c-4c3a-a63d-52a044bfe9ce/volumes" Nov 03 02:58:00.850762 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:00.850747 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e82f0187-0392-499b-ac69-0e7e03eebbf7" path="/var/lib/kubelet/pods/e82f0187-0392-499b-ac69-0e7e03eebbf7/volumes" Nov 03 02:58:04.837055 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:04.837017 5605 scope.go:122] "RemoveContainer" containerID="7641e4a1a1616d73dc084b0c67212e4c6026397094d265b794ff393f82e85a7a" Nov 03 02:58:04.847443 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:04.847424 5605 scope.go:122] "RemoveContainer" containerID="e741815a500582ca12f95cc4543d74c996ba67f32e87f6395531ceeecb8190be" Nov 03 02:58:05.495953 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:05.495889 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7" Nov 03 02:58:05.646989 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:05.646955 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f7abd33-f094-4091-b738-d77e72cfbf05-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f7abd33-f094-4091-b738-d77e72cfbf05-istio-envoy\") pod \"6f7abd33-f094-4091-b738-d77e72cfbf05\" (UID: \"6f7abd33-f094-4091-b738-d77e72cfbf05\") " Nov 03 02:58:05.647138 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:05.647021 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f7abd33-f094-4091-b738-d77e72cfbf05-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f7abd33-f094-4091-b738-d77e72cfbf05-workload-socket\") pod \"6f7abd33-f094-4091-b738-d77e72cfbf05\" (UID: \"6f7abd33-f094-4091-b738-d77e72cfbf05\") " Nov 03 02:58:05.647138 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:05.647084 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f7abd33-f094-4091-b738-d77e72cfbf05-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f7abd33-f094-4091-b738-d77e72cfbf05-istio-data\") pod \"6f7abd33-f094-4091-b738-d77e72cfbf05\" (UID: \"6f7abd33-f094-4091-b738-d77e72cfbf05\") " Nov 03 02:58:05.647138 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:05.647115 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f7abd33-f094-4091-b738-d77e72cfbf05-kube-api-access-4br27\" (UniqueName: \"kubernetes.io/projected/6f7abd33-f094-4091-b738-d77e72cfbf05-kube-api-access-4br27\") pod \"6f7abd33-f094-4091-b738-d77e72cfbf05\" (UID: \"6f7abd33-f094-4091-b738-d77e72cfbf05\") " Nov 03 02:58:05.647263 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:05.647167 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f7abd33-f094-4091-b738-d77e72cfbf05-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f7abd33-f094-4091-b738-d77e72cfbf05-istiod-ca-cert\") pod \"6f7abd33-f094-4091-b738-d77e72cfbf05\" (UID: \"6f7abd33-f094-4091-b738-d77e72cfbf05\") " Nov 03 02:58:05.647263 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:05.647209 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f7abd33-f094-4091-b738-d77e72cfbf05-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f7abd33-f094-4091-b738-d77e72cfbf05-istio-podinfo\") pod \"6f7abd33-f094-4091-b738-d77e72cfbf05\" (UID: \"6f7abd33-f094-4091-b738-d77e72cfbf05\") " Nov 03 02:58:05.647263 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:05.647252 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f7abd33-f094-4091-b738-d77e72cfbf05-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f7abd33-f094-4091-b738-d77e72cfbf05-workload-certs\") pod \"6f7abd33-f094-4091-b738-d77e72cfbf05\" (UID: \"6f7abd33-f094-4091-b738-d77e72cfbf05\") " Nov 03 02:58:05.647447 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:05.647279 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f7abd33-f094-4091-b738-d77e72cfbf05-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f7abd33-f094-4091-b738-d77e72cfbf05-credential-socket\") pod \"6f7abd33-f094-4091-b738-d77e72cfbf05\" (UID: \"6f7abd33-f094-4091-b738-d77e72cfbf05\") " Nov 03 02:58:05.647447 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:05.647308 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f7abd33-f094-4091-b738-d77e72cfbf05-istio-token\" (UniqueName: \"kubernetes.io/projected/6f7abd33-f094-4091-b738-d77e72cfbf05-istio-token\") pod \"6f7abd33-f094-4091-b738-d77e72cfbf05\" (UID: \"6f7abd33-f094-4091-b738-d77e72cfbf05\") " Nov 03 02:58:05.647447 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:05.647311 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f7abd33-f094-4091-b738-d77e72cfbf05-workload-socket" pod "6f7abd33-f094-4091-b738-d77e72cfbf05" (UID: "6f7abd33-f094-4091-b738-d77e72cfbf05"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:58:05.647447 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:05.647428 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f7abd33-f094-4091-b738-d77e72cfbf05-istio-data" pod "6f7abd33-f094-4091-b738-d77e72cfbf05" (UID: "6f7abd33-f094-4091-b738-d77e72cfbf05"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:58:05.647665 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:05.647588 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f7abd33-f094-4091-b738-d77e72cfbf05-credential-socket" pod "6f7abd33-f094-4091-b738-d77e72cfbf05" (UID: "6f7abd33-f094-4091-b738-d77e72cfbf05"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:58:05.647665 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:05.647601 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f7abd33-f094-4091-b738-d77e72cfbf05-workload-certs" pod "6f7abd33-f094-4091-b738-d77e72cfbf05" (UID: "6f7abd33-f094-4091-b738-d77e72cfbf05"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:58:05.647772 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:05.647677 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f7abd33-f094-4091-b738-d77e72cfbf05-istiod-ca-cert" pod "6f7abd33-f094-4091-b738-d77e72cfbf05" (UID: "6f7abd33-f094-4091-b738-d77e72cfbf05"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:58:05.648016 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:05.647994 5605 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f7abd33-f094-4091-b738-d77e72cfbf05-istiod-ca-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:58:05.648103 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:05.648021 5605 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f7abd33-f094-4091-b738-d77e72cfbf05-workload-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:58:05.648103 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:05.648039 5605 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f7abd33-f094-4091-b738-d77e72cfbf05-credential-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:58:05.648103 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:05.648053 5605 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f7abd33-f094-4091-b738-d77e72cfbf05-workload-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:58:05.648103 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:05.648066 5605 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f7abd33-f094-4091-b738-d77e72cfbf05-istio-data\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:58:05.649190 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:05.649162 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f7abd33-f094-4091-b738-d77e72cfbf05-istio-envoy" pod "6f7abd33-f094-4091-b738-d77e72cfbf05" (UID: "6f7abd33-f094-4091-b738-d77e72cfbf05"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:58:05.649453 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:05.649427 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f7abd33-f094-4091-b738-d77e72cfbf05-istio-token" pod "6f7abd33-f094-4091-b738-d77e72cfbf05" (UID: "6f7abd33-f094-4091-b738-d77e72cfbf05"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:58:05.649751 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:05.649732 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f7abd33-f094-4091-b738-d77e72cfbf05-kube-api-access-4br27" pod "6f7abd33-f094-4091-b738-d77e72cfbf05" (UID: "6f7abd33-f094-4091-b738-d77e72cfbf05"). InnerVolumeSpecName "kube-api-access-4br27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:58:05.649821 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:05.649766 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f7abd33-f094-4091-b738-d77e72cfbf05-istio-podinfo" pod "6f7abd33-f094-4091-b738-d77e72cfbf05" (UID: "6f7abd33-f094-4091-b738-d77e72cfbf05"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:58:05.749273 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:05.749199 5605 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f7abd33-f094-4091-b738-d77e72cfbf05-istio-envoy\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:58:05.749273 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:05.749227 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4br27\" (UniqueName: \"kubernetes.io/projected/6f7abd33-f094-4091-b738-d77e72cfbf05-kube-api-access-4br27\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:58:05.749273 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:05.749238 5605 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f7abd33-f094-4091-b738-d77e72cfbf05-istio-podinfo\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:58:05.749273 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:05.749246 5605 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f7abd33-f094-4091-b738-d77e72cfbf05-istio-token\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:58:05.782945 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:05.782914 5605 generic.go:356] "Generic (PLEG): container finished" podID="6f7abd33-f094-4091-b738-d77e72cfbf05" containerID="4e58ab496ccc37268cfa0e6186642c487313a315eb113d311fa56000561fd062" exitCode=0 Nov 03 02:58:05.783068 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:05.782964 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7" Nov 03 02:58:05.783068 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:05.782966 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7" event={"ID":"6f7abd33-f094-4091-b738-d77e72cfbf05","Type":"ContainerDied","Data":"4e58ab496ccc37268cfa0e6186642c487313a315eb113d311fa56000561fd062"} Nov 03 02:58:05.783068 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:05.783021 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7" event={"ID":"6f7abd33-f094-4091-b738-d77e72cfbf05","Type":"ContainerDied","Data":"dae15775eb9d2ee8d71cc6c0d3e19335124938ed09718ef7b9aff3c131db83fc"} Nov 03 02:58:05.783068 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:05.783054 5605 scope.go:122] "RemoveContainer" containerID="4e58ab496ccc37268cfa0e6186642c487313a315eb113d311fa56000561fd062" Nov 03 02:58:05.792988 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:05.792966 5605 scope.go:122] "RemoveContainer" containerID="ce55801f30094425de06e8c392a6eb1aecd0adbe74b9ee022bbf05a76d7b5f3b" Nov 03 02:58:05.801855 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:05.801836 5605 scope.go:122] "RemoveContainer" containerID="4e58ab496ccc37268cfa0e6186642c487313a315eb113d311fa56000561fd062" Nov 03 02:58:05.802114 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:58:05.802096 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e58ab496ccc37268cfa0e6186642c487313a315eb113d311fa56000561fd062\": container with ID starting with 4e58ab496ccc37268cfa0e6186642c487313a315eb113d311fa56000561fd062 not found: ID does not exist" containerID="4e58ab496ccc37268cfa0e6186642c487313a315eb113d311fa56000561fd062" Nov 03 02:58:05.802169 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:05.802121 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e58ab496ccc37268cfa0e6186642c487313a315eb113d311fa56000561fd062"} err="failed to get container status \"4e58ab496ccc37268cfa0e6186642c487313a315eb113d311fa56000561fd062\": rpc error: code = NotFound desc = could not find container \"4e58ab496ccc37268cfa0e6186642c487313a315eb113d311fa56000561fd062\": container with ID starting with 4e58ab496ccc37268cfa0e6186642c487313a315eb113d311fa56000561fd062 not found: ID does not exist" Nov 03 02:58:05.802169 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:05.802139 5605 scope.go:122] "RemoveContainer" containerID="ce55801f30094425de06e8c392a6eb1aecd0adbe74b9ee022bbf05a76d7b5f3b" Nov 03 02:58:05.802413 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:58:05.802395 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce55801f30094425de06e8c392a6eb1aecd0adbe74b9ee022bbf05a76d7b5f3b\": container with ID starting with ce55801f30094425de06e8c392a6eb1aecd0adbe74b9ee022bbf05a76d7b5f3b not found: ID does not exist" containerID="ce55801f30094425de06e8c392a6eb1aecd0adbe74b9ee022bbf05a76d7b5f3b" Nov 03 02:58:05.802468 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:05.802421 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce55801f30094425de06e8c392a6eb1aecd0adbe74b9ee022bbf05a76d7b5f3b"} err="failed to get container status \"ce55801f30094425de06e8c392a6eb1aecd0adbe74b9ee022bbf05a76d7b5f3b\": rpc error: code = NotFound desc = could not find container \"ce55801f30094425de06e8c392a6eb1aecd0adbe74b9ee022bbf05a76d7b5f3b\": container with ID starting with ce55801f30094425de06e8c392a6eb1aecd0adbe74b9ee022bbf05a76d7b5f3b not found: ID does not exist" Nov 03 02:58:05.936900 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:05.936867 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7"] Nov 03 02:58:05.941615 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:05.941586 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-1c43e9-openshift-default-5978fd5b8f-25bf7"] Nov 03 02:58:06.850384 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:06.850332 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f7abd33-f094-4091-b738-d77e72cfbf05" path="/var/lib/kubelet/pods/6f7abd33-f094-4091-b738-d77e72cfbf05/volumes" Nov 03 02:58:10.844461 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:10.844389 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-747c9f9449-86gct" podUID="41245c80-d885-4a8c-9ce5-24e59ed83704" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:58:19.435592 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:19.435551 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-95152a-5d959db47f-w6cgh"] Nov 03 02:58:20.424835 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:20.424798 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:58:20.844969 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:20.844930 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-747c9f9449-86gct" podUID="41245c80-d885-4a8c-9ce5-24e59ed83704" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:58:24.034382 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:58:24.030819 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:58:24.436792 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:24.436747 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-95152a-5d959db47f-w6cgh" podUID="51beebde-3a29-4cc8-8319-a91a990d4755" containerName="payload-processing" containerID="cri-o://f719709d8077c4c13bb83bc5ec51c47231685cb1b57b1d59d2b059184dae9dfe" gracePeriod=25 Nov 03 02:58:24.689907 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:24.689828 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-95152a-5d959db47f-w6cgh" Nov 03 02:58:24.740644 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:24.740608 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51beebde-3a29-4cc8-8319-a91a990d4755-kube-api-access-lscpv\" (UniqueName: \"kubernetes.io/projected/51beebde-3a29-4cc8-8319-a91a990d4755-kube-api-access-lscpv\") pod \"51beebde-3a29-4cc8-8319-a91a990d4755\" (UID: \"51beebde-3a29-4cc8-8319-a91a990d4755\") " Nov 03 02:58:24.740929 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:24.740694 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51beebde-3a29-4cc8-8319-a91a990d4755-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51beebde-3a29-4cc8-8319-a91a990d4755-plugins-config-volume\") pod \"51beebde-3a29-4cc8-8319-a91a990d4755\" (UID: \"51beebde-3a29-4cc8-8319-a91a990d4755\") " Nov 03 02:58:24.741712 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:24.741100 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51beebde-3a29-4cc8-8319-a91a990d4755-plugins-config-volume" pod "51beebde-3a29-4cc8-8319-a91a990d4755" (UID: "51beebde-3a29-4cc8-8319-a91a990d4755"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:58:24.741712 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:24.741297 5605 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51beebde-3a29-4cc8-8319-a91a990d4755-plugins-config-volume\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:58:24.747721 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:24.747681 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51beebde-3a29-4cc8-8319-a91a990d4755-kube-api-access-lscpv" pod "51beebde-3a29-4cc8-8319-a91a990d4755" (UID: "51beebde-3a29-4cc8-8319-a91a990d4755"). InnerVolumeSpecName "kube-api-access-lscpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:58:24.842135 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:24.842092 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lscpv\" (UniqueName: \"kubernetes.io/projected/51beebde-3a29-4cc8-8319-a91a990d4755-kube-api-access-lscpv\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:58:25.013956 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:25.013867 5605 generic.go:356] "Generic (PLEG): container finished" podID="51beebde-3a29-4cc8-8319-a91a990d4755" containerID="f719709d8077c4c13bb83bc5ec51c47231685cb1b57b1d59d2b059184dae9dfe" exitCode=0 Nov 03 02:58:25.013956 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:25.013939 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-95152a-5d959db47f-w6cgh" Nov 03 02:58:25.013956 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:25.013941 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-95152a-5d959db47f-w6cgh" event={"ID":"51beebde-3a29-4cc8-8319-a91a990d4755","Type":"ContainerDied","Data":"f719709d8077c4c13bb83bc5ec51c47231685cb1b57b1d59d2b059184dae9dfe"} Nov 03 02:58:25.014185 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:25.013994 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-95152a-5d959db47f-w6cgh" event={"ID":"51beebde-3a29-4cc8-8319-a91a990d4755","Type":"ContainerDied","Data":"9f154c14d48266548e43df2aceab09e39366535d17e09caccb202888f4eeee93"} Nov 03 02:58:25.014185 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:25.014023 5605 scope.go:122] "RemoveContainer" containerID="f719709d8077c4c13bb83bc5ec51c47231685cb1b57b1d59d2b059184dae9dfe" Nov 03 02:58:25.023426 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:25.023404 5605 scope.go:122] "RemoveContainer" containerID="f719709d8077c4c13bb83bc5ec51c47231685cb1b57b1d59d2b059184dae9dfe" Nov 03 02:58:25.023696 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:58:25.023670 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f719709d8077c4c13bb83bc5ec51c47231685cb1b57b1d59d2b059184dae9dfe\": container with ID starting with f719709d8077c4c13bb83bc5ec51c47231685cb1b57b1d59d2b059184dae9dfe not found: ID does not exist" containerID="f719709d8077c4c13bb83bc5ec51c47231685cb1b57b1d59d2b059184dae9dfe" Nov 03 02:58:25.023758 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:25.023707 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f719709d8077c4c13bb83bc5ec51c47231685cb1b57b1d59d2b059184dae9dfe"} err="failed to get container status \"f719709d8077c4c13bb83bc5ec51c47231685cb1b57b1d59d2b059184dae9dfe\": rpc error: code = NotFound desc = could not find container \"f719709d8077c4c13bb83bc5ec51c47231685cb1b57b1d59d2b059184dae9dfe\": container with ID starting with f719709d8077c4c13bb83bc5ec51c47231685cb1b57b1d59d2b059184dae9dfe not found: ID does not exist" Nov 03 02:58:25.167783 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:25.167744 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-95152a-5d959db47f-w6cgh"] Nov 03 02:58:25.172922 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:25.172897 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-95152a-5d959db47f-w6cgh"] Nov 03 02:58:26.850471 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:26.850439 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51beebde-3a29-4cc8-8319-a91a990d4755" path="/var/lib/kubelet/pods/51beebde-3a29-4cc8-8319-a91a990d4755/volumes" Nov 03 02:58:30.845221 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:30.845175 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-747c9f9449-86gct" podUID="41245c80-d885-4a8c-9ce5-24e59ed83704" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:58:35.325250 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:35.325212 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97dd93e8-kfr9g"] Nov 03 02:58:35.325761 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:35.325746 5605 cpu_manager.go:414] "containerMap: removing container" podUID="ad6cd805-ef8c-4c3a-a63d-52a044bfe9ce" containerName="payload-pre-processing" Nov 03 02:58:35.325761 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:35.325761 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad6cd805-ef8c-4c3a-a63d-52a044bfe9ce" containerName="payload-pre-processing" Nov 03 02:58:35.325905 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:35.325774 5605 cpu_manager.go:414] "containerMap: removing container" podUID="29a7b266-20d5-4fe6-8a0e-c57ef087f0bb" containerName="payload-processing" Nov 03 02:58:35.325905 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:35.325780 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29a7b266-20d5-4fe6-8a0e-c57ef087f0bb" containerName="payload-processing" Nov 03 02:58:35.325905 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:35.325788 5605 cpu_manager.go:414] "containerMap: removing container" podUID="51beebde-3a29-4cc8-8319-a91a990d4755" containerName="payload-processing" Nov 03 02:58:35.325905 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:35.325794 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51beebde-3a29-4cc8-8319-a91a990d4755" containerName="payload-processing" Nov 03 02:58:35.325905 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:35.325805 5605 cpu_manager.go:414] "containerMap: removing container" podUID="cec8117c-77b5-4cf8-8c6c-f3e5bffbf72b" containerName="payload-pre-processing" Nov 03 02:58:35.325905 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:35.325810 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cec8117c-77b5-4cf8-8c6c-f3e5bffbf72b" containerName="payload-pre-processing" Nov 03 02:58:35.325905 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:35.325818 5605 cpu_manager.go:414] "containerMap: removing container" podUID="6f7abd33-f094-4091-b738-d77e72cfbf05" containerName="istio-proxy" Nov 03 02:58:35.325905 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:35.325823 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f7abd33-f094-4091-b738-d77e72cfbf05" containerName="istio-proxy" Nov 03 02:58:35.325905 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:35.325829 5605 cpu_manager.go:414] "containerMap: removing container" podUID="e82f0187-0392-499b-ac69-0e7e03eebbf7" containerName="payload-processing" Nov 03 02:58:35.325905 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:35.325834 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e82f0187-0392-499b-ac69-0e7e03eebbf7" containerName="payload-processing" Nov 03 02:58:35.325905 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:35.325848 5605 cpu_manager.go:414] "containerMap: removing container" podUID="6f7abd33-f094-4091-b738-d77e72cfbf05" containerName="istio-proxy" Nov 03 02:58:35.325905 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:35.325852 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f7abd33-f094-4091-b738-d77e72cfbf05" containerName="istio-proxy" Nov 03 02:58:35.325905 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:35.325867 5605 cpu_manager.go:414] "containerMap: removing container" podUID="080ef559-4fcc-46ed-83d2-d9fa1e168792" containerName="payload-processing" Nov 03 02:58:35.325905 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:35.325872 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="080ef559-4fcc-46ed-83d2-d9fa1e168792" containerName="payload-processing" Nov 03 02:58:35.325905 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:35.325879 5605 cpu_manager.go:414] "containerMap: removing container" podUID="874c955f-0655-4260-8110-6509aef377ca" containerName="payload-pre-processing" Nov 03 02:58:35.325905 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:35.325884 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="874c955f-0655-4260-8110-6509aef377ca" containerName="payload-pre-processing" Nov 03 02:58:35.325905 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:35.325894 5605 cpu_manager.go:414] "containerMap: removing container" podUID="53640730-1899-46e3-95de-b5a502efbd89" containerName="istio-proxy" Nov 03 02:58:35.325905 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:35.325898 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53640730-1899-46e3-95de-b5a502efbd89" containerName="istio-proxy" Nov 03 02:58:35.325905 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:35.325911 5605 cpu_manager.go:414] "containerMap: removing container" podUID="53640730-1899-46e3-95de-b5a502efbd89" containerName="istio-proxy" Nov 03 02:58:35.325905 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:35.325916 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53640730-1899-46e3-95de-b5a502efbd89" containerName="istio-proxy" Nov 03 02:58:35.327007 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:35.325995 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f7abd33-f094-4091-b738-d77e72cfbf05" containerName="istio-proxy" Nov 03 02:58:35.327007 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:35.326004 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="080ef559-4fcc-46ed-83d2-d9fa1e168792" containerName="payload-processing" Nov 03 02:58:35.327007 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:35.326010 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="874c955f-0655-4260-8110-6509aef377ca" containerName="payload-pre-processing" Nov 03 02:58:35.327007 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:35.326016 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="53640730-1899-46e3-95de-b5a502efbd89" containerName="istio-proxy" Nov 03 02:58:35.327007 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:35.326024 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="53640730-1899-46e3-95de-b5a502efbd89" containerName="istio-proxy" Nov 03 02:58:35.327007 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:35.326042 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad6cd805-ef8c-4c3a-a63d-52a044bfe9ce" containerName="payload-pre-processing" Nov 03 02:58:35.327007 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:35.326047 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="29a7b266-20d5-4fe6-8a0e-c57ef087f0bb" containerName="payload-processing" Nov 03 02:58:35.327007 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:35.326052 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="51beebde-3a29-4cc8-8319-a91a990d4755" containerName="payload-processing" Nov 03 02:58:35.327007 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:35.326058 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="cec8117c-77b5-4cf8-8c6c-f3e5bffbf72b" containerName="payload-pre-processing" Nov 03 02:58:35.327007 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:35.326063 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="e82f0187-0392-499b-ac69-0e7e03eebbf7" containerName="payload-processing" Nov 03 02:58:35.330629 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:35.330608 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97dd93e8-kfr9g" Nov 03 02:58:35.383193 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:35.383155 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97dd93e8-kfr9g"] Nov 03 02:58:35.426680 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:35.426639 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:58:35.426680 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:35.426648 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:58:35.426680 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:35.426650 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:58:35.426948 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:35.426706 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:58:35.494006 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:35.493968 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ad0dc91-1b2d-49bb-81d4-6f50993b7ff9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97dd93e8-kfr9g\" (UID: \"0ad0dc91-1b2d-49bb-81d4-6f50993b7ff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97dd93e8-kfr9g" Nov 03 02:58:35.494199 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:35.494029 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p4mq\" (UniqueName: \"kubernetes.io/projected/0ad0dc91-1b2d-49bb-81d4-6f50993b7ff9-kube-api-access-6p4mq\") pod \"maas-api-revoke-sub-97dd93e8-kfr9g\" (UID: \"0ad0dc91-1b2d-49bb-81d4-6f50993b7ff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97dd93e8-kfr9g" Nov 03 02:58:35.494199 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:35.494127 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ad0dc91-1b2d-49bb-81d4-6f50993b7ff9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97dd93e8-kfr9g\" (UID: \"0ad0dc91-1b2d-49bb-81d4-6f50993b7ff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97dd93e8-kfr9g" Nov 03 02:58:35.595687 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:35.595590 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ad0dc91-1b2d-49bb-81d4-6f50993b7ff9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97dd93e8-kfr9g\" (UID: \"0ad0dc91-1b2d-49bb-81d4-6f50993b7ff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97dd93e8-kfr9g" Nov 03 02:58:35.595687 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:35.595678 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p4mq\" (UniqueName: \"kubernetes.io/projected/0ad0dc91-1b2d-49bb-81d4-6f50993b7ff9-kube-api-access-6p4mq\") pod \"maas-api-revoke-sub-97dd93e8-kfr9g\" (UID: \"0ad0dc91-1b2d-49bb-81d4-6f50993b7ff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97dd93e8-kfr9g" Nov 03 02:58:35.595913 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:35.595733 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ad0dc91-1b2d-49bb-81d4-6f50993b7ff9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97dd93e8-kfr9g\" (UID: \"0ad0dc91-1b2d-49bb-81d4-6f50993b7ff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97dd93e8-kfr9g" Nov 03 02:58:35.596329 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:35.596296 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ad0dc91-1b2d-49bb-81d4-6f50993b7ff9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97dd93e8-kfr9g\" (UID: \"0ad0dc91-1b2d-49bb-81d4-6f50993b7ff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97dd93e8-kfr9g" Nov 03 02:58:35.598062 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:35.598038 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ad0dc91-1b2d-49bb-81d4-6f50993b7ff9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97dd93e8-kfr9g\" (UID: \"0ad0dc91-1b2d-49bb-81d4-6f50993b7ff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97dd93e8-kfr9g" Nov 03 02:58:35.651936 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:35.651897 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p4mq\" (UniqueName: \"kubernetes.io/projected/0ad0dc91-1b2d-49bb-81d4-6f50993b7ff9-kube-api-access-6p4mq\") pod \"maas-api-revoke-sub-97dd93e8-kfr9g\" (UID: \"0ad0dc91-1b2d-49bb-81d4-6f50993b7ff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97dd93e8-kfr9g" Nov 03 02:58:35.943770 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:35.943682 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97dd93e8-kfr9g" Nov 03 02:58:36.313641 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:36.313596 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97dd93e8-kfr9g"] Nov 03 02:58:36.320574 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:58:36.320542 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ad0dc91_1b2d_49bb_81d4_6f50993b7ff9.slice/crio-7b52848406d5d351b977d4fe85d72ae5f570412af303dc96a52e338147b28675 WatchSource:0}: Error finding container 7b52848406d5d351b977d4fe85d72ae5f570412af303dc96a52e338147b28675: Status 404 returned error can't find the container with id 7b52848406d5d351b977d4fe85d72ae5f570412af303dc96a52e338147b28675 Nov 03 02:58:37.085743 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:37.085705 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97dd93e8-kfr9g" event={"ID":"0ad0dc91-1b2d-49bb-81d4-6f50993b7ff9","Type":"ContainerStarted","Data":"c026d0fec0ed19eb31a2b6ccaa6b996283378375ded849cfe2211bfa81b433bf"} Nov 03 02:58:37.086278 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:37.085761 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97dd93e8-kfr9g" event={"ID":"0ad0dc91-1b2d-49bb-81d4-6f50993b7ff9","Type":"ContainerStarted","Data":"7b52848406d5d351b977d4fe85d72ae5f570412af303dc96a52e338147b28675"} Nov 03 02:58:37.194908 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:37.194857 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97dd93e8-kfr9g" podStartSLOduration=2.194842718 podStartE2EDuration="2.194842718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:58:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:58:37.193162381 +0000 UTC m=+3994.452360527" watchObservedRunningTime="2026-11-03 02:58:37.194842718 +0000 UTC m=+3994.454040872" Nov 03 02:58:38.093649 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:38.093615 5605 generic.go:356] "Generic (PLEG): container finished" podID="0ad0dc91-1b2d-49bb-81d4-6f50993b7ff9" containerID="c026d0fec0ed19eb31a2b6ccaa6b996283378375ded849cfe2211bfa81b433bf" exitCode=0 Nov 03 02:58:38.094108 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:38.093704 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97dd93e8-kfr9g" event={"ID":"0ad0dc91-1b2d-49bb-81d4-6f50993b7ff9","Type":"ContainerDied","Data":"c026d0fec0ed19eb31a2b6ccaa6b996283378375ded849cfe2211bfa81b433bf"} Nov 03 02:58:38.505428 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:38.505328 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ecdbc7-557dbfdccc-7bfdq"] Nov 03 02:58:39.272936 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:39.272911 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97dd93e8-kfr9g" Nov 03 02:58:39.446694 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:39.446590 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ad0dc91-1b2d-49bb-81d4-6f50993b7ff9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ad0dc91-1b2d-49bb-81d4-6f50993b7ff9-maas-api-openshift-ca\") pod \"0ad0dc91-1b2d-49bb-81d4-6f50993b7ff9\" (UID: \"0ad0dc91-1b2d-49bb-81d4-6f50993b7ff9\") " Nov 03 02:58:39.446864 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:39.446772 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ad0dc91-1b2d-49bb-81d4-6f50993b7ff9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ad0dc91-1b2d-49bb-81d4-6f50993b7ff9-maas-api-xks-ca\") pod \"0ad0dc91-1b2d-49bb-81d4-6f50993b7ff9\" (UID: \"0ad0dc91-1b2d-49bb-81d4-6f50993b7ff9\") " Nov 03 02:58:39.446909 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:39.446869 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ad0dc91-1b2d-49bb-81d4-6f50993b7ff9-kube-api-access-6p4mq\" (UniqueName: \"kubernetes.io/projected/0ad0dc91-1b2d-49bb-81d4-6f50993b7ff9-kube-api-access-6p4mq\") pod \"0ad0dc91-1b2d-49bb-81d4-6f50993b7ff9\" (UID: \"0ad0dc91-1b2d-49bb-81d4-6f50993b7ff9\") " Nov 03 02:58:39.446993 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:39.446968 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ad0dc91-1b2d-49bb-81d4-6f50993b7ff9-maas-api-openshift-ca" pod "0ad0dc91-1b2d-49bb-81d4-6f50993b7ff9" (UID: "0ad0dc91-1b2d-49bb-81d4-6f50993b7ff9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:58:39.447394 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:39.447342 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ad0dc91-1b2d-49bb-81d4-6f50993b7ff9-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:58:39.448987 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:39.448965 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ad0dc91-1b2d-49bb-81d4-6f50993b7ff9-maas-api-xks-ca" pod "0ad0dc91-1b2d-49bb-81d4-6f50993b7ff9" (UID: "0ad0dc91-1b2d-49bb-81d4-6f50993b7ff9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:58:39.449066 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:39.448969 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ad0dc91-1b2d-49bb-81d4-6f50993b7ff9-kube-api-access-6p4mq" pod "0ad0dc91-1b2d-49bb-81d4-6f50993b7ff9" (UID: "0ad0dc91-1b2d-49bb-81d4-6f50993b7ff9"). InnerVolumeSpecName "kube-api-access-6p4mq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:58:39.548785 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:39.548736 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ad0dc91-1b2d-49bb-81d4-6f50993b7ff9-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:58:39.548785 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:39.548779 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6p4mq\" (UniqueName: \"kubernetes.io/projected/0ad0dc91-1b2d-49bb-81d4-6f50993b7ff9-kube-api-access-6p4mq\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:58:40.134284 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:40.131638 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97dd93e8-kfr9g" event={"ID":"0ad0dc91-1b2d-49bb-81d4-6f50993b7ff9","Type":"ContainerDied","Data":"7b52848406d5d351b977d4fe85d72ae5f570412af303dc96a52e338147b28675"} Nov 03 02:58:40.134284 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:40.131705 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b52848406d5d351b977d4fe85d72ae5f570412af303dc96a52e338147b28675" Nov 03 02:58:40.134284 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:40.131781 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97dd93e8-kfr9g" Nov 03 02:58:40.844621 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:40.844577 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-747c9f9449-86gct" podUID="41245c80-d885-4a8c-9ce5-24e59ed83704" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:58:42.848075 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:42.848033 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-747c9f9449-86gct" podUID="41245c80-d885-4a8c-9ce5-24e59ed83704" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:58:43.505813 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:43.505758 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-ecdbc7-557dbfdccc-7bfdq" podUID="072868fd-5f17-42fc-890b-e72ffec89324" containerName="payload-processing" containerID="cri-o://2432f18e908d61f4ab0fdc4b692223d4da8317a9c048c4a91b30fd1ae92fc838" gracePeriod=25 Nov 03 02:58:43.760157 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:43.760090 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ecdbc7-557dbfdccc-7bfdq" Nov 03 02:58:43.805491 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:43.805466 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/072868fd-5f17-42fc-890b-e72ffec89324-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/072868fd-5f17-42fc-890b-e72ffec89324-plugins-config-volume\") pod \"072868fd-5f17-42fc-890b-e72ffec89324\" (UID: \"072868fd-5f17-42fc-890b-e72ffec89324\") " Nov 03 02:58:43.805662 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:43.805514 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/072868fd-5f17-42fc-890b-e72ffec89324-kube-api-access-2wpk9\" (UniqueName: \"kubernetes.io/projected/072868fd-5f17-42fc-890b-e72ffec89324-kube-api-access-2wpk9\") pod \"072868fd-5f17-42fc-890b-e72ffec89324\" (UID: \"072868fd-5f17-42fc-890b-e72ffec89324\") " Nov 03 02:58:43.805886 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:43.805863 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/072868fd-5f17-42fc-890b-e72ffec89324-plugins-config-volume" pod "072868fd-5f17-42fc-890b-e72ffec89324" (UID: "072868fd-5f17-42fc-890b-e72ffec89324"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:58:43.806006 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:43.805990 5605 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/072868fd-5f17-42fc-890b-e72ffec89324-plugins-config-volume\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:58:43.807562 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:43.807542 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/072868fd-5f17-42fc-890b-e72ffec89324-kube-api-access-2wpk9" pod "072868fd-5f17-42fc-890b-e72ffec89324" (UID: "072868fd-5f17-42fc-890b-e72ffec89324"). InnerVolumeSpecName "kube-api-access-2wpk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:58:43.907172 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:43.907124 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wpk9\" (UniqueName: \"kubernetes.io/projected/072868fd-5f17-42fc-890b-e72ffec89324-kube-api-access-2wpk9\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:58:44.156840 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:44.156803 5605 generic.go:356] "Generic (PLEG): container finished" podID="072868fd-5f17-42fc-890b-e72ffec89324" containerID="2432f18e908d61f4ab0fdc4b692223d4da8317a9c048c4a91b30fd1ae92fc838" exitCode=0 Nov 03 02:58:44.157016 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:44.156857 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ecdbc7-557dbfdccc-7bfdq" Nov 03 02:58:44.157016 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:44.156882 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ecdbc7-557dbfdccc-7bfdq" event={"ID":"072868fd-5f17-42fc-890b-e72ffec89324","Type":"ContainerDied","Data":"2432f18e908d61f4ab0fdc4b692223d4da8317a9c048c4a91b30fd1ae92fc838"} Nov 03 02:58:44.157016 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:44.156935 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ecdbc7-557dbfdccc-7bfdq" event={"ID":"072868fd-5f17-42fc-890b-e72ffec89324","Type":"ContainerDied","Data":"f311d5da50b4c05920f8c7bffc454e7527788b998157956d65abbc7c7955ee2e"} Nov 03 02:58:44.157016 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:44.156959 5605 scope.go:122] "RemoveContainer" containerID="2432f18e908d61f4ab0fdc4b692223d4da8317a9c048c4a91b30fd1ae92fc838" Nov 03 02:58:44.168510 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:44.168488 5605 scope.go:122] "RemoveContainer" containerID="2432f18e908d61f4ab0fdc4b692223d4da8317a9c048c4a91b30fd1ae92fc838" Nov 03 02:58:44.169189 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:58:44.169164 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2432f18e908d61f4ab0fdc4b692223d4da8317a9c048c4a91b30fd1ae92fc838\": container with ID starting with 2432f18e908d61f4ab0fdc4b692223d4da8317a9c048c4a91b30fd1ae92fc838 not found: ID does not exist" containerID="2432f18e908d61f4ab0fdc4b692223d4da8317a9c048c4a91b30fd1ae92fc838" Nov 03 02:58:44.169302 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:44.169193 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2432f18e908d61f4ab0fdc4b692223d4da8317a9c048c4a91b30fd1ae92fc838"} err="failed to get container status \"2432f18e908d61f4ab0fdc4b692223d4da8317a9c048c4a91b30fd1ae92fc838\": rpc error: code = NotFound desc = could not find container \"2432f18e908d61f4ab0fdc4b692223d4da8317a9c048c4a91b30fd1ae92fc838\": container with ID starting with 2432f18e908d61f4ab0fdc4b692223d4da8317a9c048c4a91b30fd1ae92fc838 not found: ID does not exist" Nov 03 02:58:44.310179 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:44.310149 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ecdbc7-557dbfdccc-7bfdq"] Nov 03 02:58:44.312907 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:44.312888 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ecdbc7-557dbfdccc-7bfdq"] Nov 03 02:58:44.851694 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:44.851651 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="072868fd-5f17-42fc-890b-e72ffec89324" path="/var/lib/kubelet/pods/072868fd-5f17-42fc-890b-e72ffec89324/volumes" Nov 03 02:58:47.974807 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:47.974767 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-68686f579-ntp6g"] Nov 03 02:58:48.020094 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:48.020061 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-747c9f9449-86gct"] Nov 03 02:58:48.021294 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:48.021266 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-747c9f9449-86gct" podUID="41245c80-d885-4a8c-9ce5-24e59ed83704" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:58:52.975738 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:52.975678 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-68686f579-ntp6g" podUID="d89cb81e-1adf-4e6c-82ea-dde66712aab7" containerName="payload-processing" containerID="cri-o://8734436811ef402aa8a89f8cd7747828eea9ff341d20ce30ced6db251b33993d" gracePeriod=25 Nov 03 02:58:53.020975 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:53.020932 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-747c9f9449-86gct" podUID="41245c80-d885-4a8c-9ce5-24e59ed83704" containerName="payload-processing" containerID="cri-o://275cf0e3db22eb84d0e99f5dffc9ebcd9654d0fd4f7b2d7c1fd4f60808b312fa" gracePeriod=25 Nov 03 02:58:53.229643 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:53.229561 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-68686f579-ntp6g" event={"ID":"d89cb81e-1adf-4e6c-82ea-dde66712aab7","Type":"ContainerDied","Data":"8734436811ef402aa8a89f8cd7747828eea9ff341d20ce30ced6db251b33993d"} Nov 03 02:58:53.229643 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:53.229540 5605 generic.go:356] "Generic (PLEG): container finished" podID="d89cb81e-1adf-4e6c-82ea-dde66712aab7" containerID="8734436811ef402aa8a89f8cd7747828eea9ff341d20ce30ced6db251b33993d" exitCode=0 Nov 03 02:58:53.231635 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:53.231526 5605 generic.go:356] "Generic (PLEG): container finished" podID="41245c80-d885-4a8c-9ce5-24e59ed83704" containerID="275cf0e3db22eb84d0e99f5dffc9ebcd9654d0fd4f7b2d7c1fd4f60808b312fa" exitCode=0 Nov 03 02:58:53.231635 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:53.231556 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-747c9f9449-86gct" event={"ID":"41245c80-d885-4a8c-9ce5-24e59ed83704","Type":"ContainerDied","Data":"275cf0e3db22eb84d0e99f5dffc9ebcd9654d0fd4f7b2d7c1fd4f60808b312fa"} Nov 03 02:58:53.290134 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:53.290103 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-68686f579-ntp6g" Nov 03 02:58:53.317005 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:53.316977 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-747c9f9449-86gct" Nov 03 02:58:53.350300 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:53.350267 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d89cb81e-1adf-4e6c-82ea-dde66712aab7-kube-api-access-snm7b\" (UniqueName: \"kubernetes.io/projected/d89cb81e-1adf-4e6c-82ea-dde66712aab7-kube-api-access-snm7b\") pod \"d89cb81e-1adf-4e6c-82ea-dde66712aab7\" (UID: \"d89cb81e-1adf-4e6c-82ea-dde66712aab7\") " Nov 03 02:58:53.350465 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:53.350330 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d89cb81e-1adf-4e6c-82ea-dde66712aab7-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d89cb81e-1adf-4e6c-82ea-dde66712aab7-routing-overlay-volume\") pod \"d89cb81e-1adf-4e6c-82ea-dde66712aab7\" (UID: \"d89cb81e-1adf-4e6c-82ea-dde66712aab7\") " Nov 03 02:58:53.350465 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:53.350387 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d89cb81e-1adf-4e6c-82ea-dde66712aab7-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d89cb81e-1adf-4e6c-82ea-dde66712aab7-provider-credentials\") pod \"d89cb81e-1adf-4e6c-82ea-dde66712aab7\" (UID: \"d89cb81e-1adf-4e6c-82ea-dde66712aab7\") " Nov 03 02:58:53.350465 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:53.350441 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d89cb81e-1adf-4e6c-82ea-dde66712aab7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d89cb81e-1adf-4e6c-82ea-dde66712aab7-plugins-config-volume\") pod \"d89cb81e-1adf-4e6c-82ea-dde66712aab7\" (UID: \"d89cb81e-1adf-4e6c-82ea-dde66712aab7\") " Nov 03 02:58:53.350799 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:53.350769 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d89cb81e-1adf-4e6c-82ea-dde66712aab7-routing-overlay-volume" pod "d89cb81e-1adf-4e6c-82ea-dde66712aab7" (UID: "d89cb81e-1adf-4e6c-82ea-dde66712aab7"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:58:53.350924 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:53.350841 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d89cb81e-1adf-4e6c-82ea-dde66712aab7-plugins-config-volume" pod "d89cb81e-1adf-4e6c-82ea-dde66712aab7" (UID: "d89cb81e-1adf-4e6c-82ea-dde66712aab7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:58:53.351010 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:53.350993 5605 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d89cb81e-1adf-4e6c-82ea-dde66712aab7-routing-overlay-volume\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:58:53.351062 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:53.351015 5605 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d89cb81e-1adf-4e6c-82ea-dde66712aab7-plugins-config-volume\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:58:53.352599 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:53.352575 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d89cb81e-1adf-4e6c-82ea-dde66712aab7-kube-api-access-snm7b" pod "d89cb81e-1adf-4e6c-82ea-dde66712aab7" (UID: "d89cb81e-1adf-4e6c-82ea-dde66712aab7"). InnerVolumeSpecName "kube-api-access-snm7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:58:53.352699 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:53.352597 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d89cb81e-1adf-4e6c-82ea-dde66712aab7-provider-credentials" pod "d89cb81e-1adf-4e6c-82ea-dde66712aab7" (UID: "d89cb81e-1adf-4e6c-82ea-dde66712aab7"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:58:53.452214 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:53.452180 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41245c80-d885-4a8c-9ce5-24e59ed83704-provider-credentials\" (UniqueName: \"kubernetes.io/projected/41245c80-d885-4a8c-9ce5-24e59ed83704-provider-credentials\") pod \"41245c80-d885-4a8c-9ce5-24e59ed83704\" (UID: \"41245c80-d885-4a8c-9ce5-24e59ed83704\") " Nov 03 02:58:53.452506 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:53.452320 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41245c80-d885-4a8c-9ce5-24e59ed83704-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41245c80-d885-4a8c-9ce5-24e59ed83704-plugins-config-volume\") pod \"41245c80-d885-4a8c-9ce5-24e59ed83704\" (UID: \"41245c80-d885-4a8c-9ce5-24e59ed83704\") " Nov 03 02:58:53.452506 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:53.452455 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41245c80-d885-4a8c-9ce5-24e59ed83704-kube-api-access-w4x5f\" (UniqueName: \"kubernetes.io/projected/41245c80-d885-4a8c-9ce5-24e59ed83704-kube-api-access-w4x5f\") pod \"41245c80-d885-4a8c-9ce5-24e59ed83704\" (UID: \"41245c80-d885-4a8c-9ce5-24e59ed83704\") " Nov 03 02:58:53.452640 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:53.452526 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41245c80-d885-4a8c-9ce5-24e59ed83704-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/41245c80-d885-4a8c-9ce5-24e59ed83704-routing-overlay-volume\") pod \"41245c80-d885-4a8c-9ce5-24e59ed83704\" (UID: \"41245c80-d885-4a8c-9ce5-24e59ed83704\") " Nov 03 02:58:53.452914 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:53.452887 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41245c80-d885-4a8c-9ce5-24e59ed83704-plugins-config-volume" pod "41245c80-d885-4a8c-9ce5-24e59ed83704" (UID: "41245c80-d885-4a8c-9ce5-24e59ed83704"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:58:53.453050 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:53.453026 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41245c80-d885-4a8c-9ce5-24e59ed83704-routing-overlay-volume" pod "41245c80-d885-4a8c-9ce5-24e59ed83704" (UID: "41245c80-d885-4a8c-9ce5-24e59ed83704"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:58:53.453117 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:53.453090 5605 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41245c80-d885-4a8c-9ce5-24e59ed83704-plugins-config-volume\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:58:53.453117 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:53.453106 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snm7b\" (UniqueName: \"kubernetes.io/projected/d89cb81e-1adf-4e6c-82ea-dde66712aab7-kube-api-access-snm7b\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:58:53.453193 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:53.453119 5605 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d89cb81e-1adf-4e6c-82ea-dde66712aab7-provider-credentials\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:58:53.454600 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:53.454578 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41245c80-d885-4a8c-9ce5-24e59ed83704-provider-credentials" pod "41245c80-d885-4a8c-9ce5-24e59ed83704" (UID: "41245c80-d885-4a8c-9ce5-24e59ed83704"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:58:53.454689 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:53.454616 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41245c80-d885-4a8c-9ce5-24e59ed83704-kube-api-access-w4x5f" pod "41245c80-d885-4a8c-9ce5-24e59ed83704" (UID: "41245c80-d885-4a8c-9ce5-24e59ed83704"). InnerVolumeSpecName "kube-api-access-w4x5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:58:53.554308 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:53.554275 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4x5f\" (UniqueName: \"kubernetes.io/projected/41245c80-d885-4a8c-9ce5-24e59ed83704-kube-api-access-w4x5f\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:58:53.554308 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:53.554306 5605 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/41245c80-d885-4a8c-9ce5-24e59ed83704-routing-overlay-volume\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:58:53.554513 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:53.554315 5605 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/41245c80-d885-4a8c-9ce5-24e59ed83704-provider-credentials\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:58:53.916220 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:58:53.916125 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:58:54.239034 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:54.238943 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-68686f579-ntp6g" Nov 03 02:58:54.239034 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:54.238943 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-68686f579-ntp6g" event={"ID":"d89cb81e-1adf-4e6c-82ea-dde66712aab7","Type":"ContainerDied","Data":"42b0f290bcadbdd2c859596babbfc446eb1fc1cdee140ae8d92614b7b8831e41"} Nov 03 02:58:54.239591 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:54.239095 5605 scope.go:122] "RemoveContainer" containerID="8734436811ef402aa8a89f8cd7747828eea9ff341d20ce30ced6db251b33993d" Nov 03 02:58:54.240504 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:54.240486 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-747c9f9449-86gct" Nov 03 02:58:54.240604 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:54.240491 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-747c9f9449-86gct" event={"ID":"41245c80-d885-4a8c-9ce5-24e59ed83704","Type":"ContainerDied","Data":"e59137bf67508d80899d76a045786978fcd44b9ac5116873ba95e51b6c9906cb"} Nov 03 02:58:54.252345 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:54.252324 5605 scope.go:122] "RemoveContainer" containerID="275cf0e3db22eb84d0e99f5dffc9ebcd9654d0fd4f7b2d7c1fd4f60808b312fa" Nov 03 02:58:54.424529 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:54.424471 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-68686f579-ntp6g"] Nov 03 02:58:54.432014 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:54.431970 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-68686f579-ntp6g"] Nov 03 02:58:54.511772 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:54.511694 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-747c9f9449-86gct"] Nov 03 02:58:54.516185 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:54.516159 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-747c9f9449-86gct"] Nov 03 02:58:54.850130 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:54.850096 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41245c80-d885-4a8c-9ce5-24e59ed83704" path="/var/lib/kubelet/pods/41245c80-d885-4a8c-9ce5-24e59ed83704/volumes" Nov 03 02:58:54.850573 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:54.850559 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d89cb81e-1adf-4e6c-82ea-dde66712aab7" path="/var/lib/kubelet/pods/d89cb81e-1adf-4e6c-82ea-dde66712aab7/volumes" Nov 03 02:58:55.400801 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:55.400756 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7d4b74954f-jfkjm"] Nov 03 02:58:55.401384 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:55.401330 5605 cpu_manager.go:414] "containerMap: removing container" podUID="0ad0dc91-1b2d-49bb-81d4-6f50993b7ff9" containerName="revoke-keys" Nov 03 02:58:55.401384 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:55.401350 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ad0dc91-1b2d-49bb-81d4-6f50993b7ff9" containerName="revoke-keys" Nov 03 02:58:55.401598 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:55.401392 5605 cpu_manager.go:414] "containerMap: removing container" podUID="41245c80-d885-4a8c-9ce5-24e59ed83704" containerName="payload-processing" Nov 03 02:58:55.401598 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:55.401401 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41245c80-d885-4a8c-9ce5-24e59ed83704" containerName="payload-processing" Nov 03 02:58:55.401598 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:55.401425 5605 cpu_manager.go:414] "containerMap: removing container" podUID="d89cb81e-1adf-4e6c-82ea-dde66712aab7" containerName="payload-processing" Nov 03 02:58:55.401598 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:55.401432 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d89cb81e-1adf-4e6c-82ea-dde66712aab7" containerName="payload-processing" Nov 03 02:58:55.401598 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:55.401440 5605 cpu_manager.go:414] "containerMap: removing container" podUID="072868fd-5f17-42fc-890b-e72ffec89324" containerName="payload-processing" Nov 03 02:58:55.401598 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:55.401444 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="072868fd-5f17-42fc-890b-e72ffec89324" containerName="payload-processing" Nov 03 02:58:55.401598 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:55.401548 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="d89cb81e-1adf-4e6c-82ea-dde66712aab7" containerName="payload-processing" Nov 03 02:58:55.401598 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:55.401563 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="072868fd-5f17-42fc-890b-e72ffec89324" containerName="payload-processing" Nov 03 02:58:55.401598 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:55.401578 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ad0dc91-1b2d-49bb-81d4-6f50993b7ff9" containerName="revoke-keys" Nov 03 02:58:55.401598 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:55.401588 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f7abd33-f094-4091-b738-d77e72cfbf05" containerName="istio-proxy" Nov 03 02:58:55.401598 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:55.401599 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="41245c80-d885-4a8c-9ce5-24e59ed83704" containerName="payload-processing" Nov 03 02:58:55.407097 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:55.407063 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7d4b74954f-jfkjm" Nov 03 02:58:55.456503 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:55.456466 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 03 02:58:55.485152 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:55.485117 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2d8852b6-18c5-4570-a2ad-f5fa7a544486-provider-credentials\") pod \"payload-processing-external-model-7d4b74954f-jfkjm\" (UID: \"2d8852b6-18c5-4570-a2ad-f5fa7a544486\") " pod="models-as-a-service/payload-processing-external-model-7d4b74954f-jfkjm" Nov 03 02:58:55.485375 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:55.485194 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2d8852b6-18c5-4570-a2ad-f5fa7a544486-routing-overlay-volume\") pod \"payload-processing-external-model-7d4b74954f-jfkjm\" (UID: \"2d8852b6-18c5-4570-a2ad-f5fa7a544486\") " pod="models-as-a-service/payload-processing-external-model-7d4b74954f-jfkjm" Nov 03 02:58:55.485375 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:55.485254 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95hqp\" (UniqueName: \"kubernetes.io/projected/2d8852b6-18c5-4570-a2ad-f5fa7a544486-kube-api-access-95hqp\") pod \"payload-processing-external-model-7d4b74954f-jfkjm\" (UID: \"2d8852b6-18c5-4570-a2ad-f5fa7a544486\") " pod="models-as-a-service/payload-processing-external-model-7d4b74954f-jfkjm" Nov 03 02:58:55.485481 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:55.485455 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d8852b6-18c5-4570-a2ad-f5fa7a544486-plugins-config-volume\") pod \"payload-processing-external-model-7d4b74954f-jfkjm\" (UID: \"2d8852b6-18c5-4570-a2ad-f5fa7a544486\") " pod="models-as-a-service/payload-processing-external-model-7d4b74954f-jfkjm" Nov 03 02:58:55.503102 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:55.503049 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 03 02:58:55.503102 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:55.503049 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 03 02:58:55.503102 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:55.503051 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-bf1f66a30e76\"" Nov 03 02:58:55.503102 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:55.503052 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 03 02:58:55.503506 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:55.503162 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-rt469\"" Nov 03 02:58:55.505627 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:55.505601 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7d4b74954f-jfkjm"] Nov 03 02:58:55.586713 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:55.586663 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d8852b6-18c5-4570-a2ad-f5fa7a544486-plugins-config-volume\") pod \"payload-processing-external-model-7d4b74954f-jfkjm\" (UID: \"2d8852b6-18c5-4570-a2ad-f5fa7a544486\") " pod="models-as-a-service/payload-processing-external-model-7d4b74954f-jfkjm" Nov 03 02:58:55.586934 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:55.586815 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2d8852b6-18c5-4570-a2ad-f5fa7a544486-provider-credentials\") pod \"payload-processing-external-model-7d4b74954f-jfkjm\" (UID: \"2d8852b6-18c5-4570-a2ad-f5fa7a544486\") " pod="models-as-a-service/payload-processing-external-model-7d4b74954f-jfkjm" Nov 03 02:58:55.586934 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:55.586899 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2d8852b6-18c5-4570-a2ad-f5fa7a544486-routing-overlay-volume\") pod \"payload-processing-external-model-7d4b74954f-jfkjm\" (UID: \"2d8852b6-18c5-4570-a2ad-f5fa7a544486\") " pod="models-as-a-service/payload-processing-external-model-7d4b74954f-jfkjm" Nov 03 02:58:55.586934 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:55.586921 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95hqp\" (UniqueName: \"kubernetes.io/projected/2d8852b6-18c5-4570-a2ad-f5fa7a544486-kube-api-access-95hqp\") pod \"payload-processing-external-model-7d4b74954f-jfkjm\" (UID: \"2d8852b6-18c5-4570-a2ad-f5fa7a544486\") " pod="models-as-a-service/payload-processing-external-model-7d4b74954f-jfkjm" Nov 03 02:58:55.587568 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:55.587543 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2d8852b6-18c5-4570-a2ad-f5fa7a544486-routing-overlay-volume\") pod \"payload-processing-external-model-7d4b74954f-jfkjm\" (UID: \"2d8852b6-18c5-4570-a2ad-f5fa7a544486\") " pod="models-as-a-service/payload-processing-external-model-7d4b74954f-jfkjm" Nov 03 02:58:55.587704 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:55.587610 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d8852b6-18c5-4570-a2ad-f5fa7a544486-plugins-config-volume\") pod \"payload-processing-external-model-7d4b74954f-jfkjm\" (UID: \"2d8852b6-18c5-4570-a2ad-f5fa7a544486\") " pod="models-as-a-service/payload-processing-external-model-7d4b74954f-jfkjm" Nov 03 02:58:55.589432 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:55.589406 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2d8852b6-18c5-4570-a2ad-f5fa7a544486-provider-credentials\") pod \"payload-processing-external-model-7d4b74954f-jfkjm\" (UID: \"2d8852b6-18c5-4570-a2ad-f5fa7a544486\") " pod="models-as-a-service/payload-processing-external-model-7d4b74954f-jfkjm" Nov 03 02:58:55.640515 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:55.640487 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95hqp\" (UniqueName: \"kubernetes.io/projected/2d8852b6-18c5-4570-a2ad-f5fa7a544486-kube-api-access-95hqp\") pod \"payload-processing-external-model-7d4b74954f-jfkjm\" (UID: \"2d8852b6-18c5-4570-a2ad-f5fa7a544486\") " pod="models-as-a-service/payload-processing-external-model-7d4b74954f-jfkjm" Nov 03 02:58:55.721892 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:55.721811 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7d4b74954f-jfkjm" Nov 03 02:58:56.094001 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:56.093958 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7d4b74954f-jfkjm"] Nov 03 02:58:56.100066 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:58:56.100039 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d8852b6_18c5_4570_a2ad_f5fa7a544486.slice/crio-dfcbe911f57eff4c99143a5a0c49e38293fb9cd360f4221b67cfbf2cadbada79 WatchSource:0}: Error finding container dfcbe911f57eff4c99143a5a0c49e38293fb9cd360f4221b67cfbf2cadbada79: Status 404 returned error can't find the container with id dfcbe911f57eff4c99143a5a0c49e38293fb9cd360f4221b67cfbf2cadbada79 Nov 03 02:58:56.257687 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:56.257650 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7d4b74954f-jfkjm" event={"ID":"2d8852b6-18c5-4570-a2ad-f5fa7a544486","Type":"ContainerStarted","Data":"5d21ad885e91f3d47ba171076c2e8ec60d7c45a848a01c549d17e0f1a6676d09"} Nov 03 02:58:56.257884 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:56.257712 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7d4b74954f-jfkjm" event={"ID":"2d8852b6-18c5-4570-a2ad-f5fa7a544486","Type":"ContainerStarted","Data":"dfcbe911f57eff4c99143a5a0c49e38293fb9cd360f4221b67cfbf2cadbada79"} Nov 03 02:58:56.257884 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:56.257744 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7d4b74954f-jfkjm" Nov 03 02:58:56.367584 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:56.367479 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7d4b74954f-jfkjm" podStartSLOduration=1.367464977 podStartE2EDuration="1.367464977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:58:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:58:56.36571701 +0000 UTC m=+4013.624915164" watchObservedRunningTime="2026-11-03 02:58:56.367464977 +0000 UTC m=+4013.626663134" Nov 03 02:58:57.409406 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:58:57.409350 5605 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-bf1f66a30e76: secret "agc-cred-bf1f66a30e76" not found Nov 03 02:58:57.409406 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:58:57.409395 5605 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7d4b74954f-jfkjm: secret "agc-cred-bf1f66a30e76" not found Nov 03 02:58:57.409925 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:58:57.409464 5605 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2d8852b6-18c5-4570-a2ad-f5fa7a544486-provider-credentials podName:2d8852b6-18c5-4570-a2ad-f5fa7a544486 nodeName:}" failed. No retries permitted until 2026-11-03 02:58:57.909442875 +0000 UTC m=+4015.168641021 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2d8852b6-18c5-4570-a2ad-f5fa7a544486-provider-credentials") pod "payload-processing-external-model-7d4b74954f-jfkjm" (UID: "2d8852b6-18c5-4570-a2ad-f5fa7a544486") : secret "agc-cred-bf1f66a30e76" not found Nov 03 02:58:57.914927 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:58:57.914893 5605 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-bf1f66a30e76: secret "agc-cred-bf1f66a30e76" not found Nov 03 02:58:57.914927 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:58:57.914921 5605 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7d4b74954f-jfkjm: secret "agc-cred-bf1f66a30e76" not found Nov 03 02:58:57.915226 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:58:57.914976 5605 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2d8852b6-18c5-4570-a2ad-f5fa7a544486-provider-credentials podName:2d8852b6-18c5-4570-a2ad-f5fa7a544486 nodeName:}" failed. No retries permitted until 2026-11-03 02:58:58.914959145 +0000 UTC m=+4016.174157285 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2d8852b6-18c5-4570-a2ad-f5fa7a544486-provider-credentials") pod "payload-processing-external-model-7d4b74954f-jfkjm" (UID: "2d8852b6-18c5-4570-a2ad-f5fa7a544486") : secret "agc-cred-bf1f66a30e76" not found Nov 03 02:58:58.238651 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:58:58.238560 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7d4b74954f-jfkjm"] Nov 03 02:58:58.927020 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:58:58.926981 5605 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-bf1f66a30e76: secret "agc-cred-bf1f66a30e76" not found Nov 03 02:58:58.927020 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:58:58.927013 5605 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7d4b74954f-jfkjm: secret "agc-cred-bf1f66a30e76" not found Nov 03 02:58:58.927736 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:58:58.927078 5605 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2d8852b6-18c5-4570-a2ad-f5fa7a544486-provider-credentials podName:2d8852b6-18c5-4570-a2ad-f5fa7a544486 nodeName:}" failed. No retries permitted until 2026-11-03 02:59:00.927062893 +0000 UTC m=+4018.186261025 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2d8852b6-18c5-4570-a2ad-f5fa7a544486-provider-credentials") pod "payload-processing-external-model-7d4b74954f-jfkjm" (UID: "2d8852b6-18c5-4570-a2ad-f5fa7a544486") : secret "agc-cred-bf1f66a30e76" not found Nov 03 02:59:00.955135 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:59:00.955100 5605 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-bf1f66a30e76: secret "agc-cred-bf1f66a30e76" not found Nov 03 02:59:00.955135 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:59:00.955127 5605 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7d4b74954f-jfkjm: secret "agc-cred-bf1f66a30e76" not found Nov 03 02:59:00.955642 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:59:00.955193 5605 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2d8852b6-18c5-4570-a2ad-f5fa7a544486-provider-credentials podName:2d8852b6-18c5-4570-a2ad-f5fa7a544486 nodeName:}" failed. No retries permitted until 2026-11-03 02:59:04.955172696 +0000 UTC m=+4022.214370837 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2d8852b6-18c5-4570-a2ad-f5fa7a544486-provider-credentials") pod "payload-processing-external-model-7d4b74954f-jfkjm" (UID: "2d8852b6-18c5-4570-a2ad-f5fa7a544486") : secret "agc-cred-bf1f66a30e76" not found Nov 03 02:59:03.270982 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:03.270879 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7d4b74954f-jfkjm" podUID="2d8852b6-18c5-4570-a2ad-f5fa7a544486" containerName="payload-processing" containerID="cri-o://5d21ad885e91f3d47ba171076c2e8ec60d7c45a848a01c549d17e0f1a6676d09" gracePeriod=25 Nov 03 02:59:03.589662 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:03.589626 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7d4b74954f-jfkjm" Nov 03 02:59:03.697802 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:03.697768 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d8852b6-18c5-4570-a2ad-f5fa7a544486-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2d8852b6-18c5-4570-a2ad-f5fa7a544486-routing-overlay-volume\") pod \"2d8852b6-18c5-4570-a2ad-f5fa7a544486\" (UID: \"2d8852b6-18c5-4570-a2ad-f5fa7a544486\") " Nov 03 02:59:03.698004 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:03.697876 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d8852b6-18c5-4570-a2ad-f5fa7a544486-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2d8852b6-18c5-4570-a2ad-f5fa7a544486-provider-credentials\") pod \"2d8852b6-18c5-4570-a2ad-f5fa7a544486\" (UID: \"2d8852b6-18c5-4570-a2ad-f5fa7a544486\") " Nov 03 02:59:03.698004 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:03.697914 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d8852b6-18c5-4570-a2ad-f5fa7a544486-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d8852b6-18c5-4570-a2ad-f5fa7a544486-plugins-config-volume\") pod \"2d8852b6-18c5-4570-a2ad-f5fa7a544486\" (UID: \"2d8852b6-18c5-4570-a2ad-f5fa7a544486\") " Nov 03 02:59:03.698004 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:03.697937 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d8852b6-18c5-4570-a2ad-f5fa7a544486-kube-api-access-95hqp\" (UniqueName: \"kubernetes.io/projected/2d8852b6-18c5-4570-a2ad-f5fa7a544486-kube-api-access-95hqp\") pod \"2d8852b6-18c5-4570-a2ad-f5fa7a544486\" (UID: \"2d8852b6-18c5-4570-a2ad-f5fa7a544486\") " Nov 03 02:59:03.698216 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:03.698135 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d8852b6-18c5-4570-a2ad-f5fa7a544486-routing-overlay-volume" pod "2d8852b6-18c5-4570-a2ad-f5fa7a544486" (UID: "2d8852b6-18c5-4570-a2ad-f5fa7a544486"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:59:03.698323 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:03.698302 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d8852b6-18c5-4570-a2ad-f5fa7a544486-plugins-config-volume" pod "2d8852b6-18c5-4570-a2ad-f5fa7a544486" (UID: "2d8852b6-18c5-4570-a2ad-f5fa7a544486"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:59:03.698587 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:03.698560 5605 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2d8852b6-18c5-4570-a2ad-f5fa7a544486-routing-overlay-volume\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:59:03.698587 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:03.698586 5605 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d8852b6-18c5-4570-a2ad-f5fa7a544486-plugins-config-volume\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:59:03.700515 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:03.700486 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d8852b6-18c5-4570-a2ad-f5fa7a544486-provider-credentials" pod "2d8852b6-18c5-4570-a2ad-f5fa7a544486" (UID: "2d8852b6-18c5-4570-a2ad-f5fa7a544486"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:59:03.700829 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:03.700806 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d8852b6-18c5-4570-a2ad-f5fa7a544486-kube-api-access-95hqp" pod "2d8852b6-18c5-4570-a2ad-f5fa7a544486" (UID: "2d8852b6-18c5-4570-a2ad-f5fa7a544486"). InnerVolumeSpecName "kube-api-access-95hqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:59:03.799760 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:03.799737 5605 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2d8852b6-18c5-4570-a2ad-f5fa7a544486-provider-credentials\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:59:03.799907 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:03.799762 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95hqp\" (UniqueName: \"kubernetes.io/projected/2d8852b6-18c5-4570-a2ad-f5fa7a544486-kube-api-access-95hqp\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:59:04.309262 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:04.309220 5605 generic.go:356] "Generic (PLEG): container finished" podID="2d8852b6-18c5-4570-a2ad-f5fa7a544486" containerID="5d21ad885e91f3d47ba171076c2e8ec60d7c45a848a01c549d17e0f1a6676d09" exitCode=0 Nov 03 02:59:04.309748 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:04.309268 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7d4b74954f-jfkjm" event={"ID":"2d8852b6-18c5-4570-a2ad-f5fa7a544486","Type":"ContainerDied","Data":"5d21ad885e91f3d47ba171076c2e8ec60d7c45a848a01c549d17e0f1a6676d09"} Nov 03 02:59:04.309748 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:04.309310 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7d4b74954f-jfkjm" event={"ID":"2d8852b6-18c5-4570-a2ad-f5fa7a544486","Type":"ContainerDied","Data":"dfcbe911f57eff4c99143a5a0c49e38293fb9cd360f4221b67cfbf2cadbada79"} Nov 03 02:59:04.309748 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:04.309332 5605 scope.go:122] "RemoveContainer" containerID="5d21ad885e91f3d47ba171076c2e8ec60d7c45a848a01c549d17e0f1a6676d09" Nov 03 02:59:04.309748 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:04.309268 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7d4b74954f-jfkjm" Nov 03 02:59:04.318949 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:04.318924 5605 scope.go:122] "RemoveContainer" containerID="5d21ad885e91f3d47ba171076c2e8ec60d7c45a848a01c549d17e0f1a6676d09" Nov 03 02:59:04.319255 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:59:04.319227 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d21ad885e91f3d47ba171076c2e8ec60d7c45a848a01c549d17e0f1a6676d09\": container with ID starting with 5d21ad885e91f3d47ba171076c2e8ec60d7c45a848a01c549d17e0f1a6676d09 not found: ID does not exist" containerID="5d21ad885e91f3d47ba171076c2e8ec60d7c45a848a01c549d17e0f1a6676d09" Nov 03 02:59:04.319345 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:04.319255 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d21ad885e91f3d47ba171076c2e8ec60d7c45a848a01c549d17e0f1a6676d09"} err="failed to get container status \"5d21ad885e91f3d47ba171076c2e8ec60d7c45a848a01c549d17e0f1a6676d09\": rpc error: code = NotFound desc = could not find container \"5d21ad885e91f3d47ba171076c2e8ec60d7c45a848a01c549d17e0f1a6676d09\": container with ID starting with 5d21ad885e91f3d47ba171076c2e8ec60d7c45a848a01c549d17e0f1a6676d09 not found: ID does not exist" Nov 03 02:59:04.463441 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:04.463405 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7d4b74954f-jfkjm"] Nov 03 02:59:04.510521 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:04.510488 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7d4b74954f-jfkjm"] Nov 03 02:59:04.850092 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:04.850061 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d8852b6-18c5-4570-a2ad-f5fa7a544486" path="/var/lib/kubelet/pods/2d8852b6-18c5-4570-a2ad-f5fa7a544486/volumes" Nov 03 02:59:08.021161 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:08.021122 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7d4b74954f-rb86j"] Nov 03 02:59:08.021651 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:08.021635 5605 cpu_manager.go:414] "containerMap: removing container" podUID="2d8852b6-18c5-4570-a2ad-f5fa7a544486" containerName="payload-processing" Nov 03 02:59:08.021697 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:08.021652 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d8852b6-18c5-4570-a2ad-f5fa7a544486" containerName="payload-processing" Nov 03 02:59:08.021770 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:08.021760 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d8852b6-18c5-4570-a2ad-f5fa7a544486" containerName="payload-processing" Nov 03 02:59:08.025094 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:08.025074 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7d4b74954f-rb86j" Nov 03 02:59:08.055946 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:08.055900 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1c3839da-096c-45ec-a1d1-4501d3c2457f-routing-overlay-volume\") pod \"payload-processing-external-model-7d4b74954f-rb86j\" (UID: \"1c3839da-096c-45ec-a1d1-4501d3c2457f\") " pod="models-as-a-service/payload-processing-external-model-7d4b74954f-rb86j" Nov 03 02:59:08.056126 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:08.055956 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1c3839da-096c-45ec-a1d1-4501d3c2457f-provider-credentials\") pod \"payload-processing-external-model-7d4b74954f-rb86j\" (UID: \"1c3839da-096c-45ec-a1d1-4501d3c2457f\") " pod="models-as-a-service/payload-processing-external-model-7d4b74954f-rb86j" Nov 03 02:59:08.056176 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:08.056139 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4j2b\" (UniqueName: \"kubernetes.io/projected/1c3839da-096c-45ec-a1d1-4501d3c2457f-kube-api-access-l4j2b\") pod \"payload-processing-external-model-7d4b74954f-rb86j\" (UID: \"1c3839da-096c-45ec-a1d1-4501d3c2457f\") " pod="models-as-a-service/payload-processing-external-model-7d4b74954f-rb86j" Nov 03 02:59:08.056217 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:08.056172 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c3839da-096c-45ec-a1d1-4501d3c2457f-plugins-config-volume\") pod \"payload-processing-external-model-7d4b74954f-rb86j\" (UID: \"1c3839da-096c-45ec-a1d1-4501d3c2457f\") " pod="models-as-a-service/payload-processing-external-model-7d4b74954f-rb86j" Nov 03 02:59:08.076599 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:08.076567 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-bf1f66a30e76\"" Nov 03 02:59:08.123290 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:08.123245 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-7gpns\"" Nov 03 02:59:08.123290 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:08.123245 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 03 02:59:08.123290 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:08.123245 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 03 02:59:08.123637 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:08.123252 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 03 02:59:08.123637 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:08.123428 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 03 02:59:08.125833 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:08.125812 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7d4b74954f-rb86j"] Nov 03 02:59:08.157787 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:08.157756 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1c3839da-096c-45ec-a1d1-4501d3c2457f-provider-credentials\") pod \"payload-processing-external-model-7d4b74954f-rb86j\" (UID: \"1c3839da-096c-45ec-a1d1-4501d3c2457f\") " pod="models-as-a-service/payload-processing-external-model-7d4b74954f-rb86j" Nov 03 02:59:08.157945 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:08.157869 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4j2b\" (UniqueName: \"kubernetes.io/projected/1c3839da-096c-45ec-a1d1-4501d3c2457f-kube-api-access-l4j2b\") pod \"payload-processing-external-model-7d4b74954f-rb86j\" (UID: \"1c3839da-096c-45ec-a1d1-4501d3c2457f\") " pod="models-as-a-service/payload-processing-external-model-7d4b74954f-rb86j" Nov 03 02:59:08.157945 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:08.157893 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c3839da-096c-45ec-a1d1-4501d3c2457f-plugins-config-volume\") pod \"payload-processing-external-model-7d4b74954f-rb86j\" (UID: \"1c3839da-096c-45ec-a1d1-4501d3c2457f\") " pod="models-as-a-service/payload-processing-external-model-7d4b74954f-rb86j" Nov 03 02:59:08.158077 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:08.157952 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1c3839da-096c-45ec-a1d1-4501d3c2457f-routing-overlay-volume\") pod \"payload-processing-external-model-7d4b74954f-rb86j\" (UID: \"1c3839da-096c-45ec-a1d1-4501d3c2457f\") " pod="models-as-a-service/payload-processing-external-model-7d4b74954f-rb86j" Nov 03 02:59:08.158571 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:08.158552 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1c3839da-096c-45ec-a1d1-4501d3c2457f-routing-overlay-volume\") pod \"payload-processing-external-model-7d4b74954f-rb86j\" (UID: \"1c3839da-096c-45ec-a1d1-4501d3c2457f\") " pod="models-as-a-service/payload-processing-external-model-7d4b74954f-rb86j" Nov 03 02:59:08.158724 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:08.158704 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c3839da-096c-45ec-a1d1-4501d3c2457f-plugins-config-volume\") pod \"payload-processing-external-model-7d4b74954f-rb86j\" (UID: \"1c3839da-096c-45ec-a1d1-4501d3c2457f\") " pod="models-as-a-service/payload-processing-external-model-7d4b74954f-rb86j" Nov 03 02:59:08.160243 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:08.160222 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1c3839da-096c-45ec-a1d1-4501d3c2457f-provider-credentials\") pod \"payload-processing-external-model-7d4b74954f-rb86j\" (UID: \"1c3839da-096c-45ec-a1d1-4501d3c2457f\") " pod="models-as-a-service/payload-processing-external-model-7d4b74954f-rb86j" Nov 03 02:59:08.211400 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:08.211338 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4j2b\" (UniqueName: \"kubernetes.io/projected/1c3839da-096c-45ec-a1d1-4501d3c2457f-kube-api-access-l4j2b\") pod \"payload-processing-external-model-7d4b74954f-rb86j\" (UID: \"1c3839da-096c-45ec-a1d1-4501d3c2457f\") " pod="models-as-a-service/payload-processing-external-model-7d4b74954f-rb86j" Nov 03 02:59:08.338685 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:08.338651 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7d4b74954f-rb86j" Nov 03 02:59:08.709216 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:08.709128 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7d4b74954f-rb86j"] Nov 03 02:59:08.715010 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:59:08.714982 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c3839da_096c_45ec_a1d1_4501d3c2457f.slice/crio-8448616436a17f3c5b240b70349f3c33d0dac290e599044d81cb32b2422e4c35 WatchSource:0}: Error finding container 8448616436a17f3c5b240b70349f3c33d0dac290e599044d81cb32b2422e4c35: Status 404 returned error can't find the container with id 8448616436a17f3c5b240b70349f3c33d0dac290e599044d81cb32b2422e4c35 Nov 03 02:59:09.339373 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:09.339333 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7d4b74954f-rb86j" event={"ID":"1c3839da-096c-45ec-a1d1-4501d3c2457f","Type":"ContainerStarted","Data":"e99248260d30785efb4ffb9992f832bbdb1b7c470958fff98c2aad2df1d67072"} Nov 03 02:59:09.339794 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:09.339405 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7d4b74954f-rb86j" event={"ID":"1c3839da-096c-45ec-a1d1-4501d3c2457f","Type":"ContainerStarted","Data":"8448616436a17f3c5b240b70349f3c33d0dac290e599044d81cb32b2422e4c35"} Nov 03 02:59:09.339794 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:09.339531 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7d4b74954f-rb86j" Nov 03 02:59:09.447552 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:09.447492 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7d4b74954f-rb86j" podStartSLOduration=2.447476327 podStartE2EDuration="2.447476327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:59:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:59:09.445624958 +0000 UTC m=+4026.704823111" watchObservedRunningTime="2026-11-03 02:59:09.447476327 +0000 UTC m=+4026.706674480" Nov 03 02:59:10.387813 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:59:10.387779 5605 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-bf1f66a30e76: secret "agc-cred-bf1f66a30e76" not found Nov 03 02:59:10.387813 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:59:10.387806 5605 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7d4b74954f-rb86j: secret "agc-cred-bf1f66a30e76" not found Nov 03 02:59:10.388316 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:59:10.387866 5605 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c3839da-096c-45ec-a1d1-4501d3c2457f-provider-credentials podName:1c3839da-096c-45ec-a1d1-4501d3c2457f nodeName:}" failed. No retries permitted until 2026-11-03 02:59:10.88784679 +0000 UTC m=+4028.147044923 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1c3839da-096c-45ec-a1d1-4501d3c2457f-provider-credentials") pod "payload-processing-external-model-7d4b74954f-rb86j" (UID: "1c3839da-096c-45ec-a1d1-4501d3c2457f") : secret "agc-cred-bf1f66a30e76" not found Nov 03 02:59:10.893981 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:59:10.893938 5605 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-bf1f66a30e76: secret "agc-cred-bf1f66a30e76" not found Nov 03 02:59:10.893981 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:59:10.893971 5605 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7d4b74954f-rb86j: secret "agc-cred-bf1f66a30e76" not found Nov 03 02:59:10.894246 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:59:10.894032 5605 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c3839da-096c-45ec-a1d1-4501d3c2457f-provider-credentials podName:1c3839da-096c-45ec-a1d1-4501d3c2457f nodeName:}" failed. No retries permitted until 2026-11-03 02:59:11.894010249 +0000 UTC m=+4029.153208382 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1c3839da-096c-45ec-a1d1-4501d3c2457f-provider-credentials") pod "payload-processing-external-model-7d4b74954f-rb86j" (UID: "1c3839da-096c-45ec-a1d1-4501d3c2457f") : secret "agc-cred-bf1f66a30e76" not found Nov 03 02:59:11.906043 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:59:11.906002 5605 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-bf1f66a30e76: secret "agc-cred-bf1f66a30e76" not found Nov 03 02:59:11.906043 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:59:11.906037 5605 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7d4b74954f-rb86j: secret "agc-cred-bf1f66a30e76" not found Nov 03 02:59:11.906562 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:59:11.906109 5605 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c3839da-096c-45ec-a1d1-4501d3c2457f-provider-credentials podName:1c3839da-096c-45ec-a1d1-4501d3c2457f nodeName:}" failed. No retries permitted until 2026-11-03 02:59:13.906087349 +0000 UTC m=+4031.165285501 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1c3839da-096c-45ec-a1d1-4501d3c2457f-provider-credentials") pod "payload-processing-external-model-7d4b74954f-rb86j" (UID: "1c3839da-096c-45ec-a1d1-4501d3c2457f") : secret "agc-cred-bf1f66a30e76" not found Nov 03 02:59:13.037683 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:13.037643 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7d4b74954f-rb86j"] Nov 03 02:59:13.041108 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:13.041081 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-7d4b74954f-rb86j" Nov 03 02:59:13.937378 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:59:13.936061 5605 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-bf1f66a30e76: secret "agc-cred-bf1f66a30e76" not found Nov 03 02:59:13.937378 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:59:13.936095 5605 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7d4b74954f-rb86j: secret "agc-cred-bf1f66a30e76" not found Nov 03 02:59:13.937378 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:59:13.936155 5605 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c3839da-096c-45ec-a1d1-4501d3c2457f-provider-credentials podName:1c3839da-096c-45ec-a1d1-4501d3c2457f nodeName:}" failed. No retries permitted until 2026-11-03 02:59:17.936135992 +0000 UTC m=+4035.195334139 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1c3839da-096c-45ec-a1d1-4501d3c2457f-provider-credentials") pod "payload-processing-external-model-7d4b74954f-rb86j" (UID: "1c3839da-096c-45ec-a1d1-4501d3c2457f") : secret "agc-cred-bf1f66a30e76" not found Nov 03 02:59:14.410409 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:14.410369 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-dc14f6-r8pvg"] Nov 03 02:59:14.414738 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:14.414714 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-dc14f6-r8pvg" Nov 03 02:59:14.464123 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:14.464095 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:59:14.510689 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:14.510653 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:59:14.510689 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:14.510653 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:59:14.510882 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:14.510780 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:59:14.513218 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:14.513200 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-dc14f6-r8pvg"] Nov 03 02:59:14.541656 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:14.541632 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ea0b63b-c7c4-4d08-b281-e29e6c139462-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-dc14f6-r8pvg\" (UID: \"7ea0b63b-c7c4-4d08-b281-e29e6c139462\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-dc14f6-r8pvg" Nov 03 02:59:14.541770 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:14.541711 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqw6c\" (UniqueName: \"kubernetes.io/projected/7ea0b63b-c7c4-4d08-b281-e29e6c139462-kube-api-access-dqw6c\") pod \"maas-api-revoke-keys-e2e-extmod-b-dc14f6-r8pvg\" (UID: \"7ea0b63b-c7c4-4d08-b281-e29e6c139462\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-dc14f6-r8pvg" Nov 03 02:59:14.541856 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:14.541841 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ea0b63b-c7c4-4d08-b281-e29e6c139462-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-dc14f6-r8pvg\" (UID: \"7ea0b63b-c7c4-4d08-b281-e29e6c139462\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-dc14f6-r8pvg" Nov 03 02:59:14.643453 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:14.643419 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ea0b63b-c7c4-4d08-b281-e29e6c139462-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-dc14f6-r8pvg\" (UID: \"7ea0b63b-c7c4-4d08-b281-e29e6c139462\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-dc14f6-r8pvg" Nov 03 02:59:14.643616 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:14.643519 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqw6c\" (UniqueName: \"kubernetes.io/projected/7ea0b63b-c7c4-4d08-b281-e29e6c139462-kube-api-access-dqw6c\") pod \"maas-api-revoke-keys-e2e-extmod-b-dc14f6-r8pvg\" (UID: \"7ea0b63b-c7c4-4d08-b281-e29e6c139462\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-dc14f6-r8pvg" Nov 03 02:59:14.643616 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:14.643572 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ea0b63b-c7c4-4d08-b281-e29e6c139462-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-dc14f6-r8pvg\" (UID: \"7ea0b63b-c7c4-4d08-b281-e29e6c139462\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-dc14f6-r8pvg" Nov 03 02:59:14.644254 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:14.644231 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ea0b63b-c7c4-4d08-b281-e29e6c139462-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-dc14f6-r8pvg\" (UID: \"7ea0b63b-c7c4-4d08-b281-e29e6c139462\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-dc14f6-r8pvg" Nov 03 02:59:14.645748 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:14.645724 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ea0b63b-c7c4-4d08-b281-e29e6c139462-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-dc14f6-r8pvg\" (UID: \"7ea0b63b-c7c4-4d08-b281-e29e6c139462\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-dc14f6-r8pvg" Nov 03 02:59:14.696810 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:14.696749 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqw6c\" (UniqueName: \"kubernetes.io/projected/7ea0b63b-c7c4-4d08-b281-e29e6c139462-kube-api-access-dqw6c\") pod \"maas-api-revoke-keys-e2e-extmod-b-dc14f6-r8pvg\" (UID: \"7ea0b63b-c7c4-4d08-b281-e29e6c139462\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-dc14f6-r8pvg" Nov 03 02:59:14.728759 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:14.728733 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-dc14f6-r8pvg" Nov 03 02:59:15.102195 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:15.102159 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-dc14f6-r8pvg"] Nov 03 02:59:15.108174 ip-10-0-140-140 kubenswrapper[5605]: W1103 02:59:15.108150 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ea0b63b_c7c4_4d08_b281_e29e6c139462.slice/crio-b3814eb6d03bdd49331f5f0e8025d71918eaae762d4891c1049e6787386aaae3 WatchSource:0}: Error finding container b3814eb6d03bdd49331f5f0e8025d71918eaae762d4891c1049e6787386aaae3: Status 404 returned error can't find the container with id b3814eb6d03bdd49331f5f0e8025d71918eaae762d4891c1049e6787386aaae3 Nov 03 02:59:15.376672 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:15.376637 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-dc14f6-r8pvg" event={"ID":"7ea0b63b-c7c4-4d08-b281-e29e6c139462","Type":"ContainerStarted","Data":"9b6a81b74b31b0ea401bef5716ed8b8ac5b881c3d22849d24022ea63595fd8fd"} Nov 03 02:59:15.376817 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:15.376695 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-dc14f6-r8pvg" event={"ID":"7ea0b63b-c7c4-4d08-b281-e29e6c139462","Type":"ContainerStarted","Data":"b3814eb6d03bdd49331f5f0e8025d71918eaae762d4891c1049e6787386aaae3"} Nov 03 02:59:15.485498 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:15.485440 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-dc14f6-r8pvg" podStartSLOduration=1.485425306 podStartE2EDuration="1.485425306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:59:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:59:15.482714206 +0000 UTC m=+4032.741912362" watchObservedRunningTime="2026-11-03 02:59:15.485425306 +0000 UTC m=+4032.744623459" Nov 03 02:59:16.391114 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:16.391073 5605 generic.go:356] "Generic (PLEG): container finished" podID="7ea0b63b-c7c4-4d08-b281-e29e6c139462" containerID="9b6a81b74b31b0ea401bef5716ed8b8ac5b881c3d22849d24022ea63595fd8fd" exitCode=0 Nov 03 02:59:16.391327 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:16.391156 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-dc14f6-r8pvg" event={"ID":"7ea0b63b-c7c4-4d08-b281-e29e6c139462","Type":"ContainerDied","Data":"9b6a81b74b31b0ea401bef5716ed8b8ac5b881c3d22849d24022ea63595fd8fd"} Nov 03 02:59:17.538715 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:17.538692 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-dc14f6-r8pvg" Nov 03 02:59:17.679816 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:17.679731 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ea0b63b-c7c4-4d08-b281-e29e6c139462-kube-api-access-dqw6c\" (UniqueName: \"kubernetes.io/projected/7ea0b63b-c7c4-4d08-b281-e29e6c139462-kube-api-access-dqw6c\") pod \"7ea0b63b-c7c4-4d08-b281-e29e6c139462\" (UID: \"7ea0b63b-c7c4-4d08-b281-e29e6c139462\") " Nov 03 02:59:17.679994 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:17.679853 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ea0b63b-c7c4-4d08-b281-e29e6c139462-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ea0b63b-c7c4-4d08-b281-e29e6c139462-maas-api-openshift-ca\") pod \"7ea0b63b-c7c4-4d08-b281-e29e6c139462\" (UID: \"7ea0b63b-c7c4-4d08-b281-e29e6c139462\") " Nov 03 02:59:17.679994 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:17.679961 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ea0b63b-c7c4-4d08-b281-e29e6c139462-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ea0b63b-c7c4-4d08-b281-e29e6c139462-maas-api-xks-ca\") pod \"7ea0b63b-c7c4-4d08-b281-e29e6c139462\" (UID: \"7ea0b63b-c7c4-4d08-b281-e29e6c139462\") " Nov 03 02:59:17.680246 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:17.680216 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ea0b63b-c7c4-4d08-b281-e29e6c139462-maas-api-openshift-ca" pod "7ea0b63b-c7c4-4d08-b281-e29e6c139462" (UID: "7ea0b63b-c7c4-4d08-b281-e29e6c139462"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:59:17.680636 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:17.680609 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ea0b63b-c7c4-4d08-b281-e29e6c139462-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:59:17.681893 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:17.681872 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ea0b63b-c7c4-4d08-b281-e29e6c139462-kube-api-access-dqw6c" pod "7ea0b63b-c7c4-4d08-b281-e29e6c139462" (UID: "7ea0b63b-c7c4-4d08-b281-e29e6c139462"). InnerVolumeSpecName "kube-api-access-dqw6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:59:17.682025 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:17.682008 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ea0b63b-c7c4-4d08-b281-e29e6c139462-maas-api-xks-ca" pod "7ea0b63b-c7c4-4d08-b281-e29e6c139462" (UID: "7ea0b63b-c7c4-4d08-b281-e29e6c139462"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:59:17.781410 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:17.781385 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ea0b63b-c7c4-4d08-b281-e29e6c139462-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:59:17.781410 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:17.781406 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqw6c\" (UniqueName: \"kubernetes.io/projected/7ea0b63b-c7c4-4d08-b281-e29e6c139462-kube-api-access-dqw6c\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:59:17.984036 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:59:17.983950 5605 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-bf1f66a30e76: secret "agc-cred-bf1f66a30e76" not found Nov 03 02:59:17.984036 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:59:17.983975 5605 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7d4b74954f-rb86j: secret "agc-cred-bf1f66a30e76" not found Nov 03 02:59:17.984260 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:59:17.984051 5605 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c3839da-096c-45ec-a1d1-4501d3c2457f-provider-credentials podName:1c3839da-096c-45ec-a1d1-4501d3c2457f nodeName:}" failed. No retries permitted until 2026-11-03 02:59:25.984032236 +0000 UTC m=+4043.243230382 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1c3839da-096c-45ec-a1d1-4501d3c2457f-provider-credentials") pod "payload-processing-external-model-7d4b74954f-rb86j" (UID: "1c3839da-096c-45ec-a1d1-4501d3c2457f") : secret "agc-cred-bf1f66a30e76" not found Nov 03 02:59:18.038657 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:18.038616 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7d4b74954f-rb86j" podUID="1c3839da-096c-45ec-a1d1-4501d3c2457f" containerName="payload-processing" containerID="cri-o://e99248260d30785efb4ffb9992f832bbdb1b7c470958fff98c2aad2df1d67072" gracePeriod=25 Nov 03 02:59:18.284917 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:18.284892 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7d4b74954f-rb86j" Nov 03 02:59:18.389399 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:18.389369 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c3839da-096c-45ec-a1d1-4501d3c2457f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1c3839da-096c-45ec-a1d1-4501d3c2457f-provider-credentials\") pod \"1c3839da-096c-45ec-a1d1-4501d3c2457f\" (UID: \"1c3839da-096c-45ec-a1d1-4501d3c2457f\") " Nov 03 02:59:18.389575 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:18.389409 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c3839da-096c-45ec-a1d1-4501d3c2457f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1c3839da-096c-45ec-a1d1-4501d3c2457f-routing-overlay-volume\") pod \"1c3839da-096c-45ec-a1d1-4501d3c2457f\" (UID: \"1c3839da-096c-45ec-a1d1-4501d3c2457f\") " Nov 03 02:59:18.389672 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:18.389654 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c3839da-096c-45ec-a1d1-4501d3c2457f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c3839da-096c-45ec-a1d1-4501d3c2457f-plugins-config-volume\") pod \"1c3839da-096c-45ec-a1d1-4501d3c2457f\" (UID: \"1c3839da-096c-45ec-a1d1-4501d3c2457f\") " Nov 03 02:59:18.389747 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:18.389723 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c3839da-096c-45ec-a1d1-4501d3c2457f-kube-api-access-l4j2b\" (UniqueName: \"kubernetes.io/projected/1c3839da-096c-45ec-a1d1-4501d3c2457f-kube-api-access-l4j2b\") pod \"1c3839da-096c-45ec-a1d1-4501d3c2457f\" (UID: \"1c3839da-096c-45ec-a1d1-4501d3c2457f\") " Nov 03 02:59:18.389827 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:18.389802 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c3839da-096c-45ec-a1d1-4501d3c2457f-routing-overlay-volume" pod "1c3839da-096c-45ec-a1d1-4501d3c2457f" (UID: "1c3839da-096c-45ec-a1d1-4501d3c2457f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:59:18.390050 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:18.390027 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c3839da-096c-45ec-a1d1-4501d3c2457f-plugins-config-volume" pod "1c3839da-096c-45ec-a1d1-4501d3c2457f" (UID: "1c3839da-096c-45ec-a1d1-4501d3c2457f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:59:18.390236 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:18.390219 5605 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c3839da-096c-45ec-a1d1-4501d3c2457f-plugins-config-volume\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:59:18.390288 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:18.390242 5605 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1c3839da-096c-45ec-a1d1-4501d3c2457f-routing-overlay-volume\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:59:18.391686 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:18.391667 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c3839da-096c-45ec-a1d1-4501d3c2457f-provider-credentials" pod "1c3839da-096c-45ec-a1d1-4501d3c2457f" (UID: "1c3839da-096c-45ec-a1d1-4501d3c2457f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:59:18.391740 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:18.391689 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c3839da-096c-45ec-a1d1-4501d3c2457f-kube-api-access-l4j2b" pod "1c3839da-096c-45ec-a1d1-4501d3c2457f" (UID: "1c3839da-096c-45ec-a1d1-4501d3c2457f"). InnerVolumeSpecName "kube-api-access-l4j2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:59:18.405114 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:18.405091 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-dc14f6-r8pvg" Nov 03 02:59:18.405262 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:18.405122 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-dc14f6-r8pvg" event={"ID":"7ea0b63b-c7c4-4d08-b281-e29e6c139462","Type":"ContainerDied","Data":"b3814eb6d03bdd49331f5f0e8025d71918eaae762d4891c1049e6787386aaae3"} Nov 03 02:59:18.405262 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:18.405182 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3814eb6d03bdd49331f5f0e8025d71918eaae762d4891c1049e6787386aaae3" Nov 03 02:59:18.407155 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:18.407127 5605 generic.go:356] "Generic (PLEG): container finished" podID="1c3839da-096c-45ec-a1d1-4501d3c2457f" containerID="e99248260d30785efb4ffb9992f832bbdb1b7c470958fff98c2aad2df1d67072" exitCode=0 Nov 03 02:59:18.407304 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:18.407158 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7d4b74954f-rb86j" Nov 03 02:59:18.407304 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:18.407209 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7d4b74954f-rb86j" event={"ID":"1c3839da-096c-45ec-a1d1-4501d3c2457f","Type":"ContainerDied","Data":"e99248260d30785efb4ffb9992f832bbdb1b7c470958fff98c2aad2df1d67072"} Nov 03 02:59:18.407304 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:18.407266 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7d4b74954f-rb86j" event={"ID":"1c3839da-096c-45ec-a1d1-4501d3c2457f","Type":"ContainerDied","Data":"8448616436a17f3c5b240b70349f3c33d0dac290e599044d81cb32b2422e4c35"} Nov 03 02:59:18.407304 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:18.407298 5605 scope.go:122] "RemoveContainer" containerID="e99248260d30785efb4ffb9992f832bbdb1b7c470958fff98c2aad2df1d67072" Nov 03 02:59:18.423576 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:18.423555 5605 scope.go:122] "RemoveContainer" containerID="e99248260d30785efb4ffb9992f832bbdb1b7c470958fff98c2aad2df1d67072" Nov 03 02:59:18.423874 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:59:18.423849 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e99248260d30785efb4ffb9992f832bbdb1b7c470958fff98c2aad2df1d67072\": container with ID starting with e99248260d30785efb4ffb9992f832bbdb1b7c470958fff98c2aad2df1d67072 not found: ID does not exist" containerID="e99248260d30785efb4ffb9992f832bbdb1b7c470958fff98c2aad2df1d67072" Nov 03 02:59:18.423949 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:18.423884 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e99248260d30785efb4ffb9992f832bbdb1b7c470958fff98c2aad2df1d67072"} err="failed to get container status \"e99248260d30785efb4ffb9992f832bbdb1b7c470958fff98c2aad2df1d67072\": rpc error: code = NotFound desc = could not find container \"e99248260d30785efb4ffb9992f832bbdb1b7c470958fff98c2aad2df1d67072\": container with ID starting with e99248260d30785efb4ffb9992f832bbdb1b7c470958fff98c2aad2df1d67072 not found: ID does not exist" Nov 03 02:59:18.491046 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:18.491021 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4j2b\" (UniqueName: \"kubernetes.io/projected/1c3839da-096c-45ec-a1d1-4501d3c2457f-kube-api-access-l4j2b\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:59:18.491046 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:18.491042 5605 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1c3839da-096c-45ec-a1d1-4501d3c2457f-provider-credentials\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:59:18.610979 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:18.610955 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7d4b74954f-rb86j"] Nov 03 02:59:18.614821 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:18.614798 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7d4b74954f-rb86j"] Nov 03 02:59:18.850568 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:18.850543 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c3839da-096c-45ec-a1d1-4501d3c2457f" path="/var/lib/kubelet/pods/1c3839da-096c-45ec-a1d1-4501d3c2457f/volumes" Nov 03 02:59:23.915205 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:59:23.915166 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 02:59:27.736088 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:27.736058 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 02:59:32.059804 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:32.059771 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52f9d921-pgxg8"] Nov 03 02:59:32.103986 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:32.103957 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52f9d921-pgxg8"] Nov 03 02:59:32.858838 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:32.858804 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03610b3e-5d7c-4938-88ac-dcec852ce7a1" path="/var/lib/kubelet/pods/03610b3e-5d7c-4938-88ac-dcec852ce7a1/volumes" Nov 03 02:59:45.053997 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:45.053912 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4"] Nov 03 02:59:45.054429 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:45.054175 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4" podUID="264e26fa-8062-4f70-86c8-3ba19e1c4384" containerName="istio-proxy" containerID="cri-o://f672c89f9062847b547331ea0de276aed365db2c44de1ac896507081b75e459f" gracePeriod=30 Nov 03 02:59:50.330397 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:50.330297 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4" Nov 03 02:59:50.497630 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:50.497599 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/264e26fa-8062-4f70-86c8-3ba19e1c4384-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/264e26fa-8062-4f70-86c8-3ba19e1c4384-istiod-ca-cert\") pod \"264e26fa-8062-4f70-86c8-3ba19e1c4384\" (UID: \"264e26fa-8062-4f70-86c8-3ba19e1c4384\") " Nov 03 02:59:50.497789 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:50.497661 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/264e26fa-8062-4f70-86c8-3ba19e1c4384-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/264e26fa-8062-4f70-86c8-3ba19e1c4384-credential-socket\") pod \"264e26fa-8062-4f70-86c8-3ba19e1c4384\" (UID: \"264e26fa-8062-4f70-86c8-3ba19e1c4384\") " Nov 03 02:59:50.497789 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:50.497757 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/264e26fa-8062-4f70-86c8-3ba19e1c4384-kube-api-access-l5xdz\" (UniqueName: \"kubernetes.io/projected/264e26fa-8062-4f70-86c8-3ba19e1c4384-kube-api-access-l5xdz\") pod \"264e26fa-8062-4f70-86c8-3ba19e1c4384\" (UID: \"264e26fa-8062-4f70-86c8-3ba19e1c4384\") " Nov 03 02:59:50.497894 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:50.497864 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/264e26fa-8062-4f70-86c8-3ba19e1c4384-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/264e26fa-8062-4f70-86c8-3ba19e1c4384-workload-certs\") pod \"264e26fa-8062-4f70-86c8-3ba19e1c4384\" (UID: \"264e26fa-8062-4f70-86c8-3ba19e1c4384\") " Nov 03 02:59:50.497969 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:50.497908 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/264e26fa-8062-4f70-86c8-3ba19e1c4384-credential-socket" pod "264e26fa-8062-4f70-86c8-3ba19e1c4384" (UID: "264e26fa-8062-4f70-86c8-3ba19e1c4384"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:59:50.497969 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:50.497920 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/264e26fa-8062-4f70-86c8-3ba19e1c4384-istio-token\" (UniqueName: \"kubernetes.io/projected/264e26fa-8062-4f70-86c8-3ba19e1c4384-istio-token\") pod \"264e26fa-8062-4f70-86c8-3ba19e1c4384\" (UID: \"264e26fa-8062-4f70-86c8-3ba19e1c4384\") " Nov 03 02:59:50.498081 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:50.498003 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/264e26fa-8062-4f70-86c8-3ba19e1c4384-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/264e26fa-8062-4f70-86c8-3ba19e1c4384-workload-socket\") pod \"264e26fa-8062-4f70-86c8-3ba19e1c4384\" (UID: \"264e26fa-8062-4f70-86c8-3ba19e1c4384\") " Nov 03 02:59:50.498081 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:50.498017 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/264e26fa-8062-4f70-86c8-3ba19e1c4384-istiod-ca-cert" pod "264e26fa-8062-4f70-86c8-3ba19e1c4384" (UID: "264e26fa-8062-4f70-86c8-3ba19e1c4384"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:59:50.498196 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:50.498102 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/264e26fa-8062-4f70-86c8-3ba19e1c4384-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/264e26fa-8062-4f70-86c8-3ba19e1c4384-istio-envoy\") pod \"264e26fa-8062-4f70-86c8-3ba19e1c4384\" (UID: \"264e26fa-8062-4f70-86c8-3ba19e1c4384\") " Nov 03 02:59:50.498196 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:50.498132 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/264e26fa-8062-4f70-86c8-3ba19e1c4384-workload-certs" pod "264e26fa-8062-4f70-86c8-3ba19e1c4384" (UID: "264e26fa-8062-4f70-86c8-3ba19e1c4384"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:59:50.498196 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:50.498139 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/264e26fa-8062-4f70-86c8-3ba19e1c4384-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/264e26fa-8062-4f70-86c8-3ba19e1c4384-istio-data\") pod \"264e26fa-8062-4f70-86c8-3ba19e1c4384\" (UID: \"264e26fa-8062-4f70-86c8-3ba19e1c4384\") " Nov 03 02:59:50.498378 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:50.498249 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/264e26fa-8062-4f70-86c8-3ba19e1c4384-workload-socket" pod "264e26fa-8062-4f70-86c8-3ba19e1c4384" (UID: "264e26fa-8062-4f70-86c8-3ba19e1c4384"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:59:50.498378 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:50.498274 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/264e26fa-8062-4f70-86c8-3ba19e1c4384-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/264e26fa-8062-4f70-86c8-3ba19e1c4384-istio-podinfo\") pod \"264e26fa-8062-4f70-86c8-3ba19e1c4384\" (UID: \"264e26fa-8062-4f70-86c8-3ba19e1c4384\") " Nov 03 02:59:50.498378 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:50.498343 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/264e26fa-8062-4f70-86c8-3ba19e1c4384-istio-data" pod "264e26fa-8062-4f70-86c8-3ba19e1c4384" (UID: "264e26fa-8062-4f70-86c8-3ba19e1c4384"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:59:50.498776 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:50.498756 5605 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/264e26fa-8062-4f70-86c8-3ba19e1c4384-workload-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:59:50.498891 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:50.498780 5605 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/264e26fa-8062-4f70-86c8-3ba19e1c4384-workload-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:59:50.498891 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:50.498793 5605 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/264e26fa-8062-4f70-86c8-3ba19e1c4384-istio-data\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:59:50.498891 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:50.498805 5605 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/264e26fa-8062-4f70-86c8-3ba19e1c4384-istiod-ca-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:59:50.498891 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:50.498817 5605 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/264e26fa-8062-4f70-86c8-3ba19e1c4384-credential-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:59:50.500113 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:50.500091 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/264e26fa-8062-4f70-86c8-3ba19e1c4384-istio-envoy" pod "264e26fa-8062-4f70-86c8-3ba19e1c4384" (UID: "264e26fa-8062-4f70-86c8-3ba19e1c4384"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:59:50.500197 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:50.500155 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/264e26fa-8062-4f70-86c8-3ba19e1c4384-istio-token" pod "264e26fa-8062-4f70-86c8-3ba19e1c4384" (UID: "264e26fa-8062-4f70-86c8-3ba19e1c4384"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:59:50.500393 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:50.500373 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/264e26fa-8062-4f70-86c8-3ba19e1c4384-kube-api-access-l5xdz" pod "264e26fa-8062-4f70-86c8-3ba19e1c4384" (UID: "264e26fa-8062-4f70-86c8-3ba19e1c4384"). InnerVolumeSpecName "kube-api-access-l5xdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:59:50.500577 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:50.500560 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/264e26fa-8062-4f70-86c8-3ba19e1c4384-istio-podinfo" pod "264e26fa-8062-4f70-86c8-3ba19e1c4384" (UID: "264e26fa-8062-4f70-86c8-3ba19e1c4384"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:59:50.600404 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:50.600315 5605 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/264e26fa-8062-4f70-86c8-3ba19e1c4384-istio-token\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:59:50.600404 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:50.600347 5605 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/264e26fa-8062-4f70-86c8-3ba19e1c4384-istio-envoy\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:59:50.600404 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:50.600379 5605 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/264e26fa-8062-4f70-86c8-3ba19e1c4384-istio-podinfo\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:59:50.600404 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:50.600390 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5xdz\" (UniqueName: \"kubernetes.io/projected/264e26fa-8062-4f70-86c8-3ba19e1c4384-kube-api-access-l5xdz\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:59:50.630746 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:50.630721 5605 generic.go:356] "Generic (PLEG): container finished" podID="264e26fa-8062-4f70-86c8-3ba19e1c4384" containerID="f672c89f9062847b547331ea0de276aed365db2c44de1ac896507081b75e459f" exitCode=0 Nov 03 02:59:50.630854 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:50.630770 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4" Nov 03 02:59:50.630854 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:50.630810 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4" event={"ID":"264e26fa-8062-4f70-86c8-3ba19e1c4384","Type":"ContainerDied","Data":"f672c89f9062847b547331ea0de276aed365db2c44de1ac896507081b75e459f"} Nov 03 02:59:50.630935 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:50.630864 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4" event={"ID":"264e26fa-8062-4f70-86c8-3ba19e1c4384","Type":"ContainerDied","Data":"a32c50bca009c5c9429a9e847b562a780b81bd84e054cd36ce097e19dec96c92"} Nov 03 02:59:50.630935 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:50.630888 5605 scope.go:122] "RemoveContainer" containerID="f672c89f9062847b547331ea0de276aed365db2c44de1ac896507081b75e459f" Nov 03 02:59:50.640627 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:50.640611 5605 scope.go:122] "RemoveContainer" containerID="f48790376c3b6049b37350d92e3525239e699813ace92b7850f727d9e2a6259d" Nov 03 02:59:50.648933 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:50.648914 5605 scope.go:122] "RemoveContainer" containerID="f672c89f9062847b547331ea0de276aed365db2c44de1ac896507081b75e459f" Nov 03 02:59:50.649212 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:59:50.649193 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f672c89f9062847b547331ea0de276aed365db2c44de1ac896507081b75e459f\": container with ID starting with f672c89f9062847b547331ea0de276aed365db2c44de1ac896507081b75e459f not found: ID does not exist" containerID="f672c89f9062847b547331ea0de276aed365db2c44de1ac896507081b75e459f" Nov 03 02:59:50.649256 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:50.649220 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f672c89f9062847b547331ea0de276aed365db2c44de1ac896507081b75e459f"} err="failed to get container status \"f672c89f9062847b547331ea0de276aed365db2c44de1ac896507081b75e459f\": rpc error: code = NotFound desc = could not find container \"f672c89f9062847b547331ea0de276aed365db2c44de1ac896507081b75e459f\": container with ID starting with f672c89f9062847b547331ea0de276aed365db2c44de1ac896507081b75e459f not found: ID does not exist" Nov 03 02:59:50.649256 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:50.649236 5605 scope.go:122] "RemoveContainer" containerID="f48790376c3b6049b37350d92e3525239e699813ace92b7850f727d9e2a6259d" Nov 03 02:59:50.649488 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:59:50.649471 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f48790376c3b6049b37350d92e3525239e699813ace92b7850f727d9e2a6259d\": container with ID starting with f48790376c3b6049b37350d92e3525239e699813ace92b7850f727d9e2a6259d not found: ID does not exist" containerID="f48790376c3b6049b37350d92e3525239e699813ace92b7850f727d9e2a6259d" Nov 03 02:59:50.649535 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:50.649495 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f48790376c3b6049b37350d92e3525239e699813ace92b7850f727d9e2a6259d"} err="failed to get container status \"f48790376c3b6049b37350d92e3525239e699813ace92b7850f727d9e2a6259d\": rpc error: code = NotFound desc = could not find container \"f48790376c3b6049b37350d92e3525239e699813ace92b7850f727d9e2a6259d\": container with ID starting with f48790376c3b6049b37350d92e3525239e699813ace92b7850f727d9e2a6259d not found: ID does not exist" Nov 03 02:59:50.785572 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:50.785551 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4"] Nov 03 02:59:50.789481 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:50.789453 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-95152a-openshift-default-66bd9d6bdf-grnr4"] Nov 03 02:59:50.849766 ip-10-0-140-140 kubenswrapper[5605]: I1103 02:59:50.849743 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="264e26fa-8062-4f70-86c8-3ba19e1c4384" path="/var/lib/kubelet/pods/264e26fa-8062-4f70-86c8-3ba19e1c4384/volumes" Nov 03 02:59:53.913559 ip-10-0-140-140 kubenswrapper[5605]: E1103 02:59:53.913514 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:00:00.165017 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.164985 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894580-79l9g"] Nov 03 03:00:00.165759 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.165739 5605 cpu_manager.go:414] "containerMap: removing container" podUID="1c3839da-096c-45ec-a1d1-4501d3c2457f" containerName="payload-processing" Nov 03 03:00:00.165864 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.165761 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c3839da-096c-45ec-a1d1-4501d3c2457f" containerName="payload-processing" Nov 03 03:00:00.165864 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.165782 5605 cpu_manager.go:414] "containerMap: removing container" podUID="264e26fa-8062-4f70-86c8-3ba19e1c4384" containerName="istio-proxy" Nov 03 03:00:00.165864 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.165789 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="264e26fa-8062-4f70-86c8-3ba19e1c4384" containerName="istio-proxy" Nov 03 03:00:00.165864 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.165855 5605 cpu_manager.go:414] "containerMap: removing container" podUID="7ea0b63b-c7c4-4d08-b281-e29e6c139462" containerName="revoke-keys" Nov 03 03:00:00.165864 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.165864 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ea0b63b-c7c4-4d08-b281-e29e6c139462" containerName="revoke-keys" Nov 03 03:00:00.166124 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.166004 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ea0b63b-c7c4-4d08-b281-e29e6c139462" containerName="revoke-keys" Nov 03 03:00:00.166124 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.166017 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="264e26fa-8062-4f70-86c8-3ba19e1c4384" containerName="istio-proxy" Nov 03 03:00:00.166124 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.166029 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c3839da-096c-45ec-a1d1-4501d3c2457f" containerName="payload-processing" Nov 03 03:00:00.166124 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.166040 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="264e26fa-8062-4f70-86c8-3ba19e1c4384" containerName="istio-proxy" Nov 03 03:00:00.171021 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.170999 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894580-z9gsn"] Nov 03 03:00:00.171139 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.171114 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894580-79l9g" Nov 03 03:00:00.171602 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.171581 5605 cpu_manager.go:414] "containerMap: removing container" podUID="264e26fa-8062-4f70-86c8-3ba19e1c4384" containerName="istio-proxy" Nov 03 03:00:00.171602 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.171602 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="264e26fa-8062-4f70-86c8-3ba19e1c4384" containerName="istio-proxy" Nov 03 03:00:00.174713 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.174697 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894580-5blv2"] Nov 03 03:00:00.174844 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.174826 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894580-z9gsn" Nov 03 03:00:00.178029 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.178012 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894580-5blv2" Nov 03 03:00:00.204735 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.204711 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75117649-d9a9-4cde-a8ff-a29010aec224-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894580-79l9g\" (UID: \"75117649-d9a9-4cde-a8ff-a29010aec224\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894580-79l9g" Nov 03 03:00:00.204848 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.204768 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qskqq\" (UniqueName: \"kubernetes.io/projected/6f5b2e0e-94ec-44be-bde2-364e1022e4d1-kube-api-access-qskqq\") pod \"maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894580-z9gsn\" (UID: \"6f5b2e0e-94ec-44be-bde2-364e1022e4d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894580-z9gsn" Nov 03 03:00:00.204926 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.204908 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lhg5\" (UniqueName: \"kubernetes.io/projected/b4b45e3d-f2ba-4a40-b4f6-d2fe279e35d6-kube-api-access-5lhg5\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894580-5blv2\" (UID: \"b4b45e3d-f2ba-4a40-b4f6-d2fe279e35d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894580-5blv2" Nov 03 03:00:00.204968 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.204952 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f5b2e0e-94ec-44be-bde2-364e1022e4d1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894580-z9gsn\" (UID: \"6f5b2e0e-94ec-44be-bde2-364e1022e4d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894580-z9gsn" Nov 03 03:00:00.205043 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.205022 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92fl6\" (UniqueName: \"kubernetes.io/projected/75117649-d9a9-4cde-a8ff-a29010aec224-kube-api-access-92fl6\") pod \"maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894580-79l9g\" (UID: \"75117649-d9a9-4cde-a8ff-a29010aec224\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894580-79l9g" Nov 03 03:00:00.205179 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.205070 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4b45e3d-f2ba-4a40-b4f6-d2fe279e35d6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894580-5blv2\" (UID: \"b4b45e3d-f2ba-4a40-b4f6-d2fe279e35d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894580-5blv2" Nov 03 03:00:00.211403 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.211382 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894580-5jvt2"] Nov 03 03:00:00.214895 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.214877 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894580-5d7jp"] Nov 03 03:00:00.214994 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.214976 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894580-5jvt2" Nov 03 03:00:00.218138 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.218123 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894580-9bnc2"] Nov 03 03:00:00.218252 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.218235 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894580-5d7jp" Nov 03 03:00:00.221573 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.221465 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894580-9t987"] Nov 03 03:00:00.222059 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.222040 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894580-9bnc2" Nov 03 03:00:00.225099 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.225077 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894580-fvqqd"] Nov 03 03:00:00.225204 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.225142 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894580-9t987" Nov 03 03:00:00.228786 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.228771 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894580-qx88p"] Nov 03 03:00:00.228880 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.228868 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894580-fvqqd" Nov 03 03:00:00.232352 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.232336 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894580-qx88p" Nov 03 03:00:00.256230 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.256210 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:00:00.256320 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.256210 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:00:00.256320 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.256306 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 03:00:00.258398 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.258381 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894580-wz5ql"] Nov 03 03:00:00.262879 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.262862 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc14f6-29894580-vhmch"] Nov 03 03:00:00.262970 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.262957 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894580-wz5ql" Nov 03 03:00:00.266723 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.266707 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894580-z9gsn"] Nov 03 03:00:00.266827 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.266817 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc14f6-29894580-vhmch" Nov 03 03:00:00.269200 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.269180 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894580-79l9g"] Nov 03 03:00:00.271313 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.271294 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894580-5jvt2"] Nov 03 03:00:00.273205 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.273185 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894580-5d7jp"] Nov 03 03:00:00.307019 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.306994 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmtjv\" (UniqueName: \"kubernetes.io/projected/ce5ce822-475e-40bd-9658-cd31fc3e8eb1-kube-api-access-cmtjv\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894580-5jvt2\" (UID: \"ce5ce822-475e-40bd-9658-cd31fc3e8eb1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894580-5jvt2" Nov 03 03:00:00.307141 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.307033 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/441073fd-d6a9-4c6b-8cbe-f30638ca23ec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894580-wz5ql\" (UID: \"441073fd-d6a9-4c6b-8cbe-f30638ca23ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894580-wz5ql" Nov 03 03:00:00.307141 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.307066 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qskqq\" (UniqueName: \"kubernetes.io/projected/6f5b2e0e-94ec-44be-bde2-364e1022e4d1-kube-api-access-qskqq\") pod \"maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894580-z9gsn\" (UID: \"6f5b2e0e-94ec-44be-bde2-364e1022e4d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894580-z9gsn" Nov 03 03:00:00.307141 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.307109 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j9x7\" (UniqueName: \"kubernetes.io/projected/ad92a9ca-ff2e-4e29-a984-2322317dc196-kube-api-access-4j9x7\") pod \"maas-api-key-cleanup-e2e-worker-w2-521f1f-29894580-fvqqd\" (UID: \"ad92a9ca-ff2e-4e29-a984-2322317dc196\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894580-fvqqd" Nov 03 03:00:00.307314 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.307145 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad92a9ca-ff2e-4e29-a984-2322317dc196-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-521f1f-29894580-fvqqd\" (UID: \"ad92a9ca-ff2e-4e29-a984-2322317dc196\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894580-fvqqd" Nov 03 03:00:00.307314 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.307179 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-858hr\" (UniqueName: \"kubernetes.io/projected/c0796b4a-9e93-4cfb-b327-14e788b9dead-kube-api-access-858hr\") pod \"maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894580-qx88p\" (UID: \"c0796b4a-9e93-4cfb-b327-14e788b9dead\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894580-qx88p" Nov 03 03:00:00.307314 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.307207 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1684bfe4-f5b9-4b0a-ac70-578052e13b80-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-dc14f6-29894580-vhmch\" (UID: \"1684bfe4-f5b9-4b0a-ac70-578052e13b80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc14f6-29894580-vhmch" Nov 03 03:00:00.307314 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.307255 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0796b4a-9e93-4cfb-b327-14e788b9dead-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894580-qx88p\" (UID: \"c0796b4a-9e93-4cfb-b327-14e788b9dead\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894580-qx88p" Nov 03 03:00:00.307314 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.307286 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkxzw\" (UniqueName: \"kubernetes.io/projected/63ff2cb3-cbff-4dad-83d6-2f63cec4d1a3-kube-api-access-xkxzw\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894580-9t987\" (UID: \"63ff2cb3-cbff-4dad-83d6-2f63cec4d1a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894580-9t987" Nov 03 03:00:00.307612 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.307326 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scpbp\" (UniqueName: \"kubernetes.io/projected/1684bfe4-f5b9-4b0a-ac70-578052e13b80-kube-api-access-scpbp\") pod \"maas-api-key-cleanup-e2e-extmod-b-dc14f6-29894580-vhmch\" (UID: \"1684bfe4-f5b9-4b0a-ac70-578052e13b80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc14f6-29894580-vhmch" Nov 03 03:00:00.307612 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.307394 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba3c3c2b-2350-4bd1-b4c4-6010e888f7e0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894580-9bnc2\" (UID: \"ba3c3c2b-2350-4bd1-b4c4-6010e888f7e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894580-9bnc2" Nov 03 03:00:00.307612 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.307432 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lhg5\" (UniqueName: \"kubernetes.io/projected/b4b45e3d-f2ba-4a40-b4f6-d2fe279e35d6-kube-api-access-5lhg5\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894580-5blv2\" (UID: \"b4b45e3d-f2ba-4a40-b4f6-d2fe279e35d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894580-5blv2" Nov 03 03:00:00.307612 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.307466 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63ff2cb3-cbff-4dad-83d6-2f63cec4d1a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894580-9t987\" (UID: \"63ff2cb3-cbff-4dad-83d6-2f63cec4d1a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894580-9t987" Nov 03 03:00:00.307612 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.307500 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf66k\" (UniqueName: \"kubernetes.io/projected/ba3c3c2b-2350-4bd1-b4c4-6010e888f7e0-kube-api-access-lf66k\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894580-9bnc2\" (UID: \"ba3c3c2b-2350-4bd1-b4c4-6010e888f7e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894580-9bnc2" Nov 03 03:00:00.307612 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.307529 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvslz\" (UniqueName: \"kubernetes.io/projected/441073fd-d6a9-4c6b-8cbe-f30638ca23ec-kube-api-access-jvslz\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894580-wz5ql\" (UID: \"441073fd-d6a9-4c6b-8cbe-f30638ca23ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894580-wz5ql" Nov 03 03:00:00.307612 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.307566 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f5b2e0e-94ec-44be-bde2-364e1022e4d1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894580-z9gsn\" (UID: \"6f5b2e0e-94ec-44be-bde2-364e1022e4d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894580-z9gsn" Nov 03 03:00:00.307612 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.307606 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92fl6\" (UniqueName: \"kubernetes.io/projected/75117649-d9a9-4cde-a8ff-a29010aec224-kube-api-access-92fl6\") pod \"maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894580-79l9g\" (UID: \"75117649-d9a9-4cde-a8ff-a29010aec224\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894580-79l9g" Nov 03 03:00:00.307996 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.307634 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gts6w\" (UniqueName: \"kubernetes.io/projected/083f54a5-48ba-46e3-ab36-2bfaec4a3c9f-kube-api-access-gts6w\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894580-5d7jp\" (UID: \"083f54a5-48ba-46e3-ab36-2bfaec4a3c9f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894580-5d7jp" Nov 03 03:00:00.307996 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.307662 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce5ce822-475e-40bd-9658-cd31fc3e8eb1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894580-5jvt2\" (UID: \"ce5ce822-475e-40bd-9658-cd31fc3e8eb1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894580-5jvt2" Nov 03 03:00:00.307996 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.307707 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4b45e3d-f2ba-4a40-b4f6-d2fe279e35d6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894580-5blv2\" (UID: \"b4b45e3d-f2ba-4a40-b4f6-d2fe279e35d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894580-5blv2" Nov 03 03:00:00.307996 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.307789 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75117649-d9a9-4cde-a8ff-a29010aec224-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894580-79l9g\" (UID: \"75117649-d9a9-4cde-a8ff-a29010aec224\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894580-79l9g" Nov 03 03:00:00.307996 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.307861 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/083f54a5-48ba-46e3-ab36-2bfaec4a3c9f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894580-5d7jp\" (UID: \"083f54a5-48ba-46e3-ab36-2bfaec4a3c9f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894580-5d7jp" Nov 03 03:00:00.309044 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.309021 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75117649-d9a9-4cde-a8ff-a29010aec224-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894580-79l9g\" (UID: \"75117649-d9a9-4cde-a8ff-a29010aec224\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894580-79l9g" Nov 03 03:00:00.309219 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.309024 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f5b2e0e-94ec-44be-bde2-364e1022e4d1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894580-z9gsn\" (UID: \"6f5b2e0e-94ec-44be-bde2-364e1022e4d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894580-z9gsn" Nov 03 03:00:00.309219 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.309127 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894580-9t987"] Nov 03 03:00:00.309559 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.309532 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4b45e3d-f2ba-4a40-b4f6-d2fe279e35d6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894580-5blv2\" (UID: \"b4b45e3d-f2ba-4a40-b4f6-d2fe279e35d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894580-5blv2" Nov 03 03:00:00.363392 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.362248 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894580-fvqqd"] Nov 03 03:00:00.366590 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.366563 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92fl6\" (UniqueName: \"kubernetes.io/projected/75117649-d9a9-4cde-a8ff-a29010aec224-kube-api-access-92fl6\") pod \"maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894580-79l9g\" (UID: \"75117649-d9a9-4cde-a8ff-a29010aec224\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894580-79l9g" Nov 03 03:00:00.367044 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.367010 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894580-wz5ql"] Nov 03 03:00:00.374039 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.374003 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qskqq\" (UniqueName: \"kubernetes.io/projected/6f5b2e0e-94ec-44be-bde2-364e1022e4d1-kube-api-access-qskqq\") pod \"maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894580-z9gsn\" (UID: \"6f5b2e0e-94ec-44be-bde2-364e1022e4d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894580-z9gsn" Nov 03 03:00:00.380781 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.380757 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lhg5\" (UniqueName: \"kubernetes.io/projected/b4b45e3d-f2ba-4a40-b4f6-d2fe279e35d6-kube-api-access-5lhg5\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894580-5blv2\" (UID: \"b4b45e3d-f2ba-4a40-b4f6-d2fe279e35d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894580-5blv2" Nov 03 03:00:00.381189 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.381165 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894580-5blv2"] Nov 03 03:00:00.392047 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.392021 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894580-qx88p"] Nov 03 03:00:00.397175 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.397148 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894580-9bnc2"] Nov 03 03:00:00.409848 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.409820 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/083f54a5-48ba-46e3-ab36-2bfaec4a3c9f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894580-5d7jp\" (UID: \"083f54a5-48ba-46e3-ab36-2bfaec4a3c9f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894580-5d7jp" Nov 03 03:00:00.410002 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.409873 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmtjv\" (UniqueName: \"kubernetes.io/projected/ce5ce822-475e-40bd-9658-cd31fc3e8eb1-kube-api-access-cmtjv\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894580-5jvt2\" (UID: \"ce5ce822-475e-40bd-9658-cd31fc3e8eb1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894580-5jvt2" Nov 03 03:00:00.410002 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.409898 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/441073fd-d6a9-4c6b-8cbe-f30638ca23ec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894580-wz5ql\" (UID: \"441073fd-d6a9-4c6b-8cbe-f30638ca23ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894580-wz5ql" Nov 03 03:00:00.410002 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.409933 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j9x7\" (UniqueName: \"kubernetes.io/projected/ad92a9ca-ff2e-4e29-a984-2322317dc196-kube-api-access-4j9x7\") pod \"maas-api-key-cleanup-e2e-worker-w2-521f1f-29894580-fvqqd\" (UID: \"ad92a9ca-ff2e-4e29-a984-2322317dc196\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894580-fvqqd" Nov 03 03:00:00.410002 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.409964 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad92a9ca-ff2e-4e29-a984-2322317dc196-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-521f1f-29894580-fvqqd\" (UID: \"ad92a9ca-ff2e-4e29-a984-2322317dc196\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894580-fvqqd" Nov 03 03:00:00.410002 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.409993 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-858hr\" (UniqueName: \"kubernetes.io/projected/c0796b4a-9e93-4cfb-b327-14e788b9dead-kube-api-access-858hr\") pod \"maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894580-qx88p\" (UID: \"c0796b4a-9e93-4cfb-b327-14e788b9dead\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894580-qx88p" Nov 03 03:00:00.410296 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.410017 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1684bfe4-f5b9-4b0a-ac70-578052e13b80-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-dc14f6-29894580-vhmch\" (UID: \"1684bfe4-f5b9-4b0a-ac70-578052e13b80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc14f6-29894580-vhmch" Nov 03 03:00:00.410296 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.410056 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0796b4a-9e93-4cfb-b327-14e788b9dead-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894580-qx88p\" (UID: \"c0796b4a-9e93-4cfb-b327-14e788b9dead\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894580-qx88p" Nov 03 03:00:00.410296 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.410081 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkxzw\" (UniqueName: \"kubernetes.io/projected/63ff2cb3-cbff-4dad-83d6-2f63cec4d1a3-kube-api-access-xkxzw\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894580-9t987\" (UID: \"63ff2cb3-cbff-4dad-83d6-2f63cec4d1a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894580-9t987" Nov 03 03:00:00.410296 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.410107 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scpbp\" (UniqueName: \"kubernetes.io/projected/1684bfe4-f5b9-4b0a-ac70-578052e13b80-kube-api-access-scpbp\") pod \"maas-api-key-cleanup-e2e-extmod-b-dc14f6-29894580-vhmch\" (UID: \"1684bfe4-f5b9-4b0a-ac70-578052e13b80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc14f6-29894580-vhmch" Nov 03 03:00:00.410296 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.410151 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba3c3c2b-2350-4bd1-b4c4-6010e888f7e0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894580-9bnc2\" (UID: \"ba3c3c2b-2350-4bd1-b4c4-6010e888f7e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894580-9bnc2" Nov 03 03:00:00.410296 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.410188 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63ff2cb3-cbff-4dad-83d6-2f63cec4d1a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894580-9t987\" (UID: \"63ff2cb3-cbff-4dad-83d6-2f63cec4d1a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894580-9t987" Nov 03 03:00:00.410296 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.410216 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf66k\" (UniqueName: \"kubernetes.io/projected/ba3c3c2b-2350-4bd1-b4c4-6010e888f7e0-kube-api-access-lf66k\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894580-9bnc2\" (UID: \"ba3c3c2b-2350-4bd1-b4c4-6010e888f7e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894580-9bnc2" Nov 03 03:00:00.410296 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.410242 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvslz\" (UniqueName: \"kubernetes.io/projected/441073fd-d6a9-4c6b-8cbe-f30638ca23ec-kube-api-access-jvslz\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894580-wz5ql\" (UID: \"441073fd-d6a9-4c6b-8cbe-f30638ca23ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894580-wz5ql" Nov 03 03:00:00.410296 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.410287 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gts6w\" (UniqueName: \"kubernetes.io/projected/083f54a5-48ba-46e3-ab36-2bfaec4a3c9f-kube-api-access-gts6w\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894580-5d7jp\" (UID: \"083f54a5-48ba-46e3-ab36-2bfaec4a3c9f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894580-5d7jp" Nov 03 03:00:00.410915 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.410311 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce5ce822-475e-40bd-9658-cd31fc3e8eb1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894580-5jvt2\" (UID: \"ce5ce822-475e-40bd-9658-cd31fc3e8eb1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894580-5jvt2" Nov 03 03:00:00.411206 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.411184 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0796b4a-9e93-4cfb-b327-14e788b9dead-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894580-qx88p\" (UID: \"c0796b4a-9e93-4cfb-b327-14e788b9dead\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894580-qx88p" Nov 03 03:00:00.411338 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.411184 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce5ce822-475e-40bd-9658-cd31fc3e8eb1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894580-5jvt2\" (UID: \"ce5ce822-475e-40bd-9658-cd31fc3e8eb1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894580-5jvt2" Nov 03 03:00:00.411572 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.411546 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/083f54a5-48ba-46e3-ab36-2bfaec4a3c9f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894580-5d7jp\" (UID: \"083f54a5-48ba-46e3-ab36-2bfaec4a3c9f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894580-5d7jp" Nov 03 03:00:00.411803 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.411775 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1684bfe4-f5b9-4b0a-ac70-578052e13b80-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-dc14f6-29894580-vhmch\" (UID: \"1684bfe4-f5b9-4b0a-ac70-578052e13b80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc14f6-29894580-vhmch" Nov 03 03:00:00.411898 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.411778 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63ff2cb3-cbff-4dad-83d6-2f63cec4d1a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894580-9t987\" (UID: \"63ff2cb3-cbff-4dad-83d6-2f63cec4d1a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894580-9t987" Nov 03 03:00:00.411898 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.411866 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad92a9ca-ff2e-4e29-a984-2322317dc196-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-521f1f-29894580-fvqqd\" (UID: \"ad92a9ca-ff2e-4e29-a984-2322317dc196\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894580-fvqqd" Nov 03 03:00:00.412068 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.412043 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/441073fd-d6a9-4c6b-8cbe-f30638ca23ec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894580-wz5ql\" (UID: \"441073fd-d6a9-4c6b-8cbe-f30638ca23ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894580-wz5ql" Nov 03 03:00:00.412185 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.412171 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba3c3c2b-2350-4bd1-b4c4-6010e888f7e0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894580-9bnc2\" (UID: \"ba3c3c2b-2350-4bd1-b4c4-6010e888f7e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894580-9bnc2" Nov 03 03:00:00.423926 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.423865 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc14f6-29894580-vhmch"] Nov 03 03:00:00.487009 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.486805 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894580-79l9g" Nov 03 03:00:00.512035 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.501876 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894580-z9gsn" Nov 03 03:00:00.512035 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.504611 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894580-5blv2" Nov 03 03:00:00.522395 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.514452 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvslz\" (UniqueName: \"kubernetes.io/projected/441073fd-d6a9-4c6b-8cbe-f30638ca23ec-kube-api-access-jvslz\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894580-wz5ql\" (UID: \"441073fd-d6a9-4c6b-8cbe-f30638ca23ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894580-wz5ql" Nov 03 03:00:00.522395 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.515163 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gts6w\" (UniqueName: \"kubernetes.io/projected/083f54a5-48ba-46e3-ab36-2bfaec4a3c9f-kube-api-access-gts6w\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894580-5d7jp\" (UID: \"083f54a5-48ba-46e3-ab36-2bfaec4a3c9f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894580-5d7jp" Nov 03 03:00:00.522395 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.518204 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkxzw\" (UniqueName: \"kubernetes.io/projected/63ff2cb3-cbff-4dad-83d6-2f63cec4d1a3-kube-api-access-xkxzw\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894580-9t987\" (UID: \"63ff2cb3-cbff-4dad-83d6-2f63cec4d1a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894580-9t987" Nov 03 03:00:00.522395 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.519111 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf66k\" (UniqueName: \"kubernetes.io/projected/ba3c3c2b-2350-4bd1-b4c4-6010e888f7e0-kube-api-access-lf66k\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894580-9bnc2\" (UID: \"ba3c3c2b-2350-4bd1-b4c4-6010e888f7e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894580-9bnc2" Nov 03 03:00:00.522395 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.520074 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmtjv\" (UniqueName: \"kubernetes.io/projected/ce5ce822-475e-40bd-9658-cd31fc3e8eb1-kube-api-access-cmtjv\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894580-5jvt2\" (UID: \"ce5ce822-475e-40bd-9658-cd31fc3e8eb1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894580-5jvt2" Nov 03 03:00:00.522395 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.522309 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j9x7\" (UniqueName: \"kubernetes.io/projected/ad92a9ca-ff2e-4e29-a984-2322317dc196-kube-api-access-4j9x7\") pod \"maas-api-key-cleanup-e2e-worker-w2-521f1f-29894580-fvqqd\" (UID: \"ad92a9ca-ff2e-4e29-a984-2322317dc196\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894580-fvqqd" Nov 03 03:00:00.528039 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.525816 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-858hr\" (UniqueName: \"kubernetes.io/projected/c0796b4a-9e93-4cfb-b327-14e788b9dead-kube-api-access-858hr\") pod \"maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894580-qx88p\" (UID: \"c0796b4a-9e93-4cfb-b327-14e788b9dead\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894580-qx88p" Nov 03 03:00:00.528039 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.526453 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scpbp\" (UniqueName: \"kubernetes.io/projected/1684bfe4-f5b9-4b0a-ac70-578052e13b80-kube-api-access-scpbp\") pod \"maas-api-key-cleanup-e2e-extmod-b-dc14f6-29894580-vhmch\" (UID: \"1684bfe4-f5b9-4b0a-ac70-578052e13b80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc14f6-29894580-vhmch" Nov 03 03:00:00.530020 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.529995 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894580-5jvt2" Nov 03 03:00:00.537490 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.537468 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894580-5d7jp" Nov 03 03:00:00.546068 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.545888 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894580-9bnc2" Nov 03 03:00:00.553152 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.552995 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894580-9t987" Nov 03 03:00:00.560877 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.560797 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894580-fvqqd" Nov 03 03:00:00.568951 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.567925 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894580-qx88p" Nov 03 03:00:00.577293 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.577082 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894580-wz5ql" Nov 03 03:00:00.659368 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.590600 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc14f6-29894580-vhmch" Nov 03 03:00:00.962570 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:00.962513 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894580-79l9g"] Nov 03 03:00:00.967857 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:00:00.967827 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75117649_d9a9_4cde_a8ff_a29010aec224.slice/crio-d8d4fe7990c9ebcdad17b5772f9859b143f4f47cfe03ab2b85e9a93998281a42 WatchSource:0}: Error finding container d8d4fe7990c9ebcdad17b5772f9859b143f4f47cfe03ab2b85e9a93998281a42: Status 404 returned error can't find the container with id d8d4fe7990c9ebcdad17b5772f9859b143f4f47cfe03ab2b85e9a93998281a42 Nov 03 03:00:01.153317 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:01.153284 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894580-fvqqd"] Nov 03 03:00:01.159899 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:00:01.159868 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad92a9ca_ff2e_4e29_a984_2322317dc196.slice/crio-c174cb8c589512efb6e1309cb0b9c7dc52cb1e4dd64792dd98ab8af56b5b273b WatchSource:0}: Error finding container c174cb8c589512efb6e1309cb0b9c7dc52cb1e4dd64792dd98ab8af56b5b273b: Status 404 returned error can't find the container with id c174cb8c589512efb6e1309cb0b9c7dc52cb1e4dd64792dd98ab8af56b5b273b Nov 03 03:00:01.202943 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:01.202885 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894580-qx88p"] Nov 03 03:00:01.205588 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:01.205570 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894580-wz5ql"] Nov 03 03:00:01.207414 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:00:01.207389 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0796b4a_9e93_4cfb_b327_14e788b9dead.slice/crio-8ed9f73f3355433ab10d654d895351b211e18d30d54b37203ed01e065318f89e WatchSource:0}: Error finding container 8ed9f73f3355433ab10d654d895351b211e18d30d54b37203ed01e065318f89e: Status 404 returned error can't find the container with id 8ed9f73f3355433ab10d654d895351b211e18d30d54b37203ed01e065318f89e Nov 03 03:00:01.210636 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:00:01.210613 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod441073fd_d6a9_4c6b_8cbe_f30638ca23ec.slice/crio-745a92d3187878cb4b3ff88d57f62b1253f4f426bdb3a55f888abcbef8820adf WatchSource:0}: Error finding container 745a92d3187878cb4b3ff88d57f62b1253f4f426bdb3a55f888abcbef8820adf: Status 404 returned error can't find the container with id 745a92d3187878cb4b3ff88d57f62b1253f4f426bdb3a55f888abcbef8820adf Nov 03 03:00:01.220592 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:01.220563 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894580-z9gsn"] Nov 03 03:00:01.226776 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:00:01.226753 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f5b2e0e_94ec_44be_bde2_364e1022e4d1.slice/crio-d1a0f47f730a293bfbc9610bfeb8bf27fbb4a9daab71d4b19f82e25e58a3d02d WatchSource:0}: Error finding container d1a0f47f730a293bfbc9610bfeb8bf27fbb4a9daab71d4b19f82e25e58a3d02d: Status 404 returned error can't find the container with id d1a0f47f730a293bfbc9610bfeb8bf27fbb4a9daab71d4b19f82e25e58a3d02d Nov 03 03:00:01.244387 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:01.244334 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894580-5blv2"] Nov 03 03:00:01.247533 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:01.247510 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc14f6-29894580-vhmch"] Nov 03 03:00:01.247824 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:00:01.247758 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4b45e3d_f2ba_4a40_b4f6_d2fe279e35d6.slice/crio-fbc9b4a93006c64e4634996525f3cac1474837d884578bfb1e4ec2c7dd90ae7d WatchSource:0}: Error finding container fbc9b4a93006c64e4634996525f3cac1474837d884578bfb1e4ec2c7dd90ae7d: Status 404 returned error can't find the container with id fbc9b4a93006c64e4634996525f3cac1474837d884578bfb1e4ec2c7dd90ae7d Nov 03 03:00:01.249028 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:00:01.249002 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1684bfe4_f5b9_4b0a_ac70_578052e13b80.slice/crio-0a3b95d975852a5059ca4be5102547d867ba70317c0962042759b985af17ea4c WatchSource:0}: Error finding container 0a3b95d975852a5059ca4be5102547d867ba70317c0962042759b985af17ea4c: Status 404 returned error can't find the container with id 0a3b95d975852a5059ca4be5102547d867ba70317c0962042759b985af17ea4c Nov 03 03:00:01.255601 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:01.255385 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894580-5d7jp"] Nov 03 03:00:01.305144 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:01.305121 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894580-5jvt2"] Nov 03 03:00:01.339592 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:01.339566 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894580-9bnc2"] Nov 03 03:00:01.365951 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:01.365930 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894580-9t987"] Nov 03 03:00:01.377093 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:00:01.377065 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63ff2cb3_cbff_4dad_83d6_2f63cec4d1a3.slice/crio-dfc7be7c50721ff46020a42a5e15c83bcb0f39a6bca00514392a6d1eea5924e9 WatchSource:0}: Error finding container dfc7be7c50721ff46020a42a5e15c83bcb0f39a6bca00514392a6d1eea5924e9: Status 404 returned error can't find the container with id dfc7be7c50721ff46020a42a5e15c83bcb0f39a6bca00514392a6d1eea5924e9 Nov 03 03:00:01.733471 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:01.733434 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894580-qx88p" event={"ID":"c0796b4a-9e93-4cfb-b327-14e788b9dead","Type":"ContainerStarted","Data":"8ed9f73f3355433ab10d654d895351b211e18d30d54b37203ed01e065318f89e"} Nov 03 03:00:01.734680 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:01.734655 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc14f6-29894580-vhmch" event={"ID":"1684bfe4-f5b9-4b0a-ac70-578052e13b80","Type":"ContainerStarted","Data":"0a3b95d975852a5059ca4be5102547d867ba70317c0962042759b985af17ea4c"} Nov 03 03:00:01.735848 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:01.735820 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894580-fvqqd" event={"ID":"ad92a9ca-ff2e-4e29-a984-2322317dc196","Type":"ContainerStarted","Data":"c174cb8c589512efb6e1309cb0b9c7dc52cb1e4dd64792dd98ab8af56b5b273b"} Nov 03 03:00:01.736954 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:01.736929 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894580-9bnc2" event={"ID":"ba3c3c2b-2350-4bd1-b4c4-6010e888f7e0","Type":"ContainerStarted","Data":"b619500480c5d7b5dba218ca860bf558ba0c91a885ffd2f3e86118c2edf69683"} Nov 03 03:00:01.738074 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:01.738049 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894580-5jvt2" event={"ID":"ce5ce822-475e-40bd-9658-cd31fc3e8eb1","Type":"ContainerStarted","Data":"41f1029fb30e8457a2d3b661df8fe3cb158c7c7cd876fd43882efe7c925c6531"} Nov 03 03:00:01.739176 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:01.739158 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894580-9t987" event={"ID":"63ff2cb3-cbff-4dad-83d6-2f63cec4d1a3","Type":"ContainerStarted","Data":"dfc7be7c50721ff46020a42a5e15c83bcb0f39a6bca00514392a6d1eea5924e9"} Nov 03 03:00:01.740272 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:01.740250 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894580-z9gsn" event={"ID":"6f5b2e0e-94ec-44be-bde2-364e1022e4d1","Type":"ContainerStarted","Data":"d1a0f47f730a293bfbc9610bfeb8bf27fbb4a9daab71d4b19f82e25e58a3d02d"} Nov 03 03:00:01.741393 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:01.741368 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894580-5d7jp" event={"ID":"083f54a5-48ba-46e3-ab36-2bfaec4a3c9f","Type":"ContainerStarted","Data":"af807b2bcaf9a44dc5ecafa1f4872a3dcb99679c75507793758da28b8dc874e9"} Nov 03 03:00:01.742800 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:01.742780 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894580-79l9g" event={"ID":"75117649-d9a9-4cde-a8ff-a29010aec224","Type":"ContainerStarted","Data":"8e35b2e820264c7f0cf5a3ab9aea1a2a8798a6f26163ed8899467684b24bfec2"} Nov 03 03:00:01.742906 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:01.742816 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894580-79l9g" event={"ID":"75117649-d9a9-4cde-a8ff-a29010aec224","Type":"ContainerStarted","Data":"d8d4fe7990c9ebcdad17b5772f9859b143f4f47cfe03ab2b85e9a93998281a42"} Nov 03 03:00:01.743891 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:01.743874 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894580-wz5ql" event={"ID":"441073fd-d6a9-4c6b-8cbe-f30638ca23ec","Type":"ContainerStarted","Data":"745a92d3187878cb4b3ff88d57f62b1253f4f426bdb3a55f888abcbef8820adf"} Nov 03 03:00:01.744986 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:01.744966 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894580-5blv2" event={"ID":"b4b45e3d-f2ba-4a40-b4f6-d2fe279e35d6","Type":"ContainerStarted","Data":"fbc9b4a93006c64e4634996525f3cac1474837d884578bfb1e4ec2c7dd90ae7d"} Nov 03 03:00:01.851497 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:01.851414 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894580-79l9g" podStartSLOduration=1.851401725 podStartE2EDuration="1.851401725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:00:01.849070944 +0000 UTC m=+4079.108269098" watchObservedRunningTime="2026-11-03 03:00:01.851401725 +0000 UTC m=+4079.110599877" Nov 03 03:00:02.752432 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:02.752402 5605 generic.go:356] "Generic (PLEG): container finished" podID="083f54a5-48ba-46e3-ab36-2bfaec4a3c9f" containerID="8f5bbdae3435f7f822b105cd8205aa8a9c21840e78dd44bca087e6b175f15af6" exitCode=60 Nov 03 03:00:02.752822 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:02.752436 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894580-5d7jp" event={"ID":"083f54a5-48ba-46e3-ab36-2bfaec4a3c9f","Type":"ContainerDied","Data":"8f5bbdae3435f7f822b105cd8205aa8a9c21840e78dd44bca087e6b175f15af6"} Nov 03 03:00:02.752822 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:02.752755 5605 scope.go:122] "RemoveContainer" containerID="8f5bbdae3435f7f822b105cd8205aa8a9c21840e78dd44bca087e6b175f15af6" Nov 03 03:00:02.754199 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:02.754177 5605 generic.go:356] "Generic (PLEG): container finished" podID="75117649-d9a9-4cde-a8ff-a29010aec224" containerID="8e35b2e820264c7f0cf5a3ab9aea1a2a8798a6f26163ed8899467684b24bfec2" exitCode=60 Nov 03 03:00:02.754299 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:02.754240 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894580-79l9g" event={"ID":"75117649-d9a9-4cde-a8ff-a29010aec224","Type":"ContainerDied","Data":"8e35b2e820264c7f0cf5a3ab9aea1a2a8798a6f26163ed8899467684b24bfec2"} Nov 03 03:00:02.754488 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:02.754462 5605 scope.go:122] "RemoveContainer" containerID="8e35b2e820264c7f0cf5a3ab9aea1a2a8798a6f26163ed8899467684b24bfec2" Nov 03 03:00:02.765496 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:02.765474 5605 generic.go:356] "Generic (PLEG): container finished" podID="441073fd-d6a9-4c6b-8cbe-f30638ca23ec" containerID="92f86eb9937f9a2a127d4abb50e14ea4aeffa08edbc8e6885b241a3b41eda8be" exitCode=60 Nov 03 03:00:02.765603 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:02.765550 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894580-wz5ql" event={"ID":"441073fd-d6a9-4c6b-8cbe-f30638ca23ec","Type":"ContainerDied","Data":"92f86eb9937f9a2a127d4abb50e14ea4aeffa08edbc8e6885b241a3b41eda8be"} Nov 03 03:00:02.765746 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:02.765731 5605 scope.go:122] "RemoveContainer" containerID="92f86eb9937f9a2a127d4abb50e14ea4aeffa08edbc8e6885b241a3b41eda8be" Nov 03 03:00:02.767453 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:02.767429 5605 generic.go:356] "Generic (PLEG): container finished" podID="b4b45e3d-f2ba-4a40-b4f6-d2fe279e35d6" containerID="fa8cdaa2d50b6a7680e21228186e31fc3fff7e74305fc49d6c23f29225798703" exitCode=60 Nov 03 03:00:02.767564 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:02.767549 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894580-5blv2" event={"ID":"b4b45e3d-f2ba-4a40-b4f6-d2fe279e35d6","Type":"ContainerDied","Data":"fa8cdaa2d50b6a7680e21228186e31fc3fff7e74305fc49d6c23f29225798703"} Nov 03 03:00:02.767692 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:02.767675 5605 scope.go:122] "RemoveContainer" containerID="fa8cdaa2d50b6a7680e21228186e31fc3fff7e74305fc49d6c23f29225798703" Nov 03 03:00:02.769134 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:02.769115 5605 generic.go:356] "Generic (PLEG): container finished" podID="c0796b4a-9e93-4cfb-b327-14e788b9dead" containerID="7b1dd8459f2edb8a41b9d45af956f04810996300a16e8155b0afcffd52e3b6c0" exitCode=60 Nov 03 03:00:02.769240 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:02.769146 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894580-qx88p" event={"ID":"c0796b4a-9e93-4cfb-b327-14e788b9dead","Type":"ContainerDied","Data":"7b1dd8459f2edb8a41b9d45af956f04810996300a16e8155b0afcffd52e3b6c0"} Nov 03 03:00:02.769441 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:02.769413 5605 scope.go:122] "RemoveContainer" containerID="7b1dd8459f2edb8a41b9d45af956f04810996300a16e8155b0afcffd52e3b6c0" Nov 03 03:00:02.770665 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:02.770642 5605 generic.go:356] "Generic (PLEG): container finished" podID="1684bfe4-f5b9-4b0a-ac70-578052e13b80" containerID="bd751fb884a71ddab40cd1ae8c1c5fce8a1556f2e3517fe9b838e334d2a91f50" exitCode=60 Nov 03 03:00:02.770771 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:02.770720 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc14f6-29894580-vhmch" event={"ID":"1684bfe4-f5b9-4b0a-ac70-578052e13b80","Type":"ContainerDied","Data":"bd751fb884a71ddab40cd1ae8c1c5fce8a1556f2e3517fe9b838e334d2a91f50"} Nov 03 03:00:02.770960 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:02.770933 5605 scope.go:122] "RemoveContainer" containerID="bd751fb884a71ddab40cd1ae8c1c5fce8a1556f2e3517fe9b838e334d2a91f50" Nov 03 03:00:02.775565 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:02.775543 5605 generic.go:356] "Generic (PLEG): container finished" podID="ad92a9ca-ff2e-4e29-a984-2322317dc196" containerID="7d26c1c3de05c88886f7185305843d08dc9619e7049b57b8d2a7ecdb5c673f99" exitCode=60 Nov 03 03:00:02.775657 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:02.775622 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894580-fvqqd" event={"ID":"ad92a9ca-ff2e-4e29-a984-2322317dc196","Type":"ContainerDied","Data":"7d26c1c3de05c88886f7185305843d08dc9619e7049b57b8d2a7ecdb5c673f99"} Nov 03 03:00:02.775803 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:02.775788 5605 scope.go:122] "RemoveContainer" containerID="7d26c1c3de05c88886f7185305843d08dc9619e7049b57b8d2a7ecdb5c673f99" Nov 03 03:00:02.777298 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:02.777280 5605 generic.go:356] "Generic (PLEG): container finished" podID="ba3c3c2b-2350-4bd1-b4c4-6010e888f7e0" containerID="e9948ec83a9b1ff5c8711c52e5a572dcc48089f786f6c03799ea5b5c8454f61a" exitCode=60 Nov 03 03:00:02.777433 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:02.777342 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894580-9bnc2" event={"ID":"ba3c3c2b-2350-4bd1-b4c4-6010e888f7e0","Type":"ContainerDied","Data":"e9948ec83a9b1ff5c8711c52e5a572dcc48089f786f6c03799ea5b5c8454f61a"} Nov 03 03:00:02.777655 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:02.777638 5605 scope.go:122] "RemoveContainer" containerID="e9948ec83a9b1ff5c8711c52e5a572dcc48089f786f6c03799ea5b5c8454f61a" Nov 03 03:00:02.779083 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:02.779065 5605 generic.go:356] "Generic (PLEG): container finished" podID="ce5ce822-475e-40bd-9658-cd31fc3e8eb1" containerID="97e925ef5cb5a52ecadb0ea56cbee72223b90530aa870d71f2331ef11bce89ec" exitCode=60 Nov 03 03:00:02.779182 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:02.779123 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894580-5jvt2" event={"ID":"ce5ce822-475e-40bd-9658-cd31fc3e8eb1","Type":"ContainerDied","Data":"97e925ef5cb5a52ecadb0ea56cbee72223b90530aa870d71f2331ef11bce89ec"} Nov 03 03:00:02.779310 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:02.779296 5605 scope.go:122] "RemoveContainer" containerID="97e925ef5cb5a52ecadb0ea56cbee72223b90530aa870d71f2331ef11bce89ec" Nov 03 03:00:02.783117 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:02.783066 5605 generic.go:356] "Generic (PLEG): container finished" podID="63ff2cb3-cbff-4dad-83d6-2f63cec4d1a3" containerID="0614fe5ee8bc948187f005206e64f0b9cd0ab246779f377c575e0846720e8693" exitCode=60 Nov 03 03:00:02.783230 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:02.783198 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894580-9t987" event={"ID":"63ff2cb3-cbff-4dad-83d6-2f63cec4d1a3","Type":"ContainerDied","Data":"0614fe5ee8bc948187f005206e64f0b9cd0ab246779f377c575e0846720e8693"} Nov 03 03:00:02.783390 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:02.783346 5605 scope.go:122] "RemoveContainer" containerID="0614fe5ee8bc948187f005206e64f0b9cd0ab246779f377c575e0846720e8693" Nov 03 03:00:02.785112 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:02.785080 5605 generic.go:356] "Generic (PLEG): container finished" podID="6f5b2e0e-94ec-44be-bde2-364e1022e4d1" containerID="b0d14227b16f90057a4dd3d29fbc666355bbd36de88ff85b7a63cf91990050ff" exitCode=60 Nov 03 03:00:02.785193 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:02.785128 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894580-z9gsn" event={"ID":"6f5b2e0e-94ec-44be-bde2-364e1022e4d1","Type":"ContainerDied","Data":"b0d14227b16f90057a4dd3d29fbc666355bbd36de88ff85b7a63cf91990050ff"} Nov 03 03:00:02.785452 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:02.785324 5605 scope.go:122] "RemoveContainer" containerID="b0d14227b16f90057a4dd3d29fbc666355bbd36de88ff85b7a63cf91990050ff" Nov 03 03:00:03.793665 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:03.793590 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894580-fvqqd" event={"ID":"ad92a9ca-ff2e-4e29-a984-2322317dc196","Type":"ContainerStarted","Data":"c93ee7ae44aa44e3ea9e86ccdb3d24c924f5c61b65c2855efc6ec56e8d96eb78"} Nov 03 03:00:03.795540 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:03.795512 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894580-9bnc2" event={"ID":"ba3c3c2b-2350-4bd1-b4c4-6010e888f7e0","Type":"ContainerStarted","Data":"a4e1642058726a5e61ffc8dceb8b37561f93539487675ea741f67bd87174ba51"} Nov 03 03:00:03.797457 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:03.797435 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894580-5jvt2" event={"ID":"ce5ce822-475e-40bd-9658-cd31fc3e8eb1","Type":"ContainerStarted","Data":"a1ad9cf1df6925dea495c17a645fd7ad97a55610a20b469c0efaa71fa4e19c90"} Nov 03 03:00:03.799273 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:03.799254 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894580-9t987" event={"ID":"63ff2cb3-cbff-4dad-83d6-2f63cec4d1a3","Type":"ContainerStarted","Data":"fac2c2344db1e30b02454fb96316a1f8ff5eb773849eb393ca008c114c1de99a"} Nov 03 03:00:03.801088 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:03.801063 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894580-z9gsn" event={"ID":"6f5b2e0e-94ec-44be-bde2-364e1022e4d1","Type":"ContainerStarted","Data":"c73926ecd9f932709480c83569a48e051f0a8fe02047604a8bb37cab5eb1e685"} Nov 03 03:00:03.802837 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:03.802809 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894580-5d7jp" event={"ID":"083f54a5-48ba-46e3-ab36-2bfaec4a3c9f","Type":"ContainerStarted","Data":"119f73f3b817f14e1951ff58ab43f6429735d2afdeabbe8cee9899b2c23e5990"} Nov 03 03:00:03.804596 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:03.804570 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894580-79l9g" event={"ID":"75117649-d9a9-4cde-a8ff-a29010aec224","Type":"ContainerStarted","Data":"3532985bde0d80a87c5fbc2d184fa3c897723ef105e54a05c3153ad1d957a77e"} Nov 03 03:00:03.806484 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:03.806464 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894580-wz5ql" event={"ID":"441073fd-d6a9-4c6b-8cbe-f30638ca23ec","Type":"ContainerStarted","Data":"c33353e6f210940cdf475f6df158e56221ad49f6fc6c00f7c6b4bdf01fa34a69"} Nov 03 03:00:03.808250 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:03.808227 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894580-5blv2" event={"ID":"b4b45e3d-f2ba-4a40-b4f6-d2fe279e35d6","Type":"ContainerStarted","Data":"c825cf2848806a2ae92dc22adaf71b37cc719ef13ed2449f995a3b2ba3dfdca3"} Nov 03 03:00:03.810044 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:03.810020 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894580-qx88p" event={"ID":"c0796b4a-9e93-4cfb-b327-14e788b9dead","Type":"ContainerStarted","Data":"06bb57435ba4b791014b81b2eeceb1c1afa8534e47a60668500838d183e6b0a9"} Nov 03 03:00:03.811751 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:03.811731 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc14f6-29894580-vhmch" event={"ID":"1684bfe4-f5b9-4b0a-ac70-578052e13b80","Type":"ContainerStarted","Data":"bbcec6873d795a5d26bb5942374bb5e33e9a82a26aae9566f9f42c2f16862b14"} Nov 03 03:00:04.044814 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:04.044731 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894580-z9gsn" podStartSLOduration=4.044714879 podStartE2EDuration="4.044714879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:00:04.044104703 +0000 UTC m=+4081.303302863" watchObservedRunningTime="2026-11-03 03:00:04.044714879 +0000 UTC m=+4081.303913035" Nov 03 03:00:04.153052 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:04.153008 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894580-5jvt2" podStartSLOduration=4.152994963 podStartE2EDuration="4.152994963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:00:04.150968163 +0000 UTC m=+4081.410166314" watchObservedRunningTime="2026-11-03 03:00:04.152994963 +0000 UTC m=+4081.412193109" Nov 03 03:00:04.260629 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:04.260577 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894580-wz5ql" podStartSLOduration=4.260564226 podStartE2EDuration="4.260564226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:00:04.257935344 +0000 UTC m=+4081.517133489" watchObservedRunningTime="2026-11-03 03:00:04.260564226 +0000 UTC m=+4081.519762435" Nov 03 03:00:04.473392 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:04.473333 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894580-9t987" podStartSLOduration=4.473319754 podStartE2EDuration="4.473319754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:00:04.47173485 +0000 UTC m=+4081.730933002" watchObservedRunningTime="2026-11-03 03:00:04.473319754 +0000 UTC m=+4081.732517907" Nov 03 03:00:04.580570 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:04.580493 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894580-qx88p" podStartSLOduration=4.580479946 podStartE2EDuration="4.580479946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:00:04.57868976 +0000 UTC m=+4081.837887910" watchObservedRunningTime="2026-11-03 03:00:04.580479946 +0000 UTC m=+4081.839678100" Nov 03 03:00:04.686966 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:04.686923 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894580-fvqqd" podStartSLOduration=4.686911644 podStartE2EDuration="4.686911644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:00:04.684471401 +0000 UTC m=+4081.943669558" watchObservedRunningTime="2026-11-03 03:00:04.686911644 +0000 UTC m=+4081.946109805" Nov 03 03:00:04.789989 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:04.789945 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894580-9bnc2" podStartSLOduration=4.789928416 podStartE2EDuration="4.789928416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:00:04.789264377 +0000 UTC m=+4082.048462531" watchObservedRunningTime="2026-11-03 03:00:04.789928416 +0000 UTC m=+4082.049126615" Nov 03 03:00:04.819841 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:04.819808 5605 generic.go:356] "Generic (PLEG): container finished" podID="ce5ce822-475e-40bd-9658-cd31fc3e8eb1" containerID="a1ad9cf1df6925dea495c17a645fd7ad97a55610a20b469c0efaa71fa4e19c90" exitCode=60 Nov 03 03:00:04.820236 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:04.819897 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894580-5jvt2" event={"ID":"ce5ce822-475e-40bd-9658-cd31fc3e8eb1","Type":"ContainerDied","Data":"a1ad9cf1df6925dea495c17a645fd7ad97a55610a20b469c0efaa71fa4e19c90"} Nov 03 03:00:04.820236 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:04.819981 5605 scope.go:122] "RemoveContainer" containerID="97e925ef5cb5a52ecadb0ea56cbee72223b90530aa870d71f2331ef11bce89ec" Nov 03 03:00:04.820236 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:04.820103 5605 scope.go:122] "RemoveContainer" containerID="a1ad9cf1df6925dea495c17a645fd7ad97a55610a20b469c0efaa71fa4e19c90" Nov 03 03:00:04.820429 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:00:04.820405 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894580-5jvt2_odh-ai-gateway-infra(ce5ce822-475e-40bd-9658-cd31fc3e8eb1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894580-5jvt2" podUID="ce5ce822-475e-40bd-9658-cd31fc3e8eb1" Nov 03 03:00:04.821935 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:04.821912 5605 generic.go:356] "Generic (PLEG): container finished" podID="63ff2cb3-cbff-4dad-83d6-2f63cec4d1a3" containerID="fac2c2344db1e30b02454fb96316a1f8ff5eb773849eb393ca008c114c1de99a" exitCode=60 Nov 03 03:00:04.822043 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:04.821992 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894580-9t987" event={"ID":"63ff2cb3-cbff-4dad-83d6-2f63cec4d1a3","Type":"ContainerDied","Data":"fac2c2344db1e30b02454fb96316a1f8ff5eb773849eb393ca008c114c1de99a"} Nov 03 03:00:04.822169 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:04.822156 5605 scope.go:122] "RemoveContainer" containerID="fac2c2344db1e30b02454fb96316a1f8ff5eb773849eb393ca008c114c1de99a" Nov 03 03:00:04.822428 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:00:04.822400 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894580-9t987_odh-ai-gateway-infra(63ff2cb3-cbff-4dad-83d6-2f63cec4d1a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894580-9t987" podUID="63ff2cb3-cbff-4dad-83d6-2f63cec4d1a3" Nov 03 03:00:04.823744 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:04.823724 5605 generic.go:356] "Generic (PLEG): container finished" podID="6f5b2e0e-94ec-44be-bde2-364e1022e4d1" containerID="c73926ecd9f932709480c83569a48e051f0a8fe02047604a8bb37cab5eb1e685" exitCode=60 Nov 03 03:00:04.823832 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:04.823806 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894580-z9gsn" event={"ID":"6f5b2e0e-94ec-44be-bde2-364e1022e4d1","Type":"ContainerDied","Data":"c73926ecd9f932709480c83569a48e051f0a8fe02047604a8bb37cab5eb1e685"} Nov 03 03:00:04.823951 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:04.823937 5605 scope.go:122] "RemoveContainer" containerID="c73926ecd9f932709480c83569a48e051f0a8fe02047604a8bb37cab5eb1e685" Nov 03 03:00:04.824164 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:00:04.824144 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894580-z9gsn_odh-ai-gateway-infra(6f5b2e0e-94ec-44be-bde2-364e1022e4d1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894580-z9gsn" podUID="6f5b2e0e-94ec-44be-bde2-364e1022e4d1" Nov 03 03:00:04.825969 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:04.825946 5605 generic.go:356] "Generic (PLEG): container finished" podID="083f54a5-48ba-46e3-ab36-2bfaec4a3c9f" containerID="119f73f3b817f14e1951ff58ab43f6429735d2afdeabbe8cee9899b2c23e5990" exitCode=60 Nov 03 03:00:04.826067 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:04.826024 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894580-5d7jp" event={"ID":"083f54a5-48ba-46e3-ab36-2bfaec4a3c9f","Type":"ContainerDied","Data":"119f73f3b817f14e1951ff58ab43f6429735d2afdeabbe8cee9899b2c23e5990"} Nov 03 03:00:04.826203 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:04.826187 5605 scope.go:122] "RemoveContainer" containerID="119f73f3b817f14e1951ff58ab43f6429735d2afdeabbe8cee9899b2c23e5990" Nov 03 03:00:04.826445 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:00:04.826424 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894580-5d7jp_odh-ai-gateway-infra(083f54a5-48ba-46e3-ab36-2bfaec4a3c9f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894580-5d7jp" podUID="083f54a5-48ba-46e3-ab36-2bfaec4a3c9f" Nov 03 03:00:04.827943 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:04.827926 5605 generic.go:356] "Generic (PLEG): container finished" podID="75117649-d9a9-4cde-a8ff-a29010aec224" containerID="3532985bde0d80a87c5fbc2d184fa3c897723ef105e54a05c3153ad1d957a77e" exitCode=60 Nov 03 03:00:04.828023 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:04.828005 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894580-79l9g" event={"ID":"75117649-d9a9-4cde-a8ff-a29010aec224","Type":"ContainerDied","Data":"3532985bde0d80a87c5fbc2d184fa3c897723ef105e54a05c3153ad1d957a77e"} Nov 03 03:00:04.828238 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:04.828221 5605 scope.go:122] "RemoveContainer" containerID="3532985bde0d80a87c5fbc2d184fa3c897723ef105e54a05c3153ad1d957a77e" Nov 03 03:00:04.828489 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:00:04.828469 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894580-79l9g_odh-ai-gateway-infra(75117649-d9a9-4cde-a8ff-a29010aec224)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894580-79l9g" podUID="75117649-d9a9-4cde-a8ff-a29010aec224" Nov 03 03:00:04.829889 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:04.829868 5605 generic.go:356] "Generic (PLEG): container finished" podID="441073fd-d6a9-4c6b-8cbe-f30638ca23ec" containerID="c33353e6f210940cdf475f6df158e56221ad49f6fc6c00f7c6b4bdf01fa34a69" exitCode=60 Nov 03 03:00:04.829974 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:04.829952 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894580-wz5ql" event={"ID":"441073fd-d6a9-4c6b-8cbe-f30638ca23ec","Type":"ContainerDied","Data":"c33353e6f210940cdf475f6df158e56221ad49f6fc6c00f7c6b4bdf01fa34a69"} Nov 03 03:00:04.830088 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:04.830074 5605 scope.go:122] "RemoveContainer" containerID="c33353e6f210940cdf475f6df158e56221ad49f6fc6c00f7c6b4bdf01fa34a69" Nov 03 03:00:04.830294 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:00:04.830272 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894580-wz5ql_odh-ai-gateway-infra(441073fd-d6a9-4c6b-8cbe-f30638ca23ec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894580-wz5ql" podUID="441073fd-d6a9-4c6b-8cbe-f30638ca23ec" Nov 03 03:00:04.831724 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:04.831678 5605 generic.go:356] "Generic (PLEG): container finished" podID="b4b45e3d-f2ba-4a40-b4f6-d2fe279e35d6" containerID="c825cf2848806a2ae92dc22adaf71b37cc719ef13ed2449f995a3b2ba3dfdca3" exitCode=60 Nov 03 03:00:04.831797 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:04.831746 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894580-5blv2" event={"ID":"b4b45e3d-f2ba-4a40-b4f6-d2fe279e35d6","Type":"ContainerDied","Data":"c825cf2848806a2ae92dc22adaf71b37cc719ef13ed2449f995a3b2ba3dfdca3"} Nov 03 03:00:04.831897 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:04.831884 5605 scope.go:122] "RemoveContainer" containerID="c825cf2848806a2ae92dc22adaf71b37cc719ef13ed2449f995a3b2ba3dfdca3" Nov 03 03:00:04.832063 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:00:04.832048 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894580-5blv2_odh-ai-gateway-infra(b4b45e3d-f2ba-4a40-b4f6-d2fe279e35d6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894580-5blv2" podUID="b4b45e3d-f2ba-4a40-b4f6-d2fe279e35d6" Nov 03 03:00:04.833718 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:04.833698 5605 generic.go:356] "Generic (PLEG): container finished" podID="c0796b4a-9e93-4cfb-b327-14e788b9dead" containerID="06bb57435ba4b791014b81b2eeceb1c1afa8534e47a60668500838d183e6b0a9" exitCode=60 Nov 03 03:00:04.833823 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:04.833761 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894580-qx88p" event={"ID":"c0796b4a-9e93-4cfb-b327-14e788b9dead","Type":"ContainerDied","Data":"06bb57435ba4b791014b81b2eeceb1c1afa8534e47a60668500838d183e6b0a9"} Nov 03 03:00:04.834028 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:04.833996 5605 scope.go:122] "RemoveContainer" containerID="06bb57435ba4b791014b81b2eeceb1c1afa8534e47a60668500838d183e6b0a9" Nov 03 03:00:04.834387 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:00:04.834223 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894580-qx88p_odh-ai-gateway-infra(c0796b4a-9e93-4cfb-b327-14e788b9dead)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894580-qx88p" podUID="c0796b4a-9e93-4cfb-b327-14e788b9dead" Nov 03 03:00:04.834465 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:04.834400 5605 scope.go:122] "RemoveContainer" containerID="0614fe5ee8bc948187f005206e64f0b9cd0ab246779f377c575e0846720e8693" Nov 03 03:00:04.835977 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:04.835960 5605 generic.go:356] "Generic (PLEG): container finished" podID="1684bfe4-f5b9-4b0a-ac70-578052e13b80" containerID="bbcec6873d795a5d26bb5942374bb5e33e9a82a26aae9566f9f42c2f16862b14" exitCode=60 Nov 03 03:00:04.836110 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:04.836053 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc14f6-29894580-vhmch" event={"ID":"1684bfe4-f5b9-4b0a-ac70-578052e13b80","Type":"ContainerDied","Data":"bbcec6873d795a5d26bb5942374bb5e33e9a82a26aae9566f9f42c2f16862b14"} Nov 03 03:00:04.836253 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:04.836234 5605 scope.go:122] "RemoveContainer" containerID="bbcec6873d795a5d26bb5942374bb5e33e9a82a26aae9566f9f42c2f16862b14" Nov 03 03:00:04.836539 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:00:04.836519 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-dc14f6-29894580-vhmch_odh-ai-gateway-infra(1684bfe4-f5b9-4b0a-ac70-578052e13b80)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc14f6-29894580-vhmch" podUID="1684bfe4-f5b9-4b0a-ac70-578052e13b80" Nov 03 03:00:04.838019 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:04.837997 5605 generic.go:356] "Generic (PLEG): container finished" podID="ad92a9ca-ff2e-4e29-a984-2322317dc196" containerID="c93ee7ae44aa44e3ea9e86ccdb3d24c924f5c61b65c2855efc6ec56e8d96eb78" exitCode=60 Nov 03 03:00:04.838122 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:04.838044 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894580-fvqqd" event={"ID":"ad92a9ca-ff2e-4e29-a984-2322317dc196","Type":"ContainerDied","Data":"c93ee7ae44aa44e3ea9e86ccdb3d24c924f5c61b65c2855efc6ec56e8d96eb78"} Nov 03 03:00:04.838266 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:04.838251 5605 scope.go:122] "RemoveContainer" containerID="c93ee7ae44aa44e3ea9e86ccdb3d24c924f5c61b65c2855efc6ec56e8d96eb78" Nov 03 03:00:04.838472 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:00:04.838451 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-521f1f-29894580-fvqqd_odh-ai-gateway-infra(ad92a9ca-ff2e-4e29-a984-2322317dc196)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894580-fvqqd" podUID="ad92a9ca-ff2e-4e29-a984-2322317dc196" Nov 03 03:00:04.839817 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:04.839798 5605 generic.go:356] "Generic (PLEG): container finished" podID="ba3c3c2b-2350-4bd1-b4c4-6010e888f7e0" containerID="a4e1642058726a5e61ffc8dceb8b37561f93539487675ea741f67bd87174ba51" exitCode=60 Nov 03 03:00:04.839907 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:04.839839 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894580-9bnc2" event={"ID":"ba3c3c2b-2350-4bd1-b4c4-6010e888f7e0","Type":"ContainerDied","Data":"a4e1642058726a5e61ffc8dceb8b37561f93539487675ea741f67bd87174ba51"} Nov 03 03:00:04.840039 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:04.840027 5605 scope.go:122] "RemoveContainer" containerID="a4e1642058726a5e61ffc8dceb8b37561f93539487675ea741f67bd87174ba51" Nov 03 03:00:04.840202 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:00:04.840187 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894580-9bnc2_odh-ai-gateway-infra(ba3c3c2b-2350-4bd1-b4c4-6010e888f7e0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894580-9bnc2" podUID="ba3c3c2b-2350-4bd1-b4c4-6010e888f7e0" Nov 03 03:00:04.847215 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:04.847199 5605 scope.go:122] "RemoveContainer" containerID="b0d14227b16f90057a4dd3d29fbc666355bbd36de88ff85b7a63cf91990050ff" Nov 03 03:00:04.859345 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:04.859329 5605 scope.go:122] "RemoveContainer" containerID="8f5bbdae3435f7f822b105cd8205aa8a9c21840e78dd44bca087e6b175f15af6" Nov 03 03:00:04.871267 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:04.871251 5605 scope.go:122] "RemoveContainer" containerID="8e35b2e820264c7f0cf5a3ab9aea1a2a8798a6f26163ed8899467684b24bfec2" Nov 03 03:00:04.882898 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:04.882878 5605 scope.go:122] "RemoveContainer" containerID="92f86eb9937f9a2a127d4abb50e14ea4aeffa08edbc8e6885b241a3b41eda8be" Nov 03 03:00:04.897892 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:04.897870 5605 scope.go:122] "RemoveContainer" containerID="fa8cdaa2d50b6a7680e21228186e31fc3fff7e74305fc49d6c23f29225798703" Nov 03 03:00:04.901173 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:04.901127 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894580-5blv2" podStartSLOduration=4.901110926 podStartE2EDuration="4.901110926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:00:04.89807434 +0000 UTC m=+4082.157272504" watchObservedRunningTime="2026-11-03 03:00:04.901110926 +0000 UTC m=+4082.160309084" Nov 03 03:00:04.912298 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:04.912276 5605 scope.go:122] "RemoveContainer" containerID="7b1dd8459f2edb8a41b9d45af956f04810996300a16e8155b0afcffd52e3b6c0" Nov 03 03:00:04.924147 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:04.924132 5605 scope.go:122] "RemoveContainer" containerID="bd751fb884a71ddab40cd1ae8c1c5fce8a1556f2e3517fe9b838e334d2a91f50" Nov 03 03:00:04.936256 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:04.936240 5605 scope.go:122] "RemoveContainer" containerID="7d26c1c3de05c88886f7185305843d08dc9619e7049b57b8d2a7ecdb5c673f99" Nov 03 03:00:04.951036 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:04.951019 5605 scope.go:122] "RemoveContainer" containerID="e9948ec83a9b1ff5c8711c52e5a572dcc48089f786f6c03799ea5b5c8454f61a" Nov 03 03:00:04.952321 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:00:04.952296 5605 kuberuntime_gc.go:392] "Failed to remove container log dead symlink" err="remove /var/log/containers/maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894580-9bnc2_odh-ai-gateway-infra_cleanup-e9948ec83a9b1ff5c8711c52e5a572dcc48089f786f6c03799ea5b5c8454f61a.log: no such file or directory" path="/var/log/containers/maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894580-9bnc2_odh-ai-gateway-infra_cleanup-e9948ec83a9b1ff5c8711c52e5a572dcc48089f786f6c03799ea5b5c8454f61a.log" Nov 03 03:00:05.009941 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:05.009898 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc14f6-29894580-vhmch" podStartSLOduration=5.009886181 podStartE2EDuration="5.009886181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:00:05.007544624 +0000 UTC m=+4082.266742777" watchObservedRunningTime="2026-11-03 03:00:05.009886181 +0000 UTC m=+4082.269084333" Nov 03 03:00:05.117226 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:05.117145 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894580-5d7jp" podStartSLOduration=5.11713243 podStartE2EDuration="5.11713243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:00:05.115784233 +0000 UTC m=+4082.374982387" watchObservedRunningTime="2026-11-03 03:00:05.11713243 +0000 UTC m=+4082.376330632" Nov 03 03:00:05.849110 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:05.849079 5605 scope.go:122] "RemoveContainer" containerID="bbcec6873d795a5d26bb5942374bb5e33e9a82a26aae9566f9f42c2f16862b14" Nov 03 03:00:05.849728 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:00:05.849339 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-dc14f6-29894580-vhmch_odh-ai-gateway-infra(1684bfe4-f5b9-4b0a-ac70-578052e13b80)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc14f6-29894580-vhmch" podUID="1684bfe4-f5b9-4b0a-ac70-578052e13b80" Nov 03 03:00:05.850790 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:05.850772 5605 scope.go:122] "RemoveContainer" containerID="c93ee7ae44aa44e3ea9e86ccdb3d24c924f5c61b65c2855efc6ec56e8d96eb78" Nov 03 03:00:05.851020 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:00:05.851000 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-521f1f-29894580-fvqqd_odh-ai-gateway-infra(ad92a9ca-ff2e-4e29-a984-2322317dc196)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894580-fvqqd" podUID="ad92a9ca-ff2e-4e29-a984-2322317dc196" Nov 03 03:00:05.852427 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:05.852406 5605 scope.go:122] "RemoveContainer" containerID="a4e1642058726a5e61ffc8dceb8b37561f93539487675ea741f67bd87174ba51" Nov 03 03:00:05.852635 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:00:05.852615 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894580-9bnc2_odh-ai-gateway-infra(ba3c3c2b-2350-4bd1-b4c4-6010e888f7e0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894580-9bnc2" podUID="ba3c3c2b-2350-4bd1-b4c4-6010e888f7e0" Nov 03 03:00:05.853867 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:05.853849 5605 scope.go:122] "RemoveContainer" containerID="a1ad9cf1df6925dea495c17a645fd7ad97a55610a20b469c0efaa71fa4e19c90" Nov 03 03:00:05.854078 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:00:05.854057 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894580-5jvt2_odh-ai-gateway-infra(ce5ce822-475e-40bd-9658-cd31fc3e8eb1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894580-5jvt2" podUID="ce5ce822-475e-40bd-9658-cd31fc3e8eb1" Nov 03 03:00:05.855240 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:05.855213 5605 scope.go:122] "RemoveContainer" containerID="fac2c2344db1e30b02454fb96316a1f8ff5eb773849eb393ca008c114c1de99a" Nov 03 03:00:05.855438 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:00:05.855414 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894580-9t987_odh-ai-gateway-infra(63ff2cb3-cbff-4dad-83d6-2f63cec4d1a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894580-9t987" podUID="63ff2cb3-cbff-4dad-83d6-2f63cec4d1a3" Nov 03 03:00:05.856716 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:05.856697 5605 scope.go:122] "RemoveContainer" containerID="c73926ecd9f932709480c83569a48e051f0a8fe02047604a8bb37cab5eb1e685" Nov 03 03:00:05.856897 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:00:05.856878 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894580-z9gsn_odh-ai-gateway-infra(6f5b2e0e-94ec-44be-bde2-364e1022e4d1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894580-z9gsn" podUID="6f5b2e0e-94ec-44be-bde2-364e1022e4d1" Nov 03 03:00:05.858118 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:05.858097 5605 scope.go:122] "RemoveContainer" containerID="119f73f3b817f14e1951ff58ab43f6429735d2afdeabbe8cee9899b2c23e5990" Nov 03 03:00:05.858307 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:00:05.858286 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894580-5d7jp_odh-ai-gateway-infra(083f54a5-48ba-46e3-ab36-2bfaec4a3c9f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894580-5d7jp" podUID="083f54a5-48ba-46e3-ab36-2bfaec4a3c9f" Nov 03 03:00:05.859513 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:05.859497 5605 scope.go:122] "RemoveContainer" containerID="3532985bde0d80a87c5fbc2d184fa3c897723ef105e54a05c3153ad1d957a77e" Nov 03 03:00:05.859689 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:00:05.859671 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894580-79l9g_odh-ai-gateway-infra(75117649-d9a9-4cde-a8ff-a29010aec224)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894580-79l9g" podUID="75117649-d9a9-4cde-a8ff-a29010aec224" Nov 03 03:00:05.860985 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:05.860968 5605 scope.go:122] "RemoveContainer" containerID="c33353e6f210940cdf475f6df158e56221ad49f6fc6c00f7c6b4bdf01fa34a69" Nov 03 03:00:05.861209 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:00:05.861177 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894580-wz5ql_odh-ai-gateway-infra(441073fd-d6a9-4c6b-8cbe-f30638ca23ec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894580-wz5ql" podUID="441073fd-d6a9-4c6b-8cbe-f30638ca23ec" Nov 03 03:00:05.862425 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:05.862409 5605 scope.go:122] "RemoveContainer" containerID="c825cf2848806a2ae92dc22adaf71b37cc719ef13ed2449f995a3b2ba3dfdca3" Nov 03 03:00:05.862619 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:00:05.862601 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894580-5blv2_odh-ai-gateway-infra(b4b45e3d-f2ba-4a40-b4f6-d2fe279e35d6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894580-5blv2" podUID="b4b45e3d-f2ba-4a40-b4f6-d2fe279e35d6" Nov 03 03:00:05.863894 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:05.863877 5605 scope.go:122] "RemoveContainer" containerID="06bb57435ba4b791014b81b2eeceb1c1afa8534e47a60668500838d183e6b0a9" Nov 03 03:00:05.864053 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:00:05.864038 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894580-qx88p_odh-ai-gateway-infra(c0796b4a-9e93-4cfb-b327-14e788b9dead)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894580-qx88p" podUID="c0796b4a-9e93-4cfb-b327-14e788b9dead" Nov 03 03:00:08.494081 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:08.494049 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9a5e31-558d4bf67-2n6dc"] Nov 03 03:00:08.604203 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:08.604167 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9a5e31-5bc595cb95-5zvn4"] Nov 03 03:00:13.494575 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:13.494532 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-9a5e31-558d4bf67-2n6dc" podUID="2f68e7c9-3df8-4df4-a9d8-0544d285c291" containerName="payload-processing" containerID="cri-o://6e835840b2e6f545be2b1801e262a1589f28a41d17cec2e41baf3ce866400b90" gracePeriod=25 Nov 03 03:00:13.604560 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:13.604529 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9a5e31-5bc595cb95-5zvn4" podUID="e5fa99f5-77dc-443e-a11b-9035ca6baf5a" containerName="payload-pre-processing" containerID="cri-o://91f37a4bb6a6c4de5620732aa6b432b477689e018c08ed267aaddf6c80a0534f" gracePeriod=25 Nov 03 03:00:13.756506 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:13.756439 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9a5e31-558d4bf67-2n6dc" Nov 03 03:00:13.772912 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:13.772886 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f68e7c9-3df8-4df4-a9d8-0544d285c291-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f68e7c9-3df8-4df4-a9d8-0544d285c291-plugins-config-volume\") pod \"2f68e7c9-3df8-4df4-a9d8-0544d285c291\" (UID: \"2f68e7c9-3df8-4df4-a9d8-0544d285c291\") " Nov 03 03:00:13.773175 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:13.773137 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f68e7c9-3df8-4df4-a9d8-0544d285c291-kube-api-access-5wkqj\" (UniqueName: \"kubernetes.io/projected/2f68e7c9-3df8-4df4-a9d8-0544d285c291-kube-api-access-5wkqj\") pod \"2f68e7c9-3df8-4df4-a9d8-0544d285c291\" (UID: \"2f68e7c9-3df8-4df4-a9d8-0544d285c291\") " Nov 03 03:00:13.773379 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:13.773336 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f68e7c9-3df8-4df4-a9d8-0544d285c291-plugins-config-volume" pod "2f68e7c9-3df8-4df4-a9d8-0544d285c291" (UID: "2f68e7c9-3df8-4df4-a9d8-0544d285c291"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:00:13.773745 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:13.773723 5605 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f68e7c9-3df8-4df4-a9d8-0544d285c291-plugins-config-volume\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:13.775322 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:13.775298 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f68e7c9-3df8-4df4-a9d8-0544d285c291-kube-api-access-5wkqj" pod "2f68e7c9-3df8-4df4-a9d8-0544d285c291" (UID: "2f68e7c9-3df8-4df4-a9d8-0544d285c291"). InnerVolumeSpecName "kube-api-access-5wkqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:00:13.874894 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:13.874598 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9a5e31-5bc595cb95-5zvn4" Nov 03 03:00:13.875624 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:13.875592 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wkqj\" (UniqueName: \"kubernetes.io/projected/2f68e7c9-3df8-4df4-a9d8-0544d285c291-kube-api-access-5wkqj\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:13.915603 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:13.915579 5605 generic.go:356] "Generic (PLEG): container finished" podID="e5fa99f5-77dc-443e-a11b-9035ca6baf5a" containerID="91f37a4bb6a6c4de5620732aa6b432b477689e018c08ed267aaddf6c80a0534f" exitCode=0 Nov 03 03:00:13.915746 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:13.915629 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9a5e31-5bc595cb95-5zvn4" Nov 03 03:00:13.915746 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:13.915660 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9a5e31-5bc595cb95-5zvn4" event={"ID":"e5fa99f5-77dc-443e-a11b-9035ca6baf5a","Type":"ContainerDied","Data":"91f37a4bb6a6c4de5620732aa6b432b477689e018c08ed267aaddf6c80a0534f"} Nov 03 03:00:13.915746 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:13.915719 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9a5e31-5bc595cb95-5zvn4" event={"ID":"e5fa99f5-77dc-443e-a11b-9035ca6baf5a","Type":"ContainerDied","Data":"71b5a9a81b6cd0dc1a597157df71cb1882bbc08657b2b3bbc75cf6e64a9ac4d1"} Nov 03 03:00:13.915911 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:13.915742 5605 scope.go:122] "RemoveContainer" containerID="91f37a4bb6a6c4de5620732aa6b432b477689e018c08ed267aaddf6c80a0534f" Nov 03 03:00:13.917907 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:13.917881 5605 generic.go:356] "Generic (PLEG): container finished" podID="2f68e7c9-3df8-4df4-a9d8-0544d285c291" containerID="6e835840b2e6f545be2b1801e262a1589f28a41d17cec2e41baf3ce866400b90" exitCode=0 Nov 03 03:00:13.918027 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:13.917912 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9a5e31-558d4bf67-2n6dc" Nov 03 03:00:13.918027 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:13.917924 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9a5e31-558d4bf67-2n6dc" event={"ID":"2f68e7c9-3df8-4df4-a9d8-0544d285c291","Type":"ContainerDied","Data":"6e835840b2e6f545be2b1801e262a1589f28a41d17cec2e41baf3ce866400b90"} Nov 03 03:00:13.918027 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:13.917965 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9a5e31-558d4bf67-2n6dc" event={"ID":"2f68e7c9-3df8-4df4-a9d8-0544d285c291","Type":"ContainerDied","Data":"0ef1497beec096ef42b5ee752924ee89f90d2ec842939acd38f5a89abc0817d4"} Nov 03 03:00:13.927643 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:13.927624 5605 scope.go:122] "RemoveContainer" containerID="91f37a4bb6a6c4de5620732aa6b432b477689e018c08ed267aaddf6c80a0534f" Nov 03 03:00:13.928093 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:00:13.928050 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91f37a4bb6a6c4de5620732aa6b432b477689e018c08ed267aaddf6c80a0534f\": container with ID starting with 91f37a4bb6a6c4de5620732aa6b432b477689e018c08ed267aaddf6c80a0534f not found: ID does not exist" containerID="91f37a4bb6a6c4de5620732aa6b432b477689e018c08ed267aaddf6c80a0534f" Nov 03 03:00:13.928331 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:13.928256 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91f37a4bb6a6c4de5620732aa6b432b477689e018c08ed267aaddf6c80a0534f"} err="failed to get container status \"91f37a4bb6a6c4de5620732aa6b432b477689e018c08ed267aaddf6c80a0534f\": rpc error: code = NotFound desc = could not find container \"91f37a4bb6a6c4de5620732aa6b432b477689e018c08ed267aaddf6c80a0534f\": container with ID starting with 91f37a4bb6a6c4de5620732aa6b432b477689e018c08ed267aaddf6c80a0534f not found: ID does not exist" Nov 03 03:00:13.928331 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:13.928283 5605 scope.go:122] "RemoveContainer" containerID="6e835840b2e6f545be2b1801e262a1589f28a41d17cec2e41baf3ce866400b90" Nov 03 03:00:13.938243 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:13.938225 5605 scope.go:122] "RemoveContainer" containerID="6e835840b2e6f545be2b1801e262a1589f28a41d17cec2e41baf3ce866400b90" Nov 03 03:00:13.938601 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:00:13.938585 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e835840b2e6f545be2b1801e262a1589f28a41d17cec2e41baf3ce866400b90\": container with ID starting with 6e835840b2e6f545be2b1801e262a1589f28a41d17cec2e41baf3ce866400b90 not found: ID does not exist" containerID="6e835840b2e6f545be2b1801e262a1589f28a41d17cec2e41baf3ce866400b90" Nov 03 03:00:13.938665 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:13.938611 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e835840b2e6f545be2b1801e262a1589f28a41d17cec2e41baf3ce866400b90"} err="failed to get container status \"6e835840b2e6f545be2b1801e262a1589f28a41d17cec2e41baf3ce866400b90\": rpc error: code = NotFound desc = could not find container \"6e835840b2e6f545be2b1801e262a1589f28a41d17cec2e41baf3ce866400b90\": container with ID starting with 6e835840b2e6f545be2b1801e262a1589f28a41d17cec2e41baf3ce866400b90 not found: ID does not exist" Nov 03 03:00:13.976744 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:13.976723 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5fa99f5-77dc-443e-a11b-9035ca6baf5a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5fa99f5-77dc-443e-a11b-9035ca6baf5a-plugins-config-volume\") pod \"e5fa99f5-77dc-443e-a11b-9035ca6baf5a\" (UID: \"e5fa99f5-77dc-443e-a11b-9035ca6baf5a\") " Nov 03 03:00:13.976823 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:13.976777 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5fa99f5-77dc-443e-a11b-9035ca6baf5a-kube-api-access-r6k4c\" (UniqueName: \"kubernetes.io/projected/e5fa99f5-77dc-443e-a11b-9035ca6baf5a-kube-api-access-r6k4c\") pod \"e5fa99f5-77dc-443e-a11b-9035ca6baf5a\" (UID: \"e5fa99f5-77dc-443e-a11b-9035ca6baf5a\") " Nov 03 03:00:13.977063 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:13.977040 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5fa99f5-77dc-443e-a11b-9035ca6baf5a-plugins-config-volume" pod "e5fa99f5-77dc-443e-a11b-9035ca6baf5a" (UID: "e5fa99f5-77dc-443e-a11b-9035ca6baf5a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:00:13.977145 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:13.977133 5605 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5fa99f5-77dc-443e-a11b-9035ca6baf5a-plugins-config-volume\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:13.978830 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:13.978811 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5fa99f5-77dc-443e-a11b-9035ca6baf5a-kube-api-access-r6k4c" pod "e5fa99f5-77dc-443e-a11b-9035ca6baf5a" (UID: "e5fa99f5-77dc-443e-a11b-9035ca6baf5a"). InnerVolumeSpecName "kube-api-access-r6k4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:00:14.078048 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:14.078016 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6k4c\" (UniqueName: \"kubernetes.io/projected/e5fa99f5-77dc-443e-a11b-9035ca6baf5a-kube-api-access-r6k4c\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:14.137984 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:14.137955 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9a5e31-558d4bf67-2n6dc"] Nov 03 03:00:14.140842 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:14.140817 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9a5e31-558d4bf67-2n6dc"] Nov 03 03:00:14.333469 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:14.333399 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9a5e31-5bc595cb95-5zvn4"] Nov 03 03:00:14.337256 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:14.337232 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9a5e31-5bc595cb95-5zvn4"] Nov 03 03:00:14.682875 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:14.682813 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj"] Nov 03 03:00:14.683260 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:14.682987 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj" podUID="115ad04c-ad71-4cf1-9f39-a22ed3224fca" containerName="istio-proxy" containerID="cri-o://91d87857cee40bfa59bdb43f6ef69bb2069040d7519390ecc2610c7608772c01" gracePeriod=30 Nov 03 03:00:14.853617 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:14.853585 5605 scope.go:122] "RemoveContainer" containerID="119f73f3b817f14e1951ff58ab43f6429735d2afdeabbe8cee9899b2c23e5990" Nov 03 03:00:14.857045 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:14.857015 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f68e7c9-3df8-4df4-a9d8-0544d285c291" path="/var/lib/kubelet/pods/2f68e7c9-3df8-4df4-a9d8-0544d285c291/volumes" Nov 03 03:00:14.857433 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:14.857416 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5fa99f5-77dc-443e-a11b-9035ca6baf5a" path="/var/lib/kubelet/pods/e5fa99f5-77dc-443e-a11b-9035ca6baf5a/volumes" Nov 03 03:00:15.844028 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:15.844002 5605 scope.go:122] "RemoveContainer" containerID="06bb57435ba4b791014b81b2eeceb1c1afa8534e47a60668500838d183e6b0a9" Nov 03 03:00:15.844350 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:15.844179 5605 scope.go:122] "RemoveContainer" containerID="a4e1642058726a5e61ffc8dceb8b37561f93539487675ea741f67bd87174ba51" Nov 03 03:00:15.844350 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:15.844295 5605 scope.go:122] "RemoveContainer" containerID="c825cf2848806a2ae92dc22adaf71b37cc719ef13ed2449f995a3b2ba3dfdca3" Nov 03 03:00:15.932943 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:15.932885 5605 generic.go:356] "Generic (PLEG): container finished" podID="083f54a5-48ba-46e3-ab36-2bfaec4a3c9f" containerID="e75ce80d8373277fb82e8051c8ad0c9192da4ce49c47348756f44022f5442630" exitCode=60 Nov 03 03:00:15.932943 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:15.932932 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894580-5d7jp" event={"ID":"083f54a5-48ba-46e3-ab36-2bfaec4a3c9f","Type":"ContainerDied","Data":"e75ce80d8373277fb82e8051c8ad0c9192da4ce49c47348756f44022f5442630"} Nov 03 03:00:15.933079 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:15.932979 5605 scope.go:122] "RemoveContainer" containerID="119f73f3b817f14e1951ff58ab43f6429735d2afdeabbe8cee9899b2c23e5990" Nov 03 03:00:15.933331 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:15.933317 5605 scope.go:122] "RemoveContainer" containerID="e75ce80d8373277fb82e8051c8ad0c9192da4ce49c47348756f44022f5442630" Nov 03 03:00:15.933564 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:00:15.933541 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894580-5d7jp_odh-ai-gateway-infra(083f54a5-48ba-46e3-ab36-2bfaec4a3c9f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894580-5d7jp" podUID="083f54a5-48ba-46e3-ab36-2bfaec4a3c9f" Nov 03 03:00:16.844075 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:16.844053 5605 scope.go:122] "RemoveContainer" containerID="a1ad9cf1df6925dea495c17a645fd7ad97a55610a20b469c0efaa71fa4e19c90" Nov 03 03:00:16.844385 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:16.844169 5605 scope.go:122] "RemoveContainer" containerID="bbcec6873d795a5d26bb5942374bb5e33e9a82a26aae9566f9f42c2f16862b14" Nov 03 03:00:16.844385 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:16.844293 5605 scope.go:122] "RemoveContainer" containerID="c33353e6f210940cdf475f6df158e56221ad49f6fc6c00f7c6b4bdf01fa34a69" Nov 03 03:00:16.941767 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:16.941740 5605 generic.go:356] "Generic (PLEG): container finished" podID="c0796b4a-9e93-4cfb-b327-14e788b9dead" containerID="9089420967af7bdebe81208820cb7d35c1033057ec1f99d3f0ab1b49b74528a1" exitCode=60 Nov 03 03:00:16.941873 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:16.941808 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894580-qx88p" event={"ID":"c0796b4a-9e93-4cfb-b327-14e788b9dead","Type":"ContainerDied","Data":"9089420967af7bdebe81208820cb7d35c1033057ec1f99d3f0ab1b49b74528a1"} Nov 03 03:00:16.941946 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:16.941872 5605 scope.go:122] "RemoveContainer" containerID="06bb57435ba4b791014b81b2eeceb1c1afa8534e47a60668500838d183e6b0a9" Nov 03 03:00:16.942203 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:16.942181 5605 scope.go:122] "RemoveContainer" containerID="9089420967af7bdebe81208820cb7d35c1033057ec1f99d3f0ab1b49b74528a1" Nov 03 03:00:16.942505 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:00:16.942485 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894580-qx88p_odh-ai-gateway-infra(c0796b4a-9e93-4cfb-b327-14e788b9dead)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894580-qx88p" podUID="c0796b4a-9e93-4cfb-b327-14e788b9dead" Nov 03 03:00:16.943916 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:16.943897 5605 generic.go:356] "Generic (PLEG): container finished" podID="ba3c3c2b-2350-4bd1-b4c4-6010e888f7e0" containerID="2a34b1c695c77650573a65c44ab096017c348554fc1ccab25decf3a94d0c7d76" exitCode=60 Nov 03 03:00:16.943984 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:16.943954 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894580-9bnc2" event={"ID":"ba3c3c2b-2350-4bd1-b4c4-6010e888f7e0","Type":"ContainerDied","Data":"2a34b1c695c77650573a65c44ab096017c348554fc1ccab25decf3a94d0c7d76"} Nov 03 03:00:16.944272 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:16.944253 5605 scope.go:122] "RemoveContainer" containerID="2a34b1c695c77650573a65c44ab096017c348554fc1ccab25decf3a94d0c7d76" Nov 03 03:00:16.944549 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:00:16.944528 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894580-9bnc2_odh-ai-gateway-infra(ba3c3c2b-2350-4bd1-b4c4-6010e888f7e0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894580-9bnc2" podUID="ba3c3c2b-2350-4bd1-b4c4-6010e888f7e0" Nov 03 03:00:16.947395 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:16.947376 5605 generic.go:356] "Generic (PLEG): container finished" podID="b4b45e3d-f2ba-4a40-b4f6-d2fe279e35d6" containerID="fce4e7f8c216378bf78919c7458106b582c57bb85ecec88d86e0e02e9057b41c" exitCode=60 Nov 03 03:00:16.947490 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:16.947441 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894580-5blv2" event={"ID":"b4b45e3d-f2ba-4a40-b4f6-d2fe279e35d6","Type":"ContainerDied","Data":"fce4e7f8c216378bf78919c7458106b582c57bb85ecec88d86e0e02e9057b41c"} Nov 03 03:00:16.947803 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:16.947787 5605 scope.go:122] "RemoveContainer" containerID="fce4e7f8c216378bf78919c7458106b582c57bb85ecec88d86e0e02e9057b41c" Nov 03 03:00:16.948050 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:00:16.948030 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894580-5blv2_odh-ai-gateway-infra(b4b45e3d-f2ba-4a40-b4f6-d2fe279e35d6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894580-5blv2" podUID="b4b45e3d-f2ba-4a40-b4f6-d2fe279e35d6" Nov 03 03:00:16.955829 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:16.955807 5605 scope.go:122] "RemoveContainer" containerID="a4e1642058726a5e61ffc8dceb8b37561f93539487675ea741f67bd87174ba51" Nov 03 03:00:16.967934 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:16.967910 5605 scope.go:122] "RemoveContainer" containerID="c825cf2848806a2ae92dc22adaf71b37cc719ef13ed2449f995a3b2ba3dfdca3" Nov 03 03:00:17.049791 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:17.049765 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894580-5d7jp"] Nov 03 03:00:17.280320 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:17.280299 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894580-5d7jp" Nov 03 03:00:17.313857 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:17.313829 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/083f54a5-48ba-46e3-ab36-2bfaec4a3c9f-kube-api-access-gts6w\" (UniqueName: \"kubernetes.io/projected/083f54a5-48ba-46e3-ab36-2bfaec4a3c9f-kube-api-access-gts6w\") pod \"083f54a5-48ba-46e3-ab36-2bfaec4a3c9f\" (UID: \"083f54a5-48ba-46e3-ab36-2bfaec4a3c9f\") " Nov 03 03:00:17.313989 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:17.313939 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/083f54a5-48ba-46e3-ab36-2bfaec4a3c9f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/083f54a5-48ba-46e3-ab36-2bfaec4a3c9f-maas-api-service-ca\") pod \"083f54a5-48ba-46e3-ab36-2bfaec4a3c9f\" (UID: \"083f54a5-48ba-46e3-ab36-2bfaec4a3c9f\") " Nov 03 03:00:17.314313 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:17.314287 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/083f54a5-48ba-46e3-ab36-2bfaec4a3c9f-maas-api-service-ca" pod "083f54a5-48ba-46e3-ab36-2bfaec4a3c9f" (UID: "083f54a5-48ba-46e3-ab36-2bfaec4a3c9f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:00:17.314473 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:17.314457 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/083f54a5-48ba-46e3-ab36-2bfaec4a3c9f-maas-api-service-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:17.315912 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:17.315890 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/083f54a5-48ba-46e3-ab36-2bfaec4a3c9f-kube-api-access-gts6w" pod "083f54a5-48ba-46e3-ab36-2bfaec4a3c9f" (UID: "083f54a5-48ba-46e3-ab36-2bfaec4a3c9f"). InnerVolumeSpecName "kube-api-access-gts6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:00:17.415136 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:17.415108 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gts6w\" (UniqueName: \"kubernetes.io/projected/083f54a5-48ba-46e3-ab36-2bfaec4a3c9f-kube-api-access-gts6w\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:17.843860 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:17.843836 5605 scope.go:122] "RemoveContainer" containerID="fac2c2344db1e30b02454fb96316a1f8ff5eb773849eb393ca008c114c1de99a" Nov 03 03:00:17.843993 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:17.843924 5605 scope.go:122] "RemoveContainer" containerID="c93ee7ae44aa44e3ea9e86ccdb3d24c924f5c61b65c2855efc6ec56e8d96eb78" Nov 03 03:00:17.844059 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:17.844027 5605 scope.go:122] "RemoveContainer" containerID="c73926ecd9f932709480c83569a48e051f0a8fe02047604a8bb37cab5eb1e685" Nov 03 03:00:17.957402 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:17.957378 5605 generic.go:356] "Generic (PLEG): container finished" podID="441073fd-d6a9-4c6b-8cbe-f30638ca23ec" containerID="e7054eb342bee6d584d95b2e35f6325b2ddcf4f59b62ecf7f0bc85e85db9839f" exitCode=60 Nov 03 03:00:17.957788 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:17.957450 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894580-wz5ql" event={"ID":"441073fd-d6a9-4c6b-8cbe-f30638ca23ec","Type":"ContainerDied","Data":"e7054eb342bee6d584d95b2e35f6325b2ddcf4f59b62ecf7f0bc85e85db9839f"} Nov 03 03:00:17.957788 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:17.957502 5605 scope.go:122] "RemoveContainer" containerID="c33353e6f210940cdf475f6df158e56221ad49f6fc6c00f7c6b4bdf01fa34a69" Nov 03 03:00:17.957912 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:17.957834 5605 scope.go:122] "RemoveContainer" containerID="e7054eb342bee6d584d95b2e35f6325b2ddcf4f59b62ecf7f0bc85e85db9839f" Nov 03 03:00:17.958085 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:00:17.958058 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894580-wz5ql_odh-ai-gateway-infra(441073fd-d6a9-4c6b-8cbe-f30638ca23ec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894580-wz5ql" podUID="441073fd-d6a9-4c6b-8cbe-f30638ca23ec" Nov 03 03:00:17.962615 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:17.962596 5605 generic.go:356] "Generic (PLEG): container finished" podID="1684bfe4-f5b9-4b0a-ac70-578052e13b80" containerID="8beffc2109fcb49f171577c4151d2b97430bcab3a2afc33d4f566c946a38525b" exitCode=60 Nov 03 03:00:17.962729 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:17.962675 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc14f6-29894580-vhmch" event={"ID":"1684bfe4-f5b9-4b0a-ac70-578052e13b80","Type":"ContainerDied","Data":"8beffc2109fcb49f171577c4151d2b97430bcab3a2afc33d4f566c946a38525b"} Nov 03 03:00:17.962997 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:17.962981 5605 scope.go:122] "RemoveContainer" containerID="8beffc2109fcb49f171577c4151d2b97430bcab3a2afc33d4f566c946a38525b" Nov 03 03:00:17.963224 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:00:17.963201 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-dc14f6-29894580-vhmch_odh-ai-gateway-infra(1684bfe4-f5b9-4b0a-ac70-578052e13b80)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc14f6-29894580-vhmch" podUID="1684bfe4-f5b9-4b0a-ac70-578052e13b80" Nov 03 03:00:17.975430 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:17.975407 5605 generic.go:356] "Generic (PLEG): container finished" podID="ce5ce822-475e-40bd-9658-cd31fc3e8eb1" containerID="a6c0b3423a011d445d328551ccadefd0909e03517b3fed3cb52799466882e0ed" exitCode=60 Nov 03 03:00:17.975522 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:17.975445 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894580-5jvt2" event={"ID":"ce5ce822-475e-40bd-9658-cd31fc3e8eb1","Type":"ContainerDied","Data":"a6c0b3423a011d445d328551ccadefd0909e03517b3fed3cb52799466882e0ed"} Nov 03 03:00:17.975806 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:17.975777 5605 scope.go:122] "RemoveContainer" containerID="a6c0b3423a011d445d328551ccadefd0909e03517b3fed3cb52799466882e0ed" Nov 03 03:00:17.976046 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:00:17.976015 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894580-5jvt2_odh-ai-gateway-infra(ce5ce822-475e-40bd-9658-cd31fc3e8eb1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894580-5jvt2" podUID="ce5ce822-475e-40bd-9658-cd31fc3e8eb1" Nov 03 03:00:17.976898 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:17.976877 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894580-5d7jp" event={"ID":"083f54a5-48ba-46e3-ab36-2bfaec4a3c9f","Type":"ContainerDied","Data":"af807b2bcaf9a44dc5ecafa1f4872a3dcb99679c75507793758da28b8dc874e9"} Nov 03 03:00:17.977004 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:17.976901 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894580-5d7jp" Nov 03 03:00:17.978737 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:17.978720 5605 scope.go:122] "RemoveContainer" containerID="bbcec6873d795a5d26bb5942374bb5e33e9a82a26aae9566f9f42c2f16862b14" Nov 03 03:00:17.990989 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:17.990968 5605 scope.go:122] "RemoveContainer" containerID="a1ad9cf1df6925dea495c17a645fd7ad97a55610a20b469c0efaa71fa4e19c90" Nov 03 03:00:18.003307 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:18.003291 5605 scope.go:122] "RemoveContainer" containerID="e75ce80d8373277fb82e8051c8ad0c9192da4ce49c47348756f44022f5442630" Nov 03 03:00:18.061789 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:18.061760 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894580-qx88p"] Nov 03 03:00:18.174670 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:18.174550 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894580-9bnc2"] Nov 03 03:00:18.196886 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:18.196863 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894580-qx88p" Nov 03 03:00:18.224768 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:18.224703 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0796b4a-9e93-4cfb-b327-14e788b9dead-kube-api-access-858hr\" (UniqueName: \"kubernetes.io/projected/c0796b4a-9e93-4cfb-b327-14e788b9dead-kube-api-access-858hr\") pod \"c0796b4a-9e93-4cfb-b327-14e788b9dead\" (UID: \"c0796b4a-9e93-4cfb-b327-14e788b9dead\") " Nov 03 03:00:18.224768 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:18.224750 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0796b4a-9e93-4cfb-b327-14e788b9dead-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0796b4a-9e93-4cfb-b327-14e788b9dead-maas-api-service-ca\") pod \"c0796b4a-9e93-4cfb-b327-14e788b9dead\" (UID: \"c0796b4a-9e93-4cfb-b327-14e788b9dead\") " Nov 03 03:00:18.225126 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:18.225106 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0796b4a-9e93-4cfb-b327-14e788b9dead-maas-api-service-ca" pod "c0796b4a-9e93-4cfb-b327-14e788b9dead" (UID: "c0796b4a-9e93-4cfb-b327-14e788b9dead"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:00:18.225213 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:18.225193 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0796b4a-9e93-4cfb-b327-14e788b9dead-maas-api-service-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:18.255294 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:18.255262 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0796b4a-9e93-4cfb-b327-14e788b9dead-kube-api-access-858hr" pod "c0796b4a-9e93-4cfb-b327-14e788b9dead" (UID: "c0796b4a-9e93-4cfb-b327-14e788b9dead"). InnerVolumeSpecName "kube-api-access-858hr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:00:18.277399 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:18.277299 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894580-5blv2"] Nov 03 03:00:18.326312 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:18.326276 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-858hr\" (UniqueName: \"kubernetes.io/projected/c0796b4a-9e93-4cfb-b327-14e788b9dead-kube-api-access-858hr\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:18.383287 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:18.383261 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894580-5d7jp"] Nov 03 03:00:18.386120 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:18.386100 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894580-9bnc2" Nov 03 03:00:18.390009 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:18.389983 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894580-5d7jp"] Nov 03 03:00:18.427336 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:18.427312 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba3c3c2b-2350-4bd1-b4c4-6010e888f7e0-kube-api-access-lf66k\" (UniqueName: \"kubernetes.io/projected/ba3c3c2b-2350-4bd1-b4c4-6010e888f7e0-kube-api-access-lf66k\") pod \"ba3c3c2b-2350-4bd1-b4c4-6010e888f7e0\" (UID: \"ba3c3c2b-2350-4bd1-b4c4-6010e888f7e0\") " Nov 03 03:00:18.427474 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:18.427461 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba3c3c2b-2350-4bd1-b4c4-6010e888f7e0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba3c3c2b-2350-4bd1-b4c4-6010e888f7e0-maas-api-service-ca\") pod \"ba3c3c2b-2350-4bd1-b4c4-6010e888f7e0\" (UID: \"ba3c3c2b-2350-4bd1-b4c4-6010e888f7e0\") " Nov 03 03:00:18.427993 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:18.427890 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba3c3c2b-2350-4bd1-b4c4-6010e888f7e0-maas-api-service-ca" pod "ba3c3c2b-2350-4bd1-b4c4-6010e888f7e0" (UID: "ba3c3c2b-2350-4bd1-b4c4-6010e888f7e0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:00:18.429427 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:18.429401 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba3c3c2b-2350-4bd1-b4c4-6010e888f7e0-kube-api-access-lf66k" pod "ba3c3c2b-2350-4bd1-b4c4-6010e888f7e0" (UID: "ba3c3c2b-2350-4bd1-b4c4-6010e888f7e0"). InnerVolumeSpecName "kube-api-access-lf66k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:00:18.435774 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:18.435756 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894580-5blv2" Nov 03 03:00:18.528435 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:18.528335 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4b45e3d-f2ba-4a40-b4f6-d2fe279e35d6-kube-api-access-5lhg5\" (UniqueName: \"kubernetes.io/projected/b4b45e3d-f2ba-4a40-b4f6-d2fe279e35d6-kube-api-access-5lhg5\") pod \"b4b45e3d-f2ba-4a40-b4f6-d2fe279e35d6\" (UID: \"b4b45e3d-f2ba-4a40-b4f6-d2fe279e35d6\") " Nov 03 03:00:18.528541 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:18.528514 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4b45e3d-f2ba-4a40-b4f6-d2fe279e35d6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4b45e3d-f2ba-4a40-b4f6-d2fe279e35d6-maas-api-service-ca\") pod \"b4b45e3d-f2ba-4a40-b4f6-d2fe279e35d6\" (UID: \"b4b45e3d-f2ba-4a40-b4f6-d2fe279e35d6\") " Nov 03 03:00:18.528836 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:18.528820 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lf66k\" (UniqueName: \"kubernetes.io/projected/ba3c3c2b-2350-4bd1-b4c4-6010e888f7e0-kube-api-access-lf66k\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:18.528836 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:18.528835 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba3c3c2b-2350-4bd1-b4c4-6010e888f7e0-maas-api-service-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:18.528986 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:18.528853 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4b45e3d-f2ba-4a40-b4f6-d2fe279e35d6-maas-api-service-ca" pod "b4b45e3d-f2ba-4a40-b4f6-d2fe279e35d6" (UID: "b4b45e3d-f2ba-4a40-b4f6-d2fe279e35d6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:00:18.530319 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:18.530300 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4b45e3d-f2ba-4a40-b4f6-d2fe279e35d6-kube-api-access-5lhg5" pod "b4b45e3d-f2ba-4a40-b4f6-d2fe279e35d6" (UID: "b4b45e3d-f2ba-4a40-b4f6-d2fe279e35d6"). InnerVolumeSpecName "kube-api-access-5lhg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:00:18.629736 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:18.629708 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lhg5\" (UniqueName: \"kubernetes.io/projected/b4b45e3d-f2ba-4a40-b4f6-d2fe279e35d6-kube-api-access-5lhg5\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:18.629736 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:18.629732 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4b45e3d-f2ba-4a40-b4f6-d2fe279e35d6-maas-api-service-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:18.851315 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:18.851281 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="083f54a5-48ba-46e3-ab36-2bfaec4a3c9f" path="/var/lib/kubelet/pods/083f54a5-48ba-46e3-ab36-2bfaec4a3c9f/volumes" Nov 03 03:00:18.852389 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:18.852339 5605 scope.go:122] "RemoveContainer" containerID="3532985bde0d80a87c5fbc2d184fa3c897723ef105e54a05c3153ad1d957a77e" Nov 03 03:00:18.985966 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:18.985941 5605 generic.go:356] "Generic (PLEG): container finished" podID="63ff2cb3-cbff-4dad-83d6-2f63cec4d1a3" containerID="f8e506efdaa610c629f38cbabeb34ba0f0b2dded0e3fbe78d93d8b0e0045019a" exitCode=60 Nov 03 03:00:18.986333 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:18.986010 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894580-9t987" event={"ID":"63ff2cb3-cbff-4dad-83d6-2f63cec4d1a3","Type":"ContainerDied","Data":"f8e506efdaa610c629f38cbabeb34ba0f0b2dded0e3fbe78d93d8b0e0045019a"} Nov 03 03:00:18.986333 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:18.986073 5605 scope.go:122] "RemoveContainer" containerID="fac2c2344db1e30b02454fb96316a1f8ff5eb773849eb393ca008c114c1de99a" Nov 03 03:00:18.986490 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:18.986470 5605 scope.go:122] "RemoveContainer" containerID="f8e506efdaa610c629f38cbabeb34ba0f0b2dded0e3fbe78d93d8b0e0045019a" Nov 03 03:00:18.986791 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:00:18.986766 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894580-9t987_odh-ai-gateway-infra(63ff2cb3-cbff-4dad-83d6-2f63cec4d1a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894580-9t987" podUID="63ff2cb3-cbff-4dad-83d6-2f63cec4d1a3" Nov 03 03:00:18.988084 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:18.988066 5605 generic.go:356] "Generic (PLEG): container finished" podID="6f5b2e0e-94ec-44be-bde2-364e1022e4d1" containerID="6b8400c9f2e2038245db62b0934bdb935035237556398b1407b156e77515eb8c" exitCode=60 Nov 03 03:00:18.988159 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:18.988140 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894580-z9gsn" event={"ID":"6f5b2e0e-94ec-44be-bde2-364e1022e4d1","Type":"ContainerDied","Data":"6b8400c9f2e2038245db62b0934bdb935035237556398b1407b156e77515eb8c"} Nov 03 03:00:18.988540 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:18.988521 5605 scope.go:122] "RemoveContainer" containerID="6b8400c9f2e2038245db62b0934bdb935035237556398b1407b156e77515eb8c" Nov 03 03:00:18.988817 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:00:18.988794 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894580-z9gsn_odh-ai-gateway-infra(6f5b2e0e-94ec-44be-bde2-364e1022e4d1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894580-z9gsn" podUID="6f5b2e0e-94ec-44be-bde2-364e1022e4d1" Nov 03 03:00:18.991647 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:18.991632 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894580-5blv2" Nov 03 03:00:18.991741 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:18.991665 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894580-5blv2" event={"ID":"b4b45e3d-f2ba-4a40-b4f6-d2fe279e35d6","Type":"ContainerDied","Data":"fbc9b4a93006c64e4634996525f3cac1474837d884578bfb1e4ec2c7dd90ae7d"} Nov 03 03:00:18.993798 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:18.993782 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894580-qx88p" Nov 03 03:00:18.993893 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:18.993800 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894580-qx88p" event={"ID":"c0796b4a-9e93-4cfb-b327-14e788b9dead","Type":"ContainerDied","Data":"8ed9f73f3355433ab10d654d895351b211e18d30d54b37203ed01e065318f89e"} Nov 03 03:00:18.996735 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:18.996717 5605 generic.go:356] "Generic (PLEG): container finished" podID="ad92a9ca-ff2e-4e29-a984-2322317dc196" containerID="ba45b80f2a9f85ac65c61fd44221634bb6fa955f3a5ff1a30695cadfeb43dafd" exitCode=60 Nov 03 03:00:18.996820 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:18.996789 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894580-fvqqd" event={"ID":"ad92a9ca-ff2e-4e29-a984-2322317dc196","Type":"ContainerDied","Data":"ba45b80f2a9f85ac65c61fd44221634bb6fa955f3a5ff1a30695cadfeb43dafd"} Nov 03 03:00:18.997094 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:18.997078 5605 scope.go:122] "RemoveContainer" containerID="ba45b80f2a9f85ac65c61fd44221634bb6fa955f3a5ff1a30695cadfeb43dafd" Nov 03 03:00:18.997295 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:00:18.997275 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-521f1f-29894580-fvqqd_odh-ai-gateway-infra(ad92a9ca-ff2e-4e29-a984-2322317dc196)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894580-fvqqd" podUID="ad92a9ca-ff2e-4e29-a984-2322317dc196" Nov 03 03:00:18.997859 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:18.997845 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894580-9bnc2" Nov 03 03:00:18.997915 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:18.997861 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894580-9bnc2" event={"ID":"ba3c3c2b-2350-4bd1-b4c4-6010e888f7e0","Type":"ContainerDied","Data":"b619500480c5d7b5dba218ca860bf558ba0c91a885ffd2f3e86118c2edf69683"} Nov 03 03:00:19.074040 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:19.074008 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894580-wz5ql"] Nov 03 03:00:19.102256 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:19.102236 5605 scope.go:122] "RemoveContainer" containerID="c73926ecd9f932709480c83569a48e051f0a8fe02047604a8bb37cab5eb1e685" Nov 03 03:00:19.116949 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:19.116930 5605 scope.go:122] "RemoveContainer" containerID="fce4e7f8c216378bf78919c7458106b582c57bb85ecec88d86e0e02e9057b41c" Nov 03 03:00:19.125542 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:19.125513 5605 scope.go:122] "RemoveContainer" containerID="9089420967af7bdebe81208820cb7d35c1033057ec1f99d3f0ab1b49b74528a1" Nov 03 03:00:19.134241 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:19.134125 5605 scope.go:122] "RemoveContainer" containerID="c93ee7ae44aa44e3ea9e86ccdb3d24c924f5c61b65c2855efc6ec56e8d96eb78" Nov 03 03:00:19.149711 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:19.149691 5605 scope.go:122] "RemoveContainer" containerID="2a34b1c695c77650573a65c44ab096017c348554fc1ccab25decf3a94d0c7d76" Nov 03 03:00:19.224454 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:19.224431 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894580-wz5ql" Nov 03 03:00:19.288209 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:19.288179 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc14f6-29894580-vhmch"] Nov 03 03:00:19.338107 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:19.338069 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/441073fd-d6a9-4c6b-8cbe-f30638ca23ec-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/441073fd-d6a9-4c6b-8cbe-f30638ca23ec-maas-api-service-ca\") pod \"441073fd-d6a9-4c6b-8cbe-f30638ca23ec\" (UID: \"441073fd-d6a9-4c6b-8cbe-f30638ca23ec\") " Nov 03 03:00:19.338274 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:19.338136 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/441073fd-d6a9-4c6b-8cbe-f30638ca23ec-kube-api-access-jvslz\" (UniqueName: \"kubernetes.io/projected/441073fd-d6a9-4c6b-8cbe-f30638ca23ec-kube-api-access-jvslz\") pod \"441073fd-d6a9-4c6b-8cbe-f30638ca23ec\" (UID: \"441073fd-d6a9-4c6b-8cbe-f30638ca23ec\") " Nov 03 03:00:19.338508 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:19.338408 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/441073fd-d6a9-4c6b-8cbe-f30638ca23ec-maas-api-service-ca" pod "441073fd-d6a9-4c6b-8cbe-f30638ca23ec" (UID: "441073fd-d6a9-4c6b-8cbe-f30638ca23ec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:00:19.338627 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:19.338544 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/441073fd-d6a9-4c6b-8cbe-f30638ca23ec-maas-api-service-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:19.340103 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:19.340079 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/441073fd-d6a9-4c6b-8cbe-f30638ca23ec-kube-api-access-jvslz" pod "441073fd-d6a9-4c6b-8cbe-f30638ca23ec" (UID: "441073fd-d6a9-4c6b-8cbe-f30638ca23ec"). InnerVolumeSpecName "kube-api-access-jvslz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:00:19.415083 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:19.415063 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc14f6-29894580-vhmch" Nov 03 03:00:19.439742 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:19.439710 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvslz\" (UniqueName: \"kubernetes.io/projected/441073fd-d6a9-4c6b-8cbe-f30638ca23ec-kube-api-access-jvslz\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:19.500891 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:19.500861 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894580-5jvt2"] Nov 03 03:00:19.541010 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:19.540978 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1684bfe4-f5b9-4b0a-ac70-578052e13b80-kube-api-access-scpbp\" (UniqueName: \"kubernetes.io/projected/1684bfe4-f5b9-4b0a-ac70-578052e13b80-kube-api-access-scpbp\") pod \"1684bfe4-f5b9-4b0a-ac70-578052e13b80\" (UID: \"1684bfe4-f5b9-4b0a-ac70-578052e13b80\") " Nov 03 03:00:19.541174 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:19.541025 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1684bfe4-f5b9-4b0a-ac70-578052e13b80-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1684bfe4-f5b9-4b0a-ac70-578052e13b80-maas-api-service-ca\") pod \"1684bfe4-f5b9-4b0a-ac70-578052e13b80\" (UID: \"1684bfe4-f5b9-4b0a-ac70-578052e13b80\") " Nov 03 03:00:19.541499 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:19.541460 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1684bfe4-f5b9-4b0a-ac70-578052e13b80-maas-api-service-ca" pod "1684bfe4-f5b9-4b0a-ac70-578052e13b80" (UID: "1684bfe4-f5b9-4b0a-ac70-578052e13b80"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:00:19.542949 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:19.542927 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1684bfe4-f5b9-4b0a-ac70-578052e13b80-kube-api-access-scpbp" pod "1684bfe4-f5b9-4b0a-ac70-578052e13b80" (UID: "1684bfe4-f5b9-4b0a-ac70-578052e13b80"). InnerVolumeSpecName "kube-api-access-scpbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:00:19.642649 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:19.642622 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scpbp\" (UniqueName: \"kubernetes.io/projected/1684bfe4-f5b9-4b0a-ac70-578052e13b80-kube-api-access-scpbp\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:19.642649 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:19.642648 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1684bfe4-f5b9-4b0a-ac70-578052e13b80-maas-api-service-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:19.647821 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:19.647799 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894580-5jvt2" Nov 03 03:00:19.743269 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:19.743245 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce5ce822-475e-40bd-9658-cd31fc3e8eb1-kube-api-access-cmtjv\" (UniqueName: \"kubernetes.io/projected/ce5ce822-475e-40bd-9658-cd31fc3e8eb1-kube-api-access-cmtjv\") pod \"ce5ce822-475e-40bd-9658-cd31fc3e8eb1\" (UID: \"ce5ce822-475e-40bd-9658-cd31fc3e8eb1\") " Nov 03 03:00:19.743483 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:19.743469 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce5ce822-475e-40bd-9658-cd31fc3e8eb1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce5ce822-475e-40bd-9658-cd31fc3e8eb1-maas-api-service-ca\") pod \"ce5ce822-475e-40bd-9658-cd31fc3e8eb1\" (UID: \"ce5ce822-475e-40bd-9658-cd31fc3e8eb1\") " Nov 03 03:00:19.743800 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:19.743774 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce5ce822-475e-40bd-9658-cd31fc3e8eb1-maas-api-service-ca" pod "ce5ce822-475e-40bd-9658-cd31fc3e8eb1" (UID: "ce5ce822-475e-40bd-9658-cd31fc3e8eb1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:00:19.745160 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:19.745137 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce5ce822-475e-40bd-9658-cd31fc3e8eb1-kube-api-access-cmtjv" pod "ce5ce822-475e-40bd-9658-cd31fc3e8eb1" (UID: "ce5ce822-475e-40bd-9658-cd31fc3e8eb1"). InnerVolumeSpecName "kube-api-access-cmtjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:00:19.767114 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:19.767094 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894580-qx88p"] Nov 03 03:00:19.771212 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:19.771191 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894580-qx88p"] Nov 03 03:00:19.844585 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:19.844559 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce5ce822-475e-40bd-9658-cd31fc3e8eb1-maas-api-service-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:19.844585 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:19.844585 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmtjv\" (UniqueName: \"kubernetes.io/projected/ce5ce822-475e-40bd-9658-cd31fc3e8eb1-kube-api-access-cmtjv\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:19.877985 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:19.876200 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894580-9bnc2"] Nov 03 03:00:19.889467 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:19.889444 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894580-9bnc2"] Nov 03 03:00:19.952285 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:19.952261 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj" Nov 03 03:00:19.992297 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:19.992272 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894580-5blv2"] Nov 03 03:00:19.996762 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:19.996738 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894580-5blv2"] Nov 03 03:00:20.010408 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.010383 5605 generic.go:356] "Generic (PLEG): container finished" podID="115ad04c-ad71-4cf1-9f39-a22ed3224fca" containerID="91d87857cee40bfa59bdb43f6ef69bb2069040d7519390ecc2610c7608772c01" exitCode=0 Nov 03 03:00:20.010501 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.010434 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj" event={"ID":"115ad04c-ad71-4cf1-9f39-a22ed3224fca","Type":"ContainerDied","Data":"91d87857cee40bfa59bdb43f6ef69bb2069040d7519390ecc2610c7608772c01"} Nov 03 03:00:20.010562 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.010441 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj" Nov 03 03:00:20.010562 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.010517 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj" event={"ID":"115ad04c-ad71-4cf1-9f39-a22ed3224fca","Type":"ContainerDied","Data":"7e2269fb3643630ddab686abb963a2c985894fdd730acf80b8caad2c93b1280f"} Nov 03 03:00:20.010562 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.010547 5605 scope.go:122] "RemoveContainer" containerID="91d87857cee40bfa59bdb43f6ef69bb2069040d7519390ecc2610c7608772c01" Nov 03 03:00:20.011711 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.011687 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894580-5jvt2" event={"ID":"ce5ce822-475e-40bd-9658-cd31fc3e8eb1","Type":"ContainerDied","Data":"41f1029fb30e8457a2d3b661df8fe3cb158c7c7cd876fd43882efe7c925c6531"} Nov 03 03:00:20.011816 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.011701 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894580-5jvt2" Nov 03 03:00:20.017250 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.017229 5605 generic.go:356] "Generic (PLEG): container finished" podID="75117649-d9a9-4cde-a8ff-a29010aec224" containerID="c345cbef33ff444f2eac759f6ab05fc5f2d919332dc80992884c414d10394874" exitCode=60 Nov 03 03:00:20.017338 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.017301 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894580-79l9g" event={"ID":"75117649-d9a9-4cde-a8ff-a29010aec224","Type":"ContainerDied","Data":"c345cbef33ff444f2eac759f6ab05fc5f2d919332dc80992884c414d10394874"} Nov 03 03:00:20.017754 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.017732 5605 scope.go:122] "RemoveContainer" containerID="c345cbef33ff444f2eac759f6ab05fc5f2d919332dc80992884c414d10394874" Nov 03 03:00:20.018051 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:00:20.018028 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894580-79l9g_odh-ai-gateway-infra(75117649-d9a9-4cde-a8ff-a29010aec224)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894580-79l9g" podUID="75117649-d9a9-4cde-a8ff-a29010aec224" Nov 03 03:00:20.018797 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.018774 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894580-wz5ql" event={"ID":"441073fd-d6a9-4c6b-8cbe-f30638ca23ec","Type":"ContainerDied","Data":"745a92d3187878cb4b3ff88d57f62b1253f4f426bdb3a55f888abcbef8820adf"} Nov 03 03:00:20.018901 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.018776 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894580-wz5ql" Nov 03 03:00:20.022233 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.022187 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc14f6-29894580-vhmch" event={"ID":"1684bfe4-f5b9-4b0a-ac70-578052e13b80","Type":"ContainerDied","Data":"0a3b95d975852a5059ca4be5102547d867ba70317c0962042759b985af17ea4c"} Nov 03 03:00:20.022299 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.022231 5605 scope.go:122] "RemoveContainer" containerID="91d87857cee40bfa59bdb43f6ef69bb2069040d7519390ecc2610c7608772c01" Nov 03 03:00:20.022299 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.022253 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc14f6-29894580-vhmch" Nov 03 03:00:20.022597 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:00:20.022574 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91d87857cee40bfa59bdb43f6ef69bb2069040d7519390ecc2610c7608772c01\": container with ID starting with 91d87857cee40bfa59bdb43f6ef69bb2069040d7519390ecc2610c7608772c01 not found: ID does not exist" containerID="91d87857cee40bfa59bdb43f6ef69bb2069040d7519390ecc2610c7608772c01" Nov 03 03:00:20.022672 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.022602 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91d87857cee40bfa59bdb43f6ef69bb2069040d7519390ecc2610c7608772c01"} err="failed to get container status \"91d87857cee40bfa59bdb43f6ef69bb2069040d7519390ecc2610c7608772c01\": rpc error: code = NotFound desc = could not find container \"91d87857cee40bfa59bdb43f6ef69bb2069040d7519390ecc2610c7608772c01\": container with ID starting with 91d87857cee40bfa59bdb43f6ef69bb2069040d7519390ecc2610c7608772c01 not found: ID does not exist" Nov 03 03:00:20.022672 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.022615 5605 scope.go:122] "RemoveContainer" containerID="a6c0b3423a011d445d328551ccadefd0909e03517b3fed3cb52799466882e0ed" Nov 03 03:00:20.031665 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.031649 5605 scope.go:122] "RemoveContainer" containerID="3532985bde0d80a87c5fbc2d184fa3c897723ef105e54a05c3153ad1d957a77e" Nov 03 03:00:20.043930 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.043911 5605 scope.go:122] "RemoveContainer" containerID="e7054eb342bee6d584d95b2e35f6325b2ddcf4f59b62ecf7f0bc85e85db9839f" Nov 03 03:00:20.047064 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.047044 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/115ad04c-ad71-4cf1-9f39-a22ed3224fca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/115ad04c-ad71-4cf1-9f39-a22ed3224fca-workload-certs\") pod \"115ad04c-ad71-4cf1-9f39-a22ed3224fca\" (UID: \"115ad04c-ad71-4cf1-9f39-a22ed3224fca\") " Nov 03 03:00:20.047147 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.047092 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/115ad04c-ad71-4cf1-9f39-a22ed3224fca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/115ad04c-ad71-4cf1-9f39-a22ed3224fca-workload-socket\") pod \"115ad04c-ad71-4cf1-9f39-a22ed3224fca\" (UID: \"115ad04c-ad71-4cf1-9f39-a22ed3224fca\") " Nov 03 03:00:20.047147 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.047135 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/115ad04c-ad71-4cf1-9f39-a22ed3224fca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/115ad04c-ad71-4cf1-9f39-a22ed3224fca-istio-podinfo\") pod \"115ad04c-ad71-4cf1-9f39-a22ed3224fca\" (UID: \"115ad04c-ad71-4cf1-9f39-a22ed3224fca\") " Nov 03 03:00:20.047248 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.047173 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/115ad04c-ad71-4cf1-9f39-a22ed3224fca-istio-token\" (UniqueName: \"kubernetes.io/projected/115ad04c-ad71-4cf1-9f39-a22ed3224fca-istio-token\") pod \"115ad04c-ad71-4cf1-9f39-a22ed3224fca\" (UID: \"115ad04c-ad71-4cf1-9f39-a22ed3224fca\") " Nov 03 03:00:20.047248 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.047213 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/115ad04c-ad71-4cf1-9f39-a22ed3224fca-kube-api-access-7n6sz\" (UniqueName: \"kubernetes.io/projected/115ad04c-ad71-4cf1-9f39-a22ed3224fca-kube-api-access-7n6sz\") pod \"115ad04c-ad71-4cf1-9f39-a22ed3224fca\" (UID: \"115ad04c-ad71-4cf1-9f39-a22ed3224fca\") " Nov 03 03:00:20.047324 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.047290 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/115ad04c-ad71-4cf1-9f39-a22ed3224fca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/115ad04c-ad71-4cf1-9f39-a22ed3224fca-istio-envoy\") pod \"115ad04c-ad71-4cf1-9f39-a22ed3224fca\" (UID: \"115ad04c-ad71-4cf1-9f39-a22ed3224fca\") " Nov 03 03:00:20.047416 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.047327 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/115ad04c-ad71-4cf1-9f39-a22ed3224fca-workload-certs" pod "115ad04c-ad71-4cf1-9f39-a22ed3224fca" (UID: "115ad04c-ad71-4cf1-9f39-a22ed3224fca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:00:20.047416 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.047385 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/115ad04c-ad71-4cf1-9f39-a22ed3224fca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/115ad04c-ad71-4cf1-9f39-a22ed3224fca-istio-data\") pod \"115ad04c-ad71-4cf1-9f39-a22ed3224fca\" (UID: \"115ad04c-ad71-4cf1-9f39-a22ed3224fca\") " Nov 03 03:00:20.047521 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.047500 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/115ad04c-ad71-4cf1-9f39-a22ed3224fca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/115ad04c-ad71-4cf1-9f39-a22ed3224fca-credential-socket\") pod \"115ad04c-ad71-4cf1-9f39-a22ed3224fca\" (UID: \"115ad04c-ad71-4cf1-9f39-a22ed3224fca\") " Nov 03 03:00:20.047601 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.047536 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/115ad04c-ad71-4cf1-9f39-a22ed3224fca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/115ad04c-ad71-4cf1-9f39-a22ed3224fca-istiod-ca-cert\") pod \"115ad04c-ad71-4cf1-9f39-a22ed3224fca\" (UID: \"115ad04c-ad71-4cf1-9f39-a22ed3224fca\") " Nov 03 03:00:20.048351 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.047955 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/115ad04c-ad71-4cf1-9f39-a22ed3224fca-workload-socket" pod "115ad04c-ad71-4cf1-9f39-a22ed3224fca" (UID: "115ad04c-ad71-4cf1-9f39-a22ed3224fca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:00:20.048351 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.047884 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/115ad04c-ad71-4cf1-9f39-a22ed3224fca-istio-data" pod "115ad04c-ad71-4cf1-9f39-a22ed3224fca" (UID: "115ad04c-ad71-4cf1-9f39-a22ed3224fca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:00:20.048351 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.048147 5605 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/115ad04c-ad71-4cf1-9f39-a22ed3224fca-workload-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:20.048351 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.048171 5605 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/115ad04c-ad71-4cf1-9f39-a22ed3224fca-workload-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:20.048351 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.048186 5605 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/115ad04c-ad71-4cf1-9f39-a22ed3224fca-istio-data\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:20.048700 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.048437 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/115ad04c-ad71-4cf1-9f39-a22ed3224fca-credential-socket" pod "115ad04c-ad71-4cf1-9f39-a22ed3224fca" (UID: "115ad04c-ad71-4cf1-9f39-a22ed3224fca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:00:20.048700 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.048583 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/115ad04c-ad71-4cf1-9f39-a22ed3224fca-istiod-ca-cert" pod "115ad04c-ad71-4cf1-9f39-a22ed3224fca" (UID: "115ad04c-ad71-4cf1-9f39-a22ed3224fca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:00:20.049511 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.049478 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/115ad04c-ad71-4cf1-9f39-a22ed3224fca-istio-podinfo" pod "115ad04c-ad71-4cf1-9f39-a22ed3224fca" (UID: "115ad04c-ad71-4cf1-9f39-a22ed3224fca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:00:20.049827 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.049802 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/115ad04c-ad71-4cf1-9f39-a22ed3224fca-kube-api-access-7n6sz" pod "115ad04c-ad71-4cf1-9f39-a22ed3224fca" (UID: "115ad04c-ad71-4cf1-9f39-a22ed3224fca"). InnerVolumeSpecName "kube-api-access-7n6sz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:00:20.050220 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.050199 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/115ad04c-ad71-4cf1-9f39-a22ed3224fca-istio-token" pod "115ad04c-ad71-4cf1-9f39-a22ed3224fca" (UID: "115ad04c-ad71-4cf1-9f39-a22ed3224fca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:00:20.050832 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.050816 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/115ad04c-ad71-4cf1-9f39-a22ed3224fca-istio-envoy" pod "115ad04c-ad71-4cf1-9f39-a22ed3224fca" (UID: "115ad04c-ad71-4cf1-9f39-a22ed3224fca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:00:20.054103 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.054080 5605 scope.go:122] "RemoveContainer" containerID="8beffc2109fcb49f171577c4151d2b97430bcab3a2afc33d4f566c946a38525b" Nov 03 03:00:20.109395 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.107602 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894580-9t987"] Nov 03 03:00:20.149038 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.148999 5605 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/115ad04c-ad71-4cf1-9f39-a22ed3224fca-credential-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:20.149038 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.149027 5605 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/115ad04c-ad71-4cf1-9f39-a22ed3224fca-istiod-ca-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:20.149038 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.149038 5605 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/115ad04c-ad71-4cf1-9f39-a22ed3224fca-istio-podinfo\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:20.149038 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.149047 5605 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/115ad04c-ad71-4cf1-9f39-a22ed3224fca-istio-token\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:20.149430 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.149057 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n6sz\" (UniqueName: \"kubernetes.io/projected/115ad04c-ad71-4cf1-9f39-a22ed3224fca-kube-api-access-7n6sz\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:20.149430 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.149066 5605 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/115ad04c-ad71-4cf1-9f39-a22ed3224fca-istio-envoy\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:20.212598 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.212532 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894580-fvqqd"] Nov 03 03:00:20.243297 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.243268 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894580-9t987" Nov 03 03:00:20.312603 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.312569 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894580-wz5ql"] Nov 03 03:00:20.317752 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.317725 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-957b33-29894580-wz5ql"] Nov 03 03:00:20.351526 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.351501 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63ff2cb3-cbff-4dad-83d6-2f63cec4d1a3-kube-api-access-xkxzw\" (UniqueName: \"kubernetes.io/projected/63ff2cb3-cbff-4dad-83d6-2f63cec4d1a3-kube-api-access-xkxzw\") pod \"63ff2cb3-cbff-4dad-83d6-2f63cec4d1a3\" (UID: \"63ff2cb3-cbff-4dad-83d6-2f63cec4d1a3\") " Nov 03 03:00:20.351635 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.351589 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63ff2cb3-cbff-4dad-83d6-2f63cec4d1a3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63ff2cb3-cbff-4dad-83d6-2f63cec4d1a3-maas-api-service-ca\") pod \"63ff2cb3-cbff-4dad-83d6-2f63cec4d1a3\" (UID: \"63ff2cb3-cbff-4dad-83d6-2f63cec4d1a3\") " Nov 03 03:00:20.352008 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.351981 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63ff2cb3-cbff-4dad-83d6-2f63cec4d1a3-maas-api-service-ca" pod "63ff2cb3-cbff-4dad-83d6-2f63cec4d1a3" (UID: "63ff2cb3-cbff-4dad-83d6-2f63cec4d1a3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:00:20.353812 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.353785 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63ff2cb3-cbff-4dad-83d6-2f63cec4d1a3-kube-api-access-xkxzw" pod "63ff2cb3-cbff-4dad-83d6-2f63cec4d1a3" (UID: "63ff2cb3-cbff-4dad-83d6-2f63cec4d1a3"). InnerVolumeSpecName "kube-api-access-xkxzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:00:20.362226 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.362205 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894580-fvqqd" Nov 03 03:00:20.410803 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.410772 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894580-z9gsn"] Nov 03 03:00:20.452262 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.452236 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad92a9ca-ff2e-4e29-a984-2322317dc196-kube-api-access-4j9x7\" (UniqueName: \"kubernetes.io/projected/ad92a9ca-ff2e-4e29-a984-2322317dc196-kube-api-access-4j9x7\") pod \"ad92a9ca-ff2e-4e29-a984-2322317dc196\" (UID: \"ad92a9ca-ff2e-4e29-a984-2322317dc196\") " Nov 03 03:00:20.452525 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.452436 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad92a9ca-ff2e-4e29-a984-2322317dc196-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad92a9ca-ff2e-4e29-a984-2322317dc196-maas-api-service-ca\") pod \"ad92a9ca-ff2e-4e29-a984-2322317dc196\" (UID: \"ad92a9ca-ff2e-4e29-a984-2322317dc196\") " Nov 03 03:00:20.452799 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.452773 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad92a9ca-ff2e-4e29-a984-2322317dc196-maas-api-service-ca" pod "ad92a9ca-ff2e-4e29-a984-2322317dc196" (UID: "ad92a9ca-ff2e-4e29-a984-2322317dc196"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:00:20.452799 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.452793 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkxzw\" (UniqueName: \"kubernetes.io/projected/63ff2cb3-cbff-4dad-83d6-2f63cec4d1a3-kube-api-access-xkxzw\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:20.452977 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.452809 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63ff2cb3-cbff-4dad-83d6-2f63cec4d1a3-maas-api-service-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:20.454301 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.454281 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad92a9ca-ff2e-4e29-a984-2322317dc196-kube-api-access-4j9x7" pod "ad92a9ca-ff2e-4e29-a984-2322317dc196" (UID: "ad92a9ca-ff2e-4e29-a984-2322317dc196"). InnerVolumeSpecName "kube-api-access-4j9x7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:00:20.528443 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.528416 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894580-z9gsn" Nov 03 03:00:20.554014 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.553986 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad92a9ca-ff2e-4e29-a984-2322317dc196-maas-api-service-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:20.554014 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.554011 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4j9x7\" (UniqueName: \"kubernetes.io/projected/ad92a9ca-ff2e-4e29-a984-2322317dc196-kube-api-access-4j9x7\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:20.631753 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.631720 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc14f6-29894580-vhmch"] Nov 03 03:00:20.635570 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.635550 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc14f6-29894580-vhmch"] Nov 03 03:00:20.655257 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.655233 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f5b2e0e-94ec-44be-bde2-364e1022e4d1-kube-api-access-qskqq\" (UniqueName: \"kubernetes.io/projected/6f5b2e0e-94ec-44be-bde2-364e1022e4d1-kube-api-access-qskqq\") pod \"6f5b2e0e-94ec-44be-bde2-364e1022e4d1\" (UID: \"6f5b2e0e-94ec-44be-bde2-364e1022e4d1\") " Nov 03 03:00:20.655375 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.655288 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f5b2e0e-94ec-44be-bde2-364e1022e4d1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f5b2e0e-94ec-44be-bde2-364e1022e4d1-maas-api-service-ca\") pod \"6f5b2e0e-94ec-44be-bde2-364e1022e4d1\" (UID: \"6f5b2e0e-94ec-44be-bde2-364e1022e4d1\") " Nov 03 03:00:20.655678 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.655657 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f5b2e0e-94ec-44be-bde2-364e1022e4d1-maas-api-service-ca" pod "6f5b2e0e-94ec-44be-bde2-364e1022e4d1" (UID: "6f5b2e0e-94ec-44be-bde2-364e1022e4d1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:00:20.657346 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.657321 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f5b2e0e-94ec-44be-bde2-364e1022e4d1-kube-api-access-qskqq" pod "6f5b2e0e-94ec-44be-bde2-364e1022e4d1" (UID: "6f5b2e0e-94ec-44be-bde2-364e1022e4d1"). InnerVolumeSpecName "kube-api-access-qskqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:00:20.756551 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.756523 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qskqq\" (UniqueName: \"kubernetes.io/projected/6f5b2e0e-94ec-44be-bde2-364e1022e4d1-kube-api-access-qskqq\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:20.756551 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.756548 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f5b2e0e-94ec-44be-bde2-364e1022e4d1-maas-api-service-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:20.858641 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.858603 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1684bfe4-f5b9-4b0a-ac70-578052e13b80" path="/var/lib/kubelet/pods/1684bfe4-f5b9-4b0a-ac70-578052e13b80/volumes" Nov 03 03:00:20.859195 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.859166 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="441073fd-d6a9-4c6b-8cbe-f30638ca23ec" path="/var/lib/kubelet/pods/441073fd-d6a9-4c6b-8cbe-f30638ca23ec/volumes" Nov 03 03:00:20.859671 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.859652 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4b45e3d-f2ba-4a40-b4f6-d2fe279e35d6" path="/var/lib/kubelet/pods/b4b45e3d-f2ba-4a40-b4f6-d2fe279e35d6/volumes" Nov 03 03:00:20.860536 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.860518 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba3c3c2b-2350-4bd1-b4c4-6010e888f7e0" path="/var/lib/kubelet/pods/ba3c3c2b-2350-4bd1-b4c4-6010e888f7e0/volumes" Nov 03 03:00:20.860924 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.860911 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0796b4a-9e93-4cfb-b327-14e788b9dead" path="/var/lib/kubelet/pods/c0796b4a-9e93-4cfb-b327-14e788b9dead/volumes" Nov 03 03:00:20.902583 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.902559 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894580-5jvt2"] Nov 03 03:00:20.905042 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:20.905022 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894580-5jvt2"] Nov 03 03:00:21.033528 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:21.033444 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894580-fvqqd" event={"ID":"ad92a9ca-ff2e-4e29-a984-2322317dc196","Type":"ContainerDied","Data":"c174cb8c589512efb6e1309cb0b9c7dc52cb1e4dd64792dd98ab8af56b5b273b"} Nov 03 03:00:21.033528 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:21.033510 5605 scope.go:122] "RemoveContainer" containerID="ba45b80f2a9f85ac65c61fd44221634bb6fa955f3a5ff1a30695cadfeb43dafd" Nov 03 03:00:21.034041 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:21.033448 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894580-fvqqd" Nov 03 03:00:21.037938 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:21.037920 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894580-9t987" Nov 03 03:00:21.038064 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:21.037938 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894580-9t987" event={"ID":"63ff2cb3-cbff-4dad-83d6-2f63cec4d1a3","Type":"ContainerDied","Data":"dfc7be7c50721ff46020a42a5e15c83bcb0f39a6bca00514392a6d1eea5924e9"} Nov 03 03:00:21.044399 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:21.039717 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894580-z9gsn" event={"ID":"6f5b2e0e-94ec-44be-bde2-364e1022e4d1","Type":"ContainerDied","Data":"d1a0f47f730a293bfbc9610bfeb8bf27fbb4a9daab71d4b19f82e25e58a3d02d"} Nov 03 03:00:21.044399 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:21.039803 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894580-z9gsn" Nov 03 03:00:21.049658 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:21.049640 5605 scope.go:122] "RemoveContainer" containerID="f8e506efdaa610c629f38cbabeb34ba0f0b2dded0e3fbe78d93d8b0e0045019a" Nov 03 03:00:21.058121 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:21.058105 5605 scope.go:122] "RemoveContainer" containerID="6b8400c9f2e2038245db62b0934bdb935035237556398b1407b156e77515eb8c" Nov 03 03:00:21.223367 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:21.223321 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj"] Nov 03 03:00:21.231191 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:21.231156 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-djsqj"] Nov 03 03:00:21.430748 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:21.430716 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894580-79l9g"] Nov 03 03:00:21.443338 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:21.443306 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894580-z9gsn"] Nov 03 03:00:21.449592 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:21.449568 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1bccbd-29894580-z9gsn"] Nov 03 03:00:21.551474 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:21.551444 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894580-9t987"] Nov 03 03:00:21.555322 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:21.555293 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f735f4-29894580-9t987"] Nov 03 03:00:21.575582 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:21.575563 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894580-79l9g" Nov 03 03:00:21.656668 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:21.656644 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894580-fvqqd"] Nov 03 03:00:21.659817 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:21.659797 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-521f1f-29894580-fvqqd"] Nov 03 03:00:21.671782 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:21.671760 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75117649-d9a9-4cde-a8ff-a29010aec224-kube-api-access-92fl6\" (UniqueName: \"kubernetes.io/projected/75117649-d9a9-4cde-a8ff-a29010aec224-kube-api-access-92fl6\") pod \"75117649-d9a9-4cde-a8ff-a29010aec224\" (UID: \"75117649-d9a9-4cde-a8ff-a29010aec224\") " Nov 03 03:00:21.671975 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:21.671956 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75117649-d9a9-4cde-a8ff-a29010aec224-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75117649-d9a9-4cde-a8ff-a29010aec224-maas-api-service-ca\") pod \"75117649-d9a9-4cde-a8ff-a29010aec224\" (UID: \"75117649-d9a9-4cde-a8ff-a29010aec224\") " Nov 03 03:00:21.672312 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:21.672291 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75117649-d9a9-4cde-a8ff-a29010aec224-maas-api-service-ca" pod "75117649-d9a9-4cde-a8ff-a29010aec224" (UID: "75117649-d9a9-4cde-a8ff-a29010aec224"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:00:21.672466 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:21.672447 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75117649-d9a9-4cde-a8ff-a29010aec224-maas-api-service-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:21.673869 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:21.673848 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75117649-d9a9-4cde-a8ff-a29010aec224-kube-api-access-92fl6" pod "75117649-d9a9-4cde-a8ff-a29010aec224" (UID: "75117649-d9a9-4cde-a8ff-a29010aec224"). InnerVolumeSpecName "kube-api-access-92fl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:00:21.774504 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:21.774420 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92fl6\" (UniqueName: \"kubernetes.io/projected/75117649-d9a9-4cde-a8ff-a29010aec224-kube-api-access-92fl6\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:22.052607 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:22.052581 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894580-79l9g" Nov 03 03:00:22.053051 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:22.052609 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894580-79l9g" event={"ID":"75117649-d9a9-4cde-a8ff-a29010aec224","Type":"ContainerDied","Data":"d8d4fe7990c9ebcdad17b5772f9859b143f4f47cfe03ab2b85e9a93998281a42"} Nov 03 03:00:22.053051 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:22.052666 5605 scope.go:122] "RemoveContainer" containerID="c345cbef33ff444f2eac759f6ab05fc5f2d919332dc80992884c414d10394874" Nov 03 03:00:22.205440 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:22.205413 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894580-79l9g"] Nov 03 03:00:22.209040 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:22.209014 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894580-79l9g"] Nov 03 03:00:22.850743 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:22.850714 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="115ad04c-ad71-4cf1-9f39-a22ed3224fca" path="/var/lib/kubelet/pods/115ad04c-ad71-4cf1-9f39-a22ed3224fca/volumes" Nov 03 03:00:22.851104 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:22.851091 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63ff2cb3-cbff-4dad-83d6-2f63cec4d1a3" path="/var/lib/kubelet/pods/63ff2cb3-cbff-4dad-83d6-2f63cec4d1a3/volumes" Nov 03 03:00:22.851499 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:22.851485 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f5b2e0e-94ec-44be-bde2-364e1022e4d1" path="/var/lib/kubelet/pods/6f5b2e0e-94ec-44be-bde2-364e1022e4d1/volumes" Nov 03 03:00:22.852156 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:22.852142 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75117649-d9a9-4cde-a8ff-a29010aec224" path="/var/lib/kubelet/pods/75117649-d9a9-4cde-a8ff-a29010aec224/volumes" Nov 03 03:00:22.852579 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:22.852565 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad92a9ca-ff2e-4e29-a984-2322317dc196" path="/var/lib/kubelet/pods/ad92a9ca-ff2e-4e29-a984-2322317dc196/volumes" Nov 03 03:00:22.852925 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:22.852914 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce5ce822-475e-40bd-9658-cd31fc3e8eb1" path="/var/lib/kubelet/pods/ce5ce822-475e-40bd-9658-cd31fc3e8eb1/volumes" Nov 03 03:00:23.915741 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:00:23.915699 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:00:27.443292 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:27.443259 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:00:41.594394 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:41.594344 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-dc14f6-r8pvg"] Nov 03 03:00:41.640724 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:41.640694 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-dc14f6-r8pvg"] Nov 03 03:00:41.997210 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:41.997136 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk"] Nov 03 03:00:42.005792 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:42.005764 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh"] Nov 03 03:00:42.005991 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:42.005966 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh" podUID="e99c38f1-3005-4582-86d8-71e93dd4c1b8" containerName="istio-proxy" containerID="cri-o://3550b274544a13b74102ca60034b637e7b754477895e5f4adfe63a132bcefb48" gracePeriod=30 Nov 03 03:00:42.182608 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:42.182537 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk" podUID="2905ac02-0ee5-4c4d-bf53-f51adabf9744" containerName="istio-proxy" containerID="cri-o://ee1909a1d7abc5ed8895fb454e848c16a18a5b6e51571d3e7a1c756e8abe102b" gracePeriod=30 Nov 03 03:00:42.850476 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:42.850442 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ea0b63b-c7c4-4d08-b281-e29e6c139462" path="/var/lib/kubelet/pods/7ea0b63b-c7c4-4d08-b281-e29e6c139462/volumes" Nov 03 03:00:46.888050 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888018 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e003baac-rb25p"] Nov 03 03:00:46.888501 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888459 5605 cpu_manager.go:414] "containerMap: removing container" podUID="2f68e7c9-3df8-4df4-a9d8-0544d285c291" containerName="payload-processing" Nov 03 03:00:46.888501 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888472 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f68e7c9-3df8-4df4-a9d8-0544d285c291" containerName="payload-processing" Nov 03 03:00:46.888501 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888483 5605 cpu_manager.go:414] "containerMap: removing container" podUID="ba3c3c2b-2350-4bd1-b4c4-6010e888f7e0" containerName="cleanup" Nov 03 03:00:46.888501 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888488 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba3c3c2b-2350-4bd1-b4c4-6010e888f7e0" containerName="cleanup" Nov 03 03:00:46.888501 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888495 5605 cpu_manager.go:414] "containerMap: removing container" podUID="b4b45e3d-f2ba-4a40-b4f6-d2fe279e35d6" containerName="cleanup" Nov 03 03:00:46.888501 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888500 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4b45e3d-f2ba-4a40-b4f6-d2fe279e35d6" containerName="cleanup" Nov 03 03:00:46.888706 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888514 5605 cpu_manager.go:414] "containerMap: removing container" podUID="083f54a5-48ba-46e3-ab36-2bfaec4a3c9f" containerName="cleanup" Nov 03 03:00:46.888706 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888519 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="083f54a5-48ba-46e3-ab36-2bfaec4a3c9f" containerName="cleanup" Nov 03 03:00:46.888706 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888524 5605 cpu_manager.go:414] "containerMap: removing container" podUID="ce5ce822-475e-40bd-9658-cd31fc3e8eb1" containerName="cleanup" Nov 03 03:00:46.888706 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888529 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce5ce822-475e-40bd-9658-cd31fc3e8eb1" containerName="cleanup" Nov 03 03:00:46.888706 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888535 5605 cpu_manager.go:414] "containerMap: removing container" podUID="b4b45e3d-f2ba-4a40-b4f6-d2fe279e35d6" containerName="cleanup" Nov 03 03:00:46.888706 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888539 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4b45e3d-f2ba-4a40-b4f6-d2fe279e35d6" containerName="cleanup" Nov 03 03:00:46.888706 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888546 5605 cpu_manager.go:414] "containerMap: removing container" podUID="6f5b2e0e-94ec-44be-bde2-364e1022e4d1" containerName="cleanup" Nov 03 03:00:46.888706 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888550 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f5b2e0e-94ec-44be-bde2-364e1022e4d1" containerName="cleanup" Nov 03 03:00:46.888706 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888560 5605 cpu_manager.go:414] "containerMap: removing container" podUID="75117649-d9a9-4cde-a8ff-a29010aec224" containerName="cleanup" Nov 03 03:00:46.888706 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888564 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75117649-d9a9-4cde-a8ff-a29010aec224" containerName="cleanup" Nov 03 03:00:46.888706 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888570 5605 cpu_manager.go:414] "containerMap: removing container" podUID="c0796b4a-9e93-4cfb-b327-14e788b9dead" containerName="cleanup" Nov 03 03:00:46.888706 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888575 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0796b4a-9e93-4cfb-b327-14e788b9dead" containerName="cleanup" Nov 03 03:00:46.888706 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888580 5605 cpu_manager.go:414] "containerMap: removing container" podUID="63ff2cb3-cbff-4dad-83d6-2f63cec4d1a3" containerName="cleanup" Nov 03 03:00:46.888706 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888584 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63ff2cb3-cbff-4dad-83d6-2f63cec4d1a3" containerName="cleanup" Nov 03 03:00:46.888706 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888589 5605 cpu_manager.go:414] "containerMap: removing container" podUID="ad92a9ca-ff2e-4e29-a984-2322317dc196" containerName="cleanup" Nov 03 03:00:46.888706 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888594 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad92a9ca-ff2e-4e29-a984-2322317dc196" containerName="cleanup" Nov 03 03:00:46.888706 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888602 5605 cpu_manager.go:414] "containerMap: removing container" podUID="1684bfe4-f5b9-4b0a-ac70-578052e13b80" containerName="cleanup" Nov 03 03:00:46.888706 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888606 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1684bfe4-f5b9-4b0a-ac70-578052e13b80" containerName="cleanup" Nov 03 03:00:46.888706 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888613 5605 cpu_manager.go:414] "containerMap: removing container" podUID="e5fa99f5-77dc-443e-a11b-9035ca6baf5a" containerName="payload-pre-processing" Nov 03 03:00:46.888706 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888617 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5fa99f5-77dc-443e-a11b-9035ca6baf5a" containerName="payload-pre-processing" Nov 03 03:00:46.888706 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888626 5605 cpu_manager.go:414] "containerMap: removing container" podUID="083f54a5-48ba-46e3-ab36-2bfaec4a3c9f" containerName="cleanup" Nov 03 03:00:46.888706 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888630 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="083f54a5-48ba-46e3-ab36-2bfaec4a3c9f" containerName="cleanup" Nov 03 03:00:46.888706 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888637 5605 cpu_manager.go:414] "containerMap: removing container" podUID="6f5b2e0e-94ec-44be-bde2-364e1022e4d1" containerName="cleanup" Nov 03 03:00:46.888706 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888641 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f5b2e0e-94ec-44be-bde2-364e1022e4d1" containerName="cleanup" Nov 03 03:00:46.888706 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888648 5605 cpu_manager.go:414] "containerMap: removing container" podUID="c0796b4a-9e93-4cfb-b327-14e788b9dead" containerName="cleanup" Nov 03 03:00:46.888706 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888653 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0796b4a-9e93-4cfb-b327-14e788b9dead" containerName="cleanup" Nov 03 03:00:46.888706 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888662 5605 cpu_manager.go:414] "containerMap: removing container" podUID="441073fd-d6a9-4c6b-8cbe-f30638ca23ec" containerName="cleanup" Nov 03 03:00:46.888706 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888667 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="441073fd-d6a9-4c6b-8cbe-f30638ca23ec" containerName="cleanup" Nov 03 03:00:46.888706 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888676 5605 cpu_manager.go:414] "containerMap: removing container" podUID="63ff2cb3-cbff-4dad-83d6-2f63cec4d1a3" containerName="cleanup" Nov 03 03:00:46.888706 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888680 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63ff2cb3-cbff-4dad-83d6-2f63cec4d1a3" containerName="cleanup" Nov 03 03:00:46.888706 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888686 5605 cpu_manager.go:414] "containerMap: removing container" podUID="c0796b4a-9e93-4cfb-b327-14e788b9dead" containerName="cleanup" Nov 03 03:00:46.888706 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888690 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0796b4a-9e93-4cfb-b327-14e788b9dead" containerName="cleanup" Nov 03 03:00:46.888706 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888697 5605 cpu_manager.go:414] "containerMap: removing container" podUID="ad92a9ca-ff2e-4e29-a984-2322317dc196" containerName="cleanup" Nov 03 03:00:46.888706 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888701 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad92a9ca-ff2e-4e29-a984-2322317dc196" containerName="cleanup" Nov 03 03:00:46.888706 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888708 5605 cpu_manager.go:414] "containerMap: removing container" podUID="ba3c3c2b-2350-4bd1-b4c4-6010e888f7e0" containerName="cleanup" Nov 03 03:00:46.888706 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888713 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba3c3c2b-2350-4bd1-b4c4-6010e888f7e0" containerName="cleanup" Nov 03 03:00:46.888706 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888719 5605 cpu_manager.go:414] "containerMap: removing container" podUID="083f54a5-48ba-46e3-ab36-2bfaec4a3c9f" containerName="cleanup" Nov 03 03:00:46.888706 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888724 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="083f54a5-48ba-46e3-ab36-2bfaec4a3c9f" containerName="cleanup" Nov 03 03:00:46.889891 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888732 5605 cpu_manager.go:414] "containerMap: removing container" podUID="1684bfe4-f5b9-4b0a-ac70-578052e13b80" containerName="cleanup" Nov 03 03:00:46.889891 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888737 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1684bfe4-f5b9-4b0a-ac70-578052e13b80" containerName="cleanup" Nov 03 03:00:46.889891 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888744 5605 cpu_manager.go:414] "containerMap: removing container" podUID="75117649-d9a9-4cde-a8ff-a29010aec224" containerName="cleanup" Nov 03 03:00:46.889891 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888748 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75117649-d9a9-4cde-a8ff-a29010aec224" containerName="cleanup" Nov 03 03:00:46.889891 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888755 5605 cpu_manager.go:414] "containerMap: removing container" podUID="441073fd-d6a9-4c6b-8cbe-f30638ca23ec" containerName="cleanup" Nov 03 03:00:46.889891 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888759 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="441073fd-d6a9-4c6b-8cbe-f30638ca23ec" containerName="cleanup" Nov 03 03:00:46.889891 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888765 5605 cpu_manager.go:414] "containerMap: removing container" podUID="441073fd-d6a9-4c6b-8cbe-f30638ca23ec" containerName="cleanup" Nov 03 03:00:46.889891 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888769 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="441073fd-d6a9-4c6b-8cbe-f30638ca23ec" containerName="cleanup" Nov 03 03:00:46.889891 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888775 5605 cpu_manager.go:414] "containerMap: removing container" podUID="115ad04c-ad71-4cf1-9f39-a22ed3224fca" containerName="istio-proxy" Nov 03 03:00:46.889891 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888779 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="115ad04c-ad71-4cf1-9f39-a22ed3224fca" containerName="istio-proxy" Nov 03 03:00:46.889891 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888789 5605 cpu_manager.go:414] "containerMap: removing container" podUID="ad92a9ca-ff2e-4e29-a984-2322317dc196" containerName="cleanup" Nov 03 03:00:46.889891 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888793 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad92a9ca-ff2e-4e29-a984-2322317dc196" containerName="cleanup" Nov 03 03:00:46.889891 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888800 5605 cpu_manager.go:414] "containerMap: removing container" podUID="63ff2cb3-cbff-4dad-83d6-2f63cec4d1a3" containerName="cleanup" Nov 03 03:00:46.889891 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888804 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63ff2cb3-cbff-4dad-83d6-2f63cec4d1a3" containerName="cleanup" Nov 03 03:00:46.889891 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888811 5605 cpu_manager.go:414] "containerMap: removing container" podUID="ce5ce822-475e-40bd-9658-cd31fc3e8eb1" containerName="cleanup" Nov 03 03:00:46.889891 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888815 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce5ce822-475e-40bd-9658-cd31fc3e8eb1" containerName="cleanup" Nov 03 03:00:46.889891 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888876 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="083f54a5-48ba-46e3-ab36-2bfaec4a3c9f" containerName="cleanup" Nov 03 03:00:46.889891 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888883 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce5ce822-475e-40bd-9658-cd31fc3e8eb1" containerName="cleanup" Nov 03 03:00:46.889891 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888889 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4b45e3d-f2ba-4a40-b4f6-d2fe279e35d6" containerName="cleanup" Nov 03 03:00:46.889891 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888897 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f5b2e0e-94ec-44be-bde2-364e1022e4d1" containerName="cleanup" Nov 03 03:00:46.889891 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888902 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="75117649-d9a9-4cde-a8ff-a29010aec224" containerName="cleanup" Nov 03 03:00:46.889891 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888907 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0796b4a-9e93-4cfb-b327-14e788b9dead" containerName="cleanup" Nov 03 03:00:46.889891 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888911 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="63ff2cb3-cbff-4dad-83d6-2f63cec4d1a3" containerName="cleanup" Nov 03 03:00:46.889891 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888917 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad92a9ca-ff2e-4e29-a984-2322317dc196" containerName="cleanup" Nov 03 03:00:46.889891 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888922 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="1684bfe4-f5b9-4b0a-ac70-578052e13b80" containerName="cleanup" Nov 03 03:00:46.889891 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888927 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5fa99f5-77dc-443e-a11b-9035ca6baf5a" containerName="payload-pre-processing" Nov 03 03:00:46.889891 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888932 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f5b2e0e-94ec-44be-bde2-364e1022e4d1" containerName="cleanup" Nov 03 03:00:46.889891 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888937 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="083f54a5-48ba-46e3-ab36-2bfaec4a3c9f" containerName="cleanup" Nov 03 03:00:46.889891 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888942 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f5b2e0e-94ec-44be-bde2-364e1022e4d1" containerName="cleanup" Nov 03 03:00:46.889891 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888946 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0796b4a-9e93-4cfb-b327-14e788b9dead" containerName="cleanup" Nov 03 03:00:46.889891 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888951 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="441073fd-d6a9-4c6b-8cbe-f30638ca23ec" containerName="cleanup" Nov 03 03:00:46.889891 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888956 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="63ff2cb3-cbff-4dad-83d6-2f63cec4d1a3" containerName="cleanup" Nov 03 03:00:46.889891 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888961 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0796b4a-9e93-4cfb-b327-14e788b9dead" containerName="cleanup" Nov 03 03:00:46.889891 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888970 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad92a9ca-ff2e-4e29-a984-2322317dc196" containerName="cleanup" Nov 03 03:00:46.889891 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888976 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce5ce822-475e-40bd-9658-cd31fc3e8eb1" containerName="cleanup" Nov 03 03:00:46.889891 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888982 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba3c3c2b-2350-4bd1-b4c4-6010e888f7e0" containerName="cleanup" Nov 03 03:00:46.889891 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888987 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="083f54a5-48ba-46e3-ab36-2bfaec4a3c9f" containerName="cleanup" Nov 03 03:00:46.889891 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888992 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="1684bfe4-f5b9-4b0a-ac70-578052e13b80" containerName="cleanup" Nov 03 03:00:46.889891 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.888997 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba3c3c2b-2350-4bd1-b4c4-6010e888f7e0" containerName="cleanup" Nov 03 03:00:46.889891 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.889004 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="75117649-d9a9-4cde-a8ff-a29010aec224" containerName="cleanup" Nov 03 03:00:46.889891 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.889009 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="441073fd-d6a9-4c6b-8cbe-f30638ca23ec" containerName="cleanup" Nov 03 03:00:46.889891 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.889013 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="115ad04c-ad71-4cf1-9f39-a22ed3224fca" containerName="istio-proxy" Nov 03 03:00:46.889891 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.889019 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="1684bfe4-f5b9-4b0a-ac70-578052e13b80" containerName="cleanup" Nov 03 03:00:46.889891 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.889024 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="63ff2cb3-cbff-4dad-83d6-2f63cec4d1a3" containerName="cleanup" Nov 03 03:00:46.889891 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.889029 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f68e7c9-3df8-4df4-a9d8-0544d285c291" containerName="payload-processing" Nov 03 03:00:46.889891 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.889035 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4b45e3d-f2ba-4a40-b4f6-d2fe279e35d6" containerName="cleanup" Nov 03 03:00:46.892293 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.892276 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e003baac-rb25p" Nov 03 03:00:46.946888 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.946855 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e003baac-rb25p"] Nov 03 03:00:46.961066 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.961031 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1679ecf5-9e89-40fe-8fee-dbd5b74d3a7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e003baac-rb25p\" (UID: \"1679ecf5-9e89-40fe-8fee-dbd5b74d3a7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e003baac-rb25p" Nov 03 03:00:46.961247 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.961193 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76m8k\" (UniqueName: \"kubernetes.io/projected/1679ecf5-9e89-40fe-8fee-dbd5b74d3a7a-kube-api-access-76m8k\") pod \"maas-api-revoke-sub-e003baac-rb25p\" (UID: \"1679ecf5-9e89-40fe-8fee-dbd5b74d3a7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e003baac-rb25p" Nov 03 03:00:46.961247 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.961225 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1679ecf5-9e89-40fe-8fee-dbd5b74d3a7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e003baac-rb25p\" (UID: \"1679ecf5-9e89-40fe-8fee-dbd5b74d3a7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e003baac-rb25p" Nov 03 03:00:46.990004 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.989969 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:00:46.990004 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.990002 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:00:46.990217 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.990022 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:00:46.990217 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:46.990059 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 03:00:47.063047 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:47.063020 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76m8k\" (UniqueName: \"kubernetes.io/projected/1679ecf5-9e89-40fe-8fee-dbd5b74d3a7a-kube-api-access-76m8k\") pod \"maas-api-revoke-sub-e003baac-rb25p\" (UID: \"1679ecf5-9e89-40fe-8fee-dbd5b74d3a7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e003baac-rb25p" Nov 03 03:00:47.063198 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:47.063054 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1679ecf5-9e89-40fe-8fee-dbd5b74d3a7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e003baac-rb25p\" (UID: \"1679ecf5-9e89-40fe-8fee-dbd5b74d3a7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e003baac-rb25p" Nov 03 03:00:47.063268 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:47.063194 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1679ecf5-9e89-40fe-8fee-dbd5b74d3a7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e003baac-rb25p\" (UID: \"1679ecf5-9e89-40fe-8fee-dbd5b74d3a7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e003baac-rb25p" Nov 03 03:00:47.063891 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:47.063871 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1679ecf5-9e89-40fe-8fee-dbd5b74d3a7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e003baac-rb25p\" (UID: \"1679ecf5-9e89-40fe-8fee-dbd5b74d3a7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e003baac-rb25p" Nov 03 03:00:47.065447 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:47.065428 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1679ecf5-9e89-40fe-8fee-dbd5b74d3a7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e003baac-rb25p\" (UID: \"1679ecf5-9e89-40fe-8fee-dbd5b74d3a7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e003baac-rb25p" Nov 03 03:00:47.117223 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:47.117197 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76m8k\" (UniqueName: \"kubernetes.io/projected/1679ecf5-9e89-40fe-8fee-dbd5b74d3a7a-kube-api-access-76m8k\") pod \"maas-api-revoke-sub-e003baac-rb25p\" (UID: \"1679ecf5-9e89-40fe-8fee-dbd5b74d3a7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e003baac-rb25p" Nov 03 03:00:47.211910 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:47.211659 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e003baac-rb25p" Nov 03 03:00:47.221773 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:47.221738 5605 generic.go:356] "Generic (PLEG): container finished" podID="e99c38f1-3005-4582-86d8-71e93dd4c1b8" containerID="3550b274544a13b74102ca60034b637e7b754477895e5f4adfe63a132bcefb48" exitCode=0 Nov 03 03:00:47.221941 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:47.221807 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh" event={"ID":"e99c38f1-3005-4582-86d8-71e93dd4c1b8","Type":"ContainerDied","Data":"3550b274544a13b74102ca60034b637e7b754477895e5f4adfe63a132bcefb48"} Nov 03 03:00:47.301536 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:47.301506 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh" Nov 03 03:00:47.366557 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:47.366521 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e99c38f1-3005-4582-86d8-71e93dd4c1b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e99c38f1-3005-4582-86d8-71e93dd4c1b8-credential-socket\") pod \"e99c38f1-3005-4582-86d8-71e93dd4c1b8\" (UID: \"e99c38f1-3005-4582-86d8-71e93dd4c1b8\") " Nov 03 03:00:47.366742 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:47.366573 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e99c38f1-3005-4582-86d8-71e93dd4c1b8-kube-api-access-gfvnm\" (UniqueName: \"kubernetes.io/projected/e99c38f1-3005-4582-86d8-71e93dd4c1b8-kube-api-access-gfvnm\") pod \"e99c38f1-3005-4582-86d8-71e93dd4c1b8\" (UID: \"e99c38f1-3005-4582-86d8-71e93dd4c1b8\") " Nov 03 03:00:47.366742 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:47.366597 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e99c38f1-3005-4582-86d8-71e93dd4c1b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e99c38f1-3005-4582-86d8-71e93dd4c1b8-istio-data\") pod \"e99c38f1-3005-4582-86d8-71e93dd4c1b8\" (UID: \"e99c38f1-3005-4582-86d8-71e93dd4c1b8\") " Nov 03 03:00:47.366742 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:47.366667 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e99c38f1-3005-4582-86d8-71e93dd4c1b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e99c38f1-3005-4582-86d8-71e93dd4c1b8-workload-socket\") pod \"e99c38f1-3005-4582-86d8-71e93dd4c1b8\" (UID: \"e99c38f1-3005-4582-86d8-71e93dd4c1b8\") " Nov 03 03:00:47.366742 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:47.366692 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e99c38f1-3005-4582-86d8-71e93dd4c1b8-istio-token\" (UniqueName: \"kubernetes.io/projected/e99c38f1-3005-4582-86d8-71e93dd4c1b8-istio-token\") pod \"e99c38f1-3005-4582-86d8-71e93dd4c1b8\" (UID: \"e99c38f1-3005-4582-86d8-71e93dd4c1b8\") " Nov 03 03:00:47.366742 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:47.366734 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e99c38f1-3005-4582-86d8-71e93dd4c1b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e99c38f1-3005-4582-86d8-71e93dd4c1b8-workload-certs\") pod \"e99c38f1-3005-4582-86d8-71e93dd4c1b8\" (UID: \"e99c38f1-3005-4582-86d8-71e93dd4c1b8\") " Nov 03 03:00:47.367059 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:47.366777 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e99c38f1-3005-4582-86d8-71e93dd4c1b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e99c38f1-3005-4582-86d8-71e93dd4c1b8-istiod-ca-cert\") pod \"e99c38f1-3005-4582-86d8-71e93dd4c1b8\" (UID: \"e99c38f1-3005-4582-86d8-71e93dd4c1b8\") " Nov 03 03:00:47.367059 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:47.366806 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e99c38f1-3005-4582-86d8-71e93dd4c1b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e99c38f1-3005-4582-86d8-71e93dd4c1b8-istio-podinfo\") pod \"e99c38f1-3005-4582-86d8-71e93dd4c1b8\" (UID: \"e99c38f1-3005-4582-86d8-71e93dd4c1b8\") " Nov 03 03:00:47.367059 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:47.366877 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e99c38f1-3005-4582-86d8-71e93dd4c1b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e99c38f1-3005-4582-86d8-71e93dd4c1b8-istio-envoy\") pod \"e99c38f1-3005-4582-86d8-71e93dd4c1b8\" (UID: \"e99c38f1-3005-4582-86d8-71e93dd4c1b8\") " Nov 03 03:00:47.367687 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:47.367656 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e99c38f1-3005-4582-86d8-71e93dd4c1b8-workload-socket" pod "e99c38f1-3005-4582-86d8-71e93dd4c1b8" (UID: "e99c38f1-3005-4582-86d8-71e93dd4c1b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:00:47.367687 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:47.367671 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e99c38f1-3005-4582-86d8-71e93dd4c1b8-workload-certs" pod "e99c38f1-3005-4582-86d8-71e93dd4c1b8" (UID: "e99c38f1-3005-4582-86d8-71e93dd4c1b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:00:47.367874 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:47.367692 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e99c38f1-3005-4582-86d8-71e93dd4c1b8-istio-data" pod "e99c38f1-3005-4582-86d8-71e93dd4c1b8" (UID: "e99c38f1-3005-4582-86d8-71e93dd4c1b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:00:47.367874 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:47.367807 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e99c38f1-3005-4582-86d8-71e93dd4c1b8-credential-socket" pod "e99c38f1-3005-4582-86d8-71e93dd4c1b8" (UID: "e99c38f1-3005-4582-86d8-71e93dd4c1b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:00:47.367999 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:47.367880 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e99c38f1-3005-4582-86d8-71e93dd4c1b8-istiod-ca-cert" pod "e99c38f1-3005-4582-86d8-71e93dd4c1b8" (UID: "e99c38f1-3005-4582-86d8-71e93dd4c1b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:00:47.370096 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:47.370065 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e99c38f1-3005-4582-86d8-71e93dd4c1b8-istio-envoy" pod "e99c38f1-3005-4582-86d8-71e93dd4c1b8" (UID: "e99c38f1-3005-4582-86d8-71e93dd4c1b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:00:47.370214 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:47.370198 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e99c38f1-3005-4582-86d8-71e93dd4c1b8-istio-podinfo" pod "e99c38f1-3005-4582-86d8-71e93dd4c1b8" (UID: "e99c38f1-3005-4582-86d8-71e93dd4c1b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:00:47.370524 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:47.370502 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e99c38f1-3005-4582-86d8-71e93dd4c1b8-istio-token" pod "e99c38f1-3005-4582-86d8-71e93dd4c1b8" (UID: "e99c38f1-3005-4582-86d8-71e93dd4c1b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:00:47.370595 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:47.370553 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e99c38f1-3005-4582-86d8-71e93dd4c1b8-kube-api-access-gfvnm" pod "e99c38f1-3005-4582-86d8-71e93dd4c1b8" (UID: "e99c38f1-3005-4582-86d8-71e93dd4c1b8"). InnerVolumeSpecName "kube-api-access-gfvnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:00:47.438316 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:47.438286 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk" Nov 03 03:00:47.468319 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:47.468287 5605 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e99c38f1-3005-4582-86d8-71e93dd4c1b8-istio-data\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:47.468319 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:47.468318 5605 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e99c38f1-3005-4582-86d8-71e93dd4c1b8-workload-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:47.468536 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:47.468338 5605 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e99c38f1-3005-4582-86d8-71e93dd4c1b8-istio-token\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:47.468536 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:47.468351 5605 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e99c38f1-3005-4582-86d8-71e93dd4c1b8-workload-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:47.468536 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:47.468379 5605 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e99c38f1-3005-4582-86d8-71e93dd4c1b8-istiod-ca-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:47.468536 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:47.468392 5605 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e99c38f1-3005-4582-86d8-71e93dd4c1b8-istio-podinfo\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:47.468536 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:47.468404 5605 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e99c38f1-3005-4582-86d8-71e93dd4c1b8-istio-envoy\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:47.468536 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:47.468415 5605 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e99c38f1-3005-4582-86d8-71e93dd4c1b8-credential-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:47.468536 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:47.468427 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfvnm\" (UniqueName: \"kubernetes.io/projected/e99c38f1-3005-4582-86d8-71e93dd4c1b8-kube-api-access-gfvnm\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:47.569063 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:47.569031 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2905ac02-0ee5-4c4d-bf53-f51adabf9744-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2905ac02-0ee5-4c4d-bf53-f51adabf9744-istio-podinfo\") pod \"2905ac02-0ee5-4c4d-bf53-f51adabf9744\" (UID: \"2905ac02-0ee5-4c4d-bf53-f51adabf9744\") " Nov 03 03:00:47.569262 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:47.569136 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2905ac02-0ee5-4c4d-bf53-f51adabf9744-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2905ac02-0ee5-4c4d-bf53-f51adabf9744-istio-data\") pod \"2905ac02-0ee5-4c4d-bf53-f51adabf9744\" (UID: \"2905ac02-0ee5-4c4d-bf53-f51adabf9744\") " Nov 03 03:00:47.569262 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:47.569168 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2905ac02-0ee5-4c4d-bf53-f51adabf9744-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2905ac02-0ee5-4c4d-bf53-f51adabf9744-istiod-ca-cert\") pod \"2905ac02-0ee5-4c4d-bf53-f51adabf9744\" (UID: \"2905ac02-0ee5-4c4d-bf53-f51adabf9744\") " Nov 03 03:00:47.569262 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:47.569212 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2905ac02-0ee5-4c4d-bf53-f51adabf9744-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2905ac02-0ee5-4c4d-bf53-f51adabf9744-workload-certs\") pod \"2905ac02-0ee5-4c4d-bf53-f51adabf9744\" (UID: \"2905ac02-0ee5-4c4d-bf53-f51adabf9744\") " Nov 03 03:00:47.569262 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:47.569242 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2905ac02-0ee5-4c4d-bf53-f51adabf9744-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2905ac02-0ee5-4c4d-bf53-f51adabf9744-credential-socket\") pod \"2905ac02-0ee5-4c4d-bf53-f51adabf9744\" (UID: \"2905ac02-0ee5-4c4d-bf53-f51adabf9744\") " Nov 03 03:00:47.569578 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:47.569343 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2905ac02-0ee5-4c4d-bf53-f51adabf9744-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2905ac02-0ee5-4c4d-bf53-f51adabf9744-istio-envoy\") pod \"2905ac02-0ee5-4c4d-bf53-f51adabf9744\" (UID: \"2905ac02-0ee5-4c4d-bf53-f51adabf9744\") " Nov 03 03:00:47.569578 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:47.569431 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2905ac02-0ee5-4c4d-bf53-f51adabf9744-kube-api-access-ps7s7\" (UniqueName: \"kubernetes.io/projected/2905ac02-0ee5-4c4d-bf53-f51adabf9744-kube-api-access-ps7s7\") pod \"2905ac02-0ee5-4c4d-bf53-f51adabf9744\" (UID: \"2905ac02-0ee5-4c4d-bf53-f51adabf9744\") " Nov 03 03:00:47.569578 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:47.569461 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2905ac02-0ee5-4c4d-bf53-f51adabf9744-istio-token\" (UniqueName: \"kubernetes.io/projected/2905ac02-0ee5-4c4d-bf53-f51adabf9744-istio-token\") pod \"2905ac02-0ee5-4c4d-bf53-f51adabf9744\" (UID: \"2905ac02-0ee5-4c4d-bf53-f51adabf9744\") " Nov 03 03:00:47.569578 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:47.569501 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2905ac02-0ee5-4c4d-bf53-f51adabf9744-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2905ac02-0ee5-4c4d-bf53-f51adabf9744-workload-socket\") pod \"2905ac02-0ee5-4c4d-bf53-f51adabf9744\" (UID: \"2905ac02-0ee5-4c4d-bf53-f51adabf9744\") " Nov 03 03:00:47.569578 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:47.569503 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2905ac02-0ee5-4c4d-bf53-f51adabf9744-istio-data" pod "2905ac02-0ee5-4c4d-bf53-f51adabf9744" (UID: "2905ac02-0ee5-4c4d-bf53-f51adabf9744"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:00:47.569847 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:47.569724 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2905ac02-0ee5-4c4d-bf53-f51adabf9744-credential-socket" pod "2905ac02-0ee5-4c4d-bf53-f51adabf9744" (UID: "2905ac02-0ee5-4c4d-bf53-f51adabf9744"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:00:47.569847 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:47.569639 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2905ac02-0ee5-4c4d-bf53-f51adabf9744-istiod-ca-cert" pod "2905ac02-0ee5-4c4d-bf53-f51adabf9744" (UID: "2905ac02-0ee5-4c4d-bf53-f51adabf9744"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:00:47.569959 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:47.569910 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2905ac02-0ee5-4c4d-bf53-f51adabf9744-workload-socket" pod "2905ac02-0ee5-4c4d-bf53-f51adabf9744" (UID: "2905ac02-0ee5-4c4d-bf53-f51adabf9744"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:00:47.570062 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:47.570038 5605 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2905ac02-0ee5-4c4d-bf53-f51adabf9744-credential-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:47.570136 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:47.570067 5605 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2905ac02-0ee5-4c4d-bf53-f51adabf9744-workload-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:47.570136 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:47.570085 5605 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2905ac02-0ee5-4c4d-bf53-f51adabf9744-istio-data\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:47.570136 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:47.570102 5605 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2905ac02-0ee5-4c4d-bf53-f51adabf9744-istiod-ca-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:47.570322 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:47.570181 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2905ac02-0ee5-4c4d-bf53-f51adabf9744-workload-certs" pod "2905ac02-0ee5-4c4d-bf53-f51adabf9744" (UID: "2905ac02-0ee5-4c4d-bf53-f51adabf9744"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:00:47.571519 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:47.571497 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2905ac02-0ee5-4c4d-bf53-f51adabf9744-istio-podinfo" pod "2905ac02-0ee5-4c4d-bf53-f51adabf9744" (UID: "2905ac02-0ee5-4c4d-bf53-f51adabf9744"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:00:47.571765 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:47.571739 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2905ac02-0ee5-4c4d-bf53-f51adabf9744-istio-token" pod "2905ac02-0ee5-4c4d-bf53-f51adabf9744" (UID: "2905ac02-0ee5-4c4d-bf53-f51adabf9744"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:00:47.572088 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:47.572074 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2905ac02-0ee5-4c4d-bf53-f51adabf9744-istio-envoy" pod "2905ac02-0ee5-4c4d-bf53-f51adabf9744" (UID: "2905ac02-0ee5-4c4d-bf53-f51adabf9744"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:00:47.572317 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:47.572292 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2905ac02-0ee5-4c4d-bf53-f51adabf9744-kube-api-access-ps7s7" pod "2905ac02-0ee5-4c4d-bf53-f51adabf9744" (UID: "2905ac02-0ee5-4c4d-bf53-f51adabf9744"). InnerVolumeSpecName "kube-api-access-ps7s7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:00:47.588037 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:47.588006 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e003baac-rb25p"] Nov 03 03:00:47.594491 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:00:47.594467 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1679ecf5_9e89_40fe_8fee_dbd5b74d3a7a.slice/crio-71e98ccd55e1af7d01247d92021d1a54984963af40a4272d1eba9d776c5edbec WatchSource:0}: Error finding container 71e98ccd55e1af7d01247d92021d1a54984963af40a4272d1eba9d776c5edbec: Status 404 returned error can't find the container with id 71e98ccd55e1af7d01247d92021d1a54984963af40a4272d1eba9d776c5edbec Nov 03 03:00:47.671691 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:47.671661 5605 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2905ac02-0ee5-4c4d-bf53-f51adabf9744-workload-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:47.671691 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:47.671689 5605 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2905ac02-0ee5-4c4d-bf53-f51adabf9744-istio-envoy\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:47.671842 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:47.671699 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ps7s7\" (UniqueName: \"kubernetes.io/projected/2905ac02-0ee5-4c4d-bf53-f51adabf9744-kube-api-access-ps7s7\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:47.671842 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:47.671710 5605 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2905ac02-0ee5-4c4d-bf53-f51adabf9744-istio-token\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:47.671842 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:47.671719 5605 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2905ac02-0ee5-4c4d-bf53-f51adabf9744-istio-podinfo\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:48.230793 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:48.230756 5605 generic.go:356] "Generic (PLEG): container finished" podID="2905ac02-0ee5-4c4d-bf53-f51adabf9744" containerID="ee1909a1d7abc5ed8895fb454e848c16a18a5b6e51571d3e7a1c756e8abe102b" exitCode=0 Nov 03 03:00:48.231221 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:48.230801 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk" Nov 03 03:00:48.231221 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:48.230848 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk" event={"ID":"2905ac02-0ee5-4c4d-bf53-f51adabf9744","Type":"ContainerDied","Data":"ee1909a1d7abc5ed8895fb454e848c16a18a5b6e51571d3e7a1c756e8abe102b"} Nov 03 03:00:48.231221 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:48.230899 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk" event={"ID":"2905ac02-0ee5-4c4d-bf53-f51adabf9744","Type":"ContainerDied","Data":"ef1071ccb8cdf538c8cee4429874be65195eaaa1b99670c2a3afcb30fcea52c2"} Nov 03 03:00:48.231221 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:48.230925 5605 scope.go:122] "RemoveContainer" containerID="ee1909a1d7abc5ed8895fb454e848c16a18a5b6e51571d3e7a1c756e8abe102b" Nov 03 03:00:48.232380 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:48.232338 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh" Nov 03 03:00:48.232486 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:48.232379 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh" event={"ID":"e99c38f1-3005-4582-86d8-71e93dd4c1b8","Type":"ContainerDied","Data":"dc002eef3cacc20fdaf287356e11106d918e5262dba4983b0238ce96db86bf73"} Nov 03 03:00:48.234130 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:48.234103 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e003baac-rb25p" event={"ID":"1679ecf5-9e89-40fe-8fee-dbd5b74d3a7a","Type":"ContainerStarted","Data":"286f6f3afe5930f0de710349721f7c08c2b8d112ee3b8945e4cd3a0ba6ff84db"} Nov 03 03:00:48.234252 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:48.234153 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e003baac-rb25p" event={"ID":"1679ecf5-9e89-40fe-8fee-dbd5b74d3a7a","Type":"ContainerStarted","Data":"71e98ccd55e1af7d01247d92021d1a54984963af40a4272d1eba9d776c5edbec"} Nov 03 03:00:48.241369 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:48.241174 5605 scope.go:122] "RemoveContainer" containerID="ee1909a1d7abc5ed8895fb454e848c16a18a5b6e51571d3e7a1c756e8abe102b" Nov 03 03:00:48.241556 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:00:48.241518 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee1909a1d7abc5ed8895fb454e848c16a18a5b6e51571d3e7a1c756e8abe102b\": container with ID starting with ee1909a1d7abc5ed8895fb454e848c16a18a5b6e51571d3e7a1c756e8abe102b not found: ID does not exist" containerID="ee1909a1d7abc5ed8895fb454e848c16a18a5b6e51571d3e7a1c756e8abe102b" Nov 03 03:00:48.241626 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:48.241568 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee1909a1d7abc5ed8895fb454e848c16a18a5b6e51571d3e7a1c756e8abe102b"} err="failed to get container status \"ee1909a1d7abc5ed8895fb454e848c16a18a5b6e51571d3e7a1c756e8abe102b\": rpc error: code = NotFound desc = could not find container \"ee1909a1d7abc5ed8895fb454e848c16a18a5b6e51571d3e7a1c756e8abe102b\": container with ID starting with ee1909a1d7abc5ed8895fb454e848c16a18a5b6e51571d3e7a1c756e8abe102b not found: ID does not exist" Nov 03 03:00:48.241626 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:48.241589 5605 scope.go:122] "RemoveContainer" containerID="3550b274544a13b74102ca60034b637e7b754477895e5f4adfe63a132bcefb48" Nov 03 03:00:48.386046 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:48.386012 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh"] Nov 03 03:00:48.432869 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:48.432842 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-csqbh"] Nov 03 03:00:48.494066 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:48.494035 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk"] Nov 03 03:00:48.499588 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:48.499561 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-lpxsk"] Nov 03 03:00:48.607549 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:48.607492 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e003baac-rb25p" podStartSLOduration=2.607477544 podStartE2EDuration="2.607477544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:00:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:00:48.606456263 +0000 UTC m=+4125.865654442" watchObservedRunningTime="2026-11-03 03:00:48.607477544 +0000 UTC m=+4125.866675699" Nov 03 03:00:48.851790 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:48.851751 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2905ac02-0ee5-4c4d-bf53-f51adabf9744" path="/var/lib/kubelet/pods/2905ac02-0ee5-4c4d-bf53-f51adabf9744/volumes" Nov 03 03:00:48.852301 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:48.852286 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e99c38f1-3005-4582-86d8-71e93dd4c1b8" path="/var/lib/kubelet/pods/e99c38f1-3005-4582-86d8-71e93dd4c1b8/volumes" Nov 03 03:00:49.243222 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:49.243120 5605 generic.go:356] "Generic (PLEG): container finished" podID="1679ecf5-9e89-40fe-8fee-dbd5b74d3a7a" containerID="286f6f3afe5930f0de710349721f7c08c2b8d112ee3b8945e4cd3a0ba6ff84db" exitCode=0 Nov 03 03:00:49.243222 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:49.243211 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e003baac-rb25p" event={"ID":"1679ecf5-9e89-40fe-8fee-dbd5b74d3a7a","Type":"ContainerDied","Data":"286f6f3afe5930f0de710349721f7c08c2b8d112ee3b8945e4cd3a0ba6ff84db"} Nov 03 03:00:50.400713 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:50.400681 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e003baac-rb25p" Nov 03 03:00:50.502517 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:50.502430 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1679ecf5-9e89-40fe-8fee-dbd5b74d3a7a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1679ecf5-9e89-40fe-8fee-dbd5b74d3a7a-maas-api-xks-ca\") pod \"1679ecf5-9e89-40fe-8fee-dbd5b74d3a7a\" (UID: \"1679ecf5-9e89-40fe-8fee-dbd5b74d3a7a\") " Nov 03 03:00:50.502517 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:50.502476 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1679ecf5-9e89-40fe-8fee-dbd5b74d3a7a-kube-api-access-76m8k\" (UniqueName: \"kubernetes.io/projected/1679ecf5-9e89-40fe-8fee-dbd5b74d3a7a-kube-api-access-76m8k\") pod \"1679ecf5-9e89-40fe-8fee-dbd5b74d3a7a\" (UID: \"1679ecf5-9e89-40fe-8fee-dbd5b74d3a7a\") " Nov 03 03:00:50.502768 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:50.502598 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1679ecf5-9e89-40fe-8fee-dbd5b74d3a7a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1679ecf5-9e89-40fe-8fee-dbd5b74d3a7a-maas-api-openshift-ca\") pod \"1679ecf5-9e89-40fe-8fee-dbd5b74d3a7a\" (UID: \"1679ecf5-9e89-40fe-8fee-dbd5b74d3a7a\") " Nov 03 03:00:50.502988 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:50.502962 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1679ecf5-9e89-40fe-8fee-dbd5b74d3a7a-maas-api-openshift-ca" pod "1679ecf5-9e89-40fe-8fee-dbd5b74d3a7a" (UID: "1679ecf5-9e89-40fe-8fee-dbd5b74d3a7a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:00:50.503123 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:50.503093 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1679ecf5-9e89-40fe-8fee-dbd5b74d3a7a-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:50.504570 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:50.504548 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1679ecf5-9e89-40fe-8fee-dbd5b74d3a7a-maas-api-xks-ca" pod "1679ecf5-9e89-40fe-8fee-dbd5b74d3a7a" (UID: "1679ecf5-9e89-40fe-8fee-dbd5b74d3a7a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:00:50.504667 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:50.504616 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1679ecf5-9e89-40fe-8fee-dbd5b74d3a7a-kube-api-access-76m8k" pod "1679ecf5-9e89-40fe-8fee-dbd5b74d3a7a" (UID: "1679ecf5-9e89-40fe-8fee-dbd5b74d3a7a"). InnerVolumeSpecName "kube-api-access-76m8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:00:50.603832 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:50.603787 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1679ecf5-9e89-40fe-8fee-dbd5b74d3a7a-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:50.603832 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:50.603827 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76m8k\" (UniqueName: \"kubernetes.io/projected/1679ecf5-9e89-40fe-8fee-dbd5b74d3a7a-kube-api-access-76m8k\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:51.258238 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:51.258147 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e003baac-rb25p" Nov 03 03:00:51.258238 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:51.258170 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e003baac-rb25p" event={"ID":"1679ecf5-9e89-40fe-8fee-dbd5b74d3a7a","Type":"ContainerDied","Data":"71e98ccd55e1af7d01247d92021d1a54984963af40a4272d1eba9d776c5edbec"} Nov 03 03:00:51.258517 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:00:51.258240 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71e98ccd55e1af7d01247d92021d1a54984963af40a4272d1eba9d776c5edbec" Nov 03 03:00:53.913376 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:00:53.913316 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:01:04.955892 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:04.955860 5605 scope.go:122] "RemoveContainer" containerID="58cc5283bd85eb6fe7ab0cf66903ad7e288cdd38a1b997544e9dd4e0edb38593" Nov 03 03:01:05.233818 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:05.233735 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0ffb465-9zhbg"] Nov 03 03:01:05.234202 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:05.234189 5605 cpu_manager.go:414] "containerMap: removing container" podUID="e99c38f1-3005-4582-86d8-71e93dd4c1b8" containerName="istio-proxy" Nov 03 03:01:05.234251 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:05.234203 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e99c38f1-3005-4582-86d8-71e93dd4c1b8" containerName="istio-proxy" Nov 03 03:01:05.234251 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:05.234226 5605 cpu_manager.go:414] "containerMap: removing container" podUID="ce5ce822-475e-40bd-9658-cd31fc3e8eb1" containerName="cleanup" Nov 03 03:01:05.234251 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:05.234231 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce5ce822-475e-40bd-9658-cd31fc3e8eb1" containerName="cleanup" Nov 03 03:01:05.234251 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:05.234240 5605 cpu_manager.go:414] "containerMap: removing container" podUID="b4b45e3d-f2ba-4a40-b4f6-d2fe279e35d6" containerName="cleanup" Nov 03 03:01:05.234251 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:05.234245 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4b45e3d-f2ba-4a40-b4f6-d2fe279e35d6" containerName="cleanup" Nov 03 03:01:05.234251 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:05.234252 5605 cpu_manager.go:414] "containerMap: removing container" podUID="2905ac02-0ee5-4c4d-bf53-f51adabf9744" containerName="istio-proxy" Nov 03 03:01:05.234468 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:05.234257 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2905ac02-0ee5-4c4d-bf53-f51adabf9744" containerName="istio-proxy" Nov 03 03:01:05.234468 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:05.234267 5605 cpu_manager.go:414] "containerMap: removing container" podUID="75117649-d9a9-4cde-a8ff-a29010aec224" containerName="cleanup" Nov 03 03:01:05.234468 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:05.234272 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75117649-d9a9-4cde-a8ff-a29010aec224" containerName="cleanup" Nov 03 03:01:05.234468 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:05.234282 5605 cpu_manager.go:414] "containerMap: removing container" podUID="1684bfe4-f5b9-4b0a-ac70-578052e13b80" containerName="cleanup" Nov 03 03:01:05.234468 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:05.234287 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1684bfe4-f5b9-4b0a-ac70-578052e13b80" containerName="cleanup" Nov 03 03:01:05.234468 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:05.234297 5605 cpu_manager.go:414] "containerMap: removing container" podUID="6f5b2e0e-94ec-44be-bde2-364e1022e4d1" containerName="cleanup" Nov 03 03:01:05.234468 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:05.234302 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f5b2e0e-94ec-44be-bde2-364e1022e4d1" containerName="cleanup" Nov 03 03:01:05.234468 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:05.234317 5605 cpu_manager.go:414] "containerMap: removing container" podUID="ba3c3c2b-2350-4bd1-b4c4-6010e888f7e0" containerName="cleanup" Nov 03 03:01:05.234468 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:05.234322 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba3c3c2b-2350-4bd1-b4c4-6010e888f7e0" containerName="cleanup" Nov 03 03:01:05.234468 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:05.234328 5605 cpu_manager.go:414] "containerMap: removing container" podUID="1679ecf5-9e89-40fe-8fee-dbd5b74d3a7a" containerName="revoke-keys" Nov 03 03:01:05.234468 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:05.234332 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1679ecf5-9e89-40fe-8fee-dbd5b74d3a7a" containerName="revoke-keys" Nov 03 03:01:05.234468 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:05.234431 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4b45e3d-f2ba-4a40-b4f6-d2fe279e35d6" containerName="cleanup" Nov 03 03:01:05.234468 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:05.234438 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="2905ac02-0ee5-4c4d-bf53-f51adabf9744" containerName="istio-proxy" Nov 03 03:01:05.234468 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:05.234448 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="75117649-d9a9-4cde-a8ff-a29010aec224" containerName="cleanup" Nov 03 03:01:05.234468 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:05.234456 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="441073fd-d6a9-4c6b-8cbe-f30638ca23ec" containerName="cleanup" Nov 03 03:01:05.234468 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:05.234464 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad92a9ca-ff2e-4e29-a984-2322317dc196" containerName="cleanup" Nov 03 03:01:05.234468 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:05.234469 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce5ce822-475e-40bd-9658-cd31fc3e8eb1" containerName="cleanup" Nov 03 03:01:05.234468 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:05.234475 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="1679ecf5-9e89-40fe-8fee-dbd5b74d3a7a" containerName="revoke-keys" Nov 03 03:01:05.235045 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:05.234481 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba3c3c2b-2350-4bd1-b4c4-6010e888f7e0" containerName="cleanup" Nov 03 03:01:05.235045 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:05.234488 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="e99c38f1-3005-4582-86d8-71e93dd4c1b8" containerName="istio-proxy" Nov 03 03:01:05.239769 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:05.239747 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ffb465-9zhbg" Nov 03 03:01:05.337225 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:05.337196 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:01:05.337225 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:05.337215 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 03:01:05.337225 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:05.337207 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:01:05.337507 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:05.337207 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:01:05.340166 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:05.340140 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0ffb465-9zhbg"] Nov 03 03:01:05.364467 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:05.364430 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2951186-428b-4b1b-9e83-2dfbcd906e82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0ffb465-9zhbg\" (UID: \"f2951186-428b-4b1b-9e83-2dfbcd906e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ffb465-9zhbg" Nov 03 03:01:05.364615 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:05.364494 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2951186-428b-4b1b-9e83-2dfbcd906e82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0ffb465-9zhbg\" (UID: \"f2951186-428b-4b1b-9e83-2dfbcd906e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ffb465-9zhbg" Nov 03 03:01:05.364615 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:05.364564 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dtwl\" (UniqueName: \"kubernetes.io/projected/f2951186-428b-4b1b-9e83-2dfbcd906e82-kube-api-access-4dtwl\") pod \"maas-api-revoke-sub-c0ffb465-9zhbg\" (UID: \"f2951186-428b-4b1b-9e83-2dfbcd906e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ffb465-9zhbg" Nov 03 03:01:05.465914 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:05.465854 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2951186-428b-4b1b-9e83-2dfbcd906e82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0ffb465-9zhbg\" (UID: \"f2951186-428b-4b1b-9e83-2dfbcd906e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ffb465-9zhbg" Nov 03 03:01:05.466119 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:05.465942 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2951186-428b-4b1b-9e83-2dfbcd906e82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0ffb465-9zhbg\" (UID: \"f2951186-428b-4b1b-9e83-2dfbcd906e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ffb465-9zhbg" Nov 03 03:01:05.466119 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:05.465964 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dtwl\" (UniqueName: \"kubernetes.io/projected/f2951186-428b-4b1b-9e83-2dfbcd906e82-kube-api-access-4dtwl\") pod \"maas-api-revoke-sub-c0ffb465-9zhbg\" (UID: \"f2951186-428b-4b1b-9e83-2dfbcd906e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ffb465-9zhbg" Nov 03 03:01:05.466778 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:05.466749 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2951186-428b-4b1b-9e83-2dfbcd906e82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0ffb465-9zhbg\" (UID: \"f2951186-428b-4b1b-9e83-2dfbcd906e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ffb465-9zhbg" Nov 03 03:01:05.468252 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:05.468234 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2951186-428b-4b1b-9e83-2dfbcd906e82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0ffb465-9zhbg\" (UID: \"f2951186-428b-4b1b-9e83-2dfbcd906e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ffb465-9zhbg" Nov 03 03:01:05.520522 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:05.520450 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dtwl\" (UniqueName: \"kubernetes.io/projected/f2951186-428b-4b1b-9e83-2dfbcd906e82-kube-api-access-4dtwl\") pod \"maas-api-revoke-sub-c0ffb465-9zhbg\" (UID: \"f2951186-428b-4b1b-9e83-2dfbcd906e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ffb465-9zhbg" Nov 03 03:01:05.553304 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:05.553276 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ffb465-9zhbg" Nov 03 03:01:05.740655 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:05.732834 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:01:05.925058 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:05.925022 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0ffb465-9zhbg"] Nov 03 03:01:05.931157 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:01:05.931127 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2951186_428b_4b1b_9e83_2dfbcd906e82.slice/crio-b1215c1eefb11cd35eed9c5b37d15f7007be9a01c4af761625021737ac3cded2 WatchSource:0}: Error finding container b1215c1eefb11cd35eed9c5b37d15f7007be9a01c4af761625021737ac3cded2: Status 404 returned error can't find the container with id b1215c1eefb11cd35eed9c5b37d15f7007be9a01c4af761625021737ac3cded2 Nov 03 03:01:06.348869 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:06.348830 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ffb465-9zhbg" event={"ID":"f2951186-428b-4b1b-9e83-2dfbcd906e82","Type":"ContainerStarted","Data":"a721dd2dd0ccf7b9b91eaad7ad2d74a3579f485e172e4015b80ad624d4ffc964"} Nov 03 03:01:06.349342 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:06.348882 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ffb465-9zhbg" event={"ID":"f2951186-428b-4b1b-9e83-2dfbcd906e82","Type":"ContainerStarted","Data":"b1215c1eefb11cd35eed9c5b37d15f7007be9a01c4af761625021737ac3cded2"} Nov 03 03:01:06.456807 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:06.456749 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ffb465-9zhbg" podStartSLOduration=1.456734447 podStartE2EDuration="1.456734447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:01:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:01:06.456186532 +0000 UTC m=+4143.715384685" watchObservedRunningTime="2026-11-03 03:01:06.456734447 +0000 UTC m=+4143.715932641" Nov 03 03:01:07.360083 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:07.360042 5605 generic.go:356] "Generic (PLEG): container finished" podID="f2951186-428b-4b1b-9e83-2dfbcd906e82" containerID="a721dd2dd0ccf7b9b91eaad7ad2d74a3579f485e172e4015b80ad624d4ffc964" exitCode=0 Nov 03 03:01:07.360694 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:07.360211 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ffb465-9zhbg" event={"ID":"f2951186-428b-4b1b-9e83-2dfbcd906e82","Type":"ContainerDied","Data":"a721dd2dd0ccf7b9b91eaad7ad2d74a3579f485e172e4015b80ad624d4ffc964"} Nov 03 03:01:09.300384 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:09.299707 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ffb465-9zhbg" Nov 03 03:01:09.404860 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:09.404766 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2951186-428b-4b1b-9e83-2dfbcd906e82-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2951186-428b-4b1b-9e83-2dfbcd906e82-maas-api-openshift-ca\") pod \"f2951186-428b-4b1b-9e83-2dfbcd906e82\" (UID: \"f2951186-428b-4b1b-9e83-2dfbcd906e82\") " Nov 03 03:01:09.405181 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:09.405161 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2951186-428b-4b1b-9e83-2dfbcd906e82-kube-api-access-4dtwl\" (UniqueName: \"kubernetes.io/projected/f2951186-428b-4b1b-9e83-2dfbcd906e82-kube-api-access-4dtwl\") pod \"f2951186-428b-4b1b-9e83-2dfbcd906e82\" (UID: \"f2951186-428b-4b1b-9e83-2dfbcd906e82\") " Nov 03 03:01:09.405439 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:09.405418 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2951186-428b-4b1b-9e83-2dfbcd906e82-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2951186-428b-4b1b-9e83-2dfbcd906e82-maas-api-xks-ca\") pod \"f2951186-428b-4b1b-9e83-2dfbcd906e82\" (UID: \"f2951186-428b-4b1b-9e83-2dfbcd906e82\") " Nov 03 03:01:09.408794 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:09.408769 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2951186-428b-4b1b-9e83-2dfbcd906e82-maas-api-openshift-ca" pod "f2951186-428b-4b1b-9e83-2dfbcd906e82" (UID: "f2951186-428b-4b1b-9e83-2dfbcd906e82"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:01:09.427994 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:09.427966 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2951186-428b-4b1b-9e83-2dfbcd906e82-maas-api-xks-ca" pod "f2951186-428b-4b1b-9e83-2dfbcd906e82" (UID: "f2951186-428b-4b1b-9e83-2dfbcd906e82"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:01:09.429838 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:09.429812 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ffb465-9zhbg" event={"ID":"f2951186-428b-4b1b-9e83-2dfbcd906e82","Type":"ContainerDied","Data":"b1215c1eefb11cd35eed9c5b37d15f7007be9a01c4af761625021737ac3cded2"} Nov 03 03:01:09.430008 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:09.429993 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1215c1eefb11cd35eed9c5b37d15f7007be9a01c4af761625021737ac3cded2" Nov 03 03:01:09.430173 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:09.430159 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ffb465-9zhbg" Nov 03 03:01:09.430521 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:09.430183 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2951186-428b-4b1b-9e83-2dfbcd906e82-kube-api-access-4dtwl" pod "f2951186-428b-4b1b-9e83-2dfbcd906e82" (UID: "f2951186-428b-4b1b-9e83-2dfbcd906e82"). InnerVolumeSpecName "kube-api-access-4dtwl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:01:09.549478 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:09.549432 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dtwl\" (UniqueName: \"kubernetes.io/projected/f2951186-428b-4b1b-9e83-2dfbcd906e82-kube-api-access-4dtwl\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:01:09.554391 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:09.549703 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2951186-428b-4b1b-9e83-2dfbcd906e82-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:01:09.554625 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:09.554603 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2951186-428b-4b1b-9e83-2dfbcd906e82-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:01:17.051190 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:17.051108 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk"] Nov 03 03:01:17.051737 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:17.051317 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk" podUID="c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4" containerName="istio-proxy" containerID="cri-o://b6753043ee27ccb709b8133d79d7736d3beaac9a305d6a08af2bb2db384619c5" gracePeriod=30 Nov 03 03:01:22.324204 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:22.324133 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk" Nov 03 03:01:22.404228 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:22.404187 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4-istiod-ca-cert\") pod \"c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4\" (UID: \"c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4\") " Nov 03 03:01:22.404459 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:22.404304 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4-istio-data\") pod \"c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4\" (UID: \"c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4\") " Nov 03 03:01:22.404459 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:22.404414 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4-workload-socket\") pod \"c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4\" (UID: \"c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4\") " Nov 03 03:01:22.404459 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:22.404435 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4-istio-token\" (UniqueName: \"kubernetes.io/projected/c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4-istio-token\") pod \"c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4\" (UID: \"c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4\") " Nov 03 03:01:22.404651 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:22.404463 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4-kube-api-access-5vld9\" (UniqueName: \"kubernetes.io/projected/c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4-kube-api-access-5vld9\") pod \"c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4\" (UID: \"c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4\") " Nov 03 03:01:22.404651 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:22.404491 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4-istio-podinfo\") pod \"c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4\" (UID: \"c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4\") " Nov 03 03:01:22.404651 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:22.404518 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4-workload-certs\") pod \"c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4\" (UID: \"c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4\") " Nov 03 03:01:22.404651 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:22.404580 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4-istio-data" pod "c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4" (UID: "c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:01:22.404651 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:22.404614 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4-istio-envoy\") pod \"c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4\" (UID: \"c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4\") " Nov 03 03:01:22.404937 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:22.404673 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4-credential-socket\") pod \"c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4\" (UID: \"c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4\") " Nov 03 03:01:22.404937 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:22.404726 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4-workload-socket" pod "c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4" (UID: "c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:01:22.404937 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:22.404771 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4-istiod-ca-cert" pod "c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4" (UID: "c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:01:22.404937 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:22.404818 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4-workload-certs" pod "c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4" (UID: "c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:01:22.405174 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:22.405145 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4-credential-socket" pod "c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4" (UID: "c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:01:22.405234 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:22.405216 5605 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4-istiod-ca-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:01:22.405291 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:22.405238 5605 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4-istio-data\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:01:22.405291 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:22.405253 5605 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4-workload-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:01:22.405291 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:22.405266 5605 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4-workload-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:01:22.406806 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:22.406780 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4-istio-token" pod "c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4" (UID: "c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:01:22.406806 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:22.406785 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4-istio-envoy" pod "c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4" (UID: "c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:01:22.406994 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:22.406815 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4-kube-api-access-5vld9" pod "c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4" (UID: "c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4"). InnerVolumeSpecName "kube-api-access-5vld9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:01:22.406994 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:22.406907 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4-istio-podinfo" pod "c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4" (UID: "c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:01:22.506200 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:22.506160 5605 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4-istio-token\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:01:22.506200 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:22.506198 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vld9\" (UniqueName: \"kubernetes.io/projected/c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4-kube-api-access-5vld9\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:01:22.506482 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:22.506213 5605 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4-istio-podinfo\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:01:22.506482 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:22.506228 5605 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4-istio-envoy\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:01:22.506482 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:22.506241 5605 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4-credential-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:01:22.507190 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:22.507166 5605 generic.go:356] "Generic (PLEG): container finished" podID="c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4" containerID="b6753043ee27ccb709b8133d79d7736d3beaac9a305d6a08af2bb2db384619c5" exitCode=0 Nov 03 03:01:22.507242 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:22.507223 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk" Nov 03 03:01:22.507281 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:22.507254 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk" event={"ID":"c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4","Type":"ContainerDied","Data":"b6753043ee27ccb709b8133d79d7736d3beaac9a305d6a08af2bb2db384619c5"} Nov 03 03:01:22.507332 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:22.507318 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk" event={"ID":"c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4","Type":"ContainerDied","Data":"11046157793fe08bc4dbf94ebac1d34ba1be9c82b5829e83d414b5912a280a16"} Nov 03 03:01:22.507387 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:22.507368 5605 scope.go:122] "RemoveContainer" containerID="b6753043ee27ccb709b8133d79d7736d3beaac9a305d6a08af2bb2db384619c5" Nov 03 03:01:22.517702 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:22.517680 5605 scope.go:122] "RemoveContainer" containerID="5e3619944c4f5eccfe047d840a9ab222a57143857bc0e30dce9dbb5339137dee" Nov 03 03:01:22.529212 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:22.529190 5605 scope.go:122] "RemoveContainer" containerID="b6753043ee27ccb709b8133d79d7736d3beaac9a305d6a08af2bb2db384619c5" Nov 03 03:01:22.529525 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:01:22.529508 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6753043ee27ccb709b8133d79d7736d3beaac9a305d6a08af2bb2db384619c5\": container with ID starting with b6753043ee27ccb709b8133d79d7736d3beaac9a305d6a08af2bb2db384619c5 not found: ID does not exist" containerID="b6753043ee27ccb709b8133d79d7736d3beaac9a305d6a08af2bb2db384619c5" Nov 03 03:01:22.529577 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:22.529535 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6753043ee27ccb709b8133d79d7736d3beaac9a305d6a08af2bb2db384619c5"} err="failed to get container status \"b6753043ee27ccb709b8133d79d7736d3beaac9a305d6a08af2bb2db384619c5\": rpc error: code = NotFound desc = could not find container \"b6753043ee27ccb709b8133d79d7736d3beaac9a305d6a08af2bb2db384619c5\": container with ID starting with b6753043ee27ccb709b8133d79d7736d3beaac9a305d6a08af2bb2db384619c5 not found: ID does not exist" Nov 03 03:01:22.529577 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:22.529552 5605 scope.go:122] "RemoveContainer" containerID="5e3619944c4f5eccfe047d840a9ab222a57143857bc0e30dce9dbb5339137dee" Nov 03 03:01:22.529754 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:01:22.529740 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e3619944c4f5eccfe047d840a9ab222a57143857bc0e30dce9dbb5339137dee\": container with ID starting with 5e3619944c4f5eccfe047d840a9ab222a57143857bc0e30dce9dbb5339137dee not found: ID does not exist" containerID="5e3619944c4f5eccfe047d840a9ab222a57143857bc0e30dce9dbb5339137dee" Nov 03 03:01:22.529790 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:22.529757 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e3619944c4f5eccfe047d840a9ab222a57143857bc0e30dce9dbb5339137dee"} err="failed to get container status \"5e3619944c4f5eccfe047d840a9ab222a57143857bc0e30dce9dbb5339137dee\": rpc error: code = NotFound desc = could not find container \"5e3619944c4f5eccfe047d840a9ab222a57143857bc0e30dce9dbb5339137dee\": container with ID starting with 5e3619944c4f5eccfe047d840a9ab222a57143857bc0e30dce9dbb5339137dee not found: ID does not exist" Nov 03 03:01:22.664614 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:22.664522 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk"] Nov 03 03:01:22.677852 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:22.672155 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-ddcbd0-openshift-default-85d8f6fcf9-ppxsk"] Nov 03 03:01:22.849915 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:22.849872 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4" path="/var/lib/kubelet/pods/c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4/volumes" Nov 03 03:01:23.919244 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:01:23.919202 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:01:37.413711 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:37.413675 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-f735f4/rate-test-model-f735f4-kserve-6c67f66677-2sqk5"] Nov 03 03:01:37.414136 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:37.414103 5605 cpu_manager.go:414] "containerMap: removing container" podUID="c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4" containerName="istio-proxy" Nov 03 03:01:37.414136 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:37.414114 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4" containerName="istio-proxy" Nov 03 03:01:37.414233 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:37.414142 5605 cpu_manager.go:414] "containerMap: removing container" podUID="f2951186-428b-4b1b-9e83-2dfbcd906e82" containerName="revoke-keys" Nov 03 03:01:37.414233 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:37.414147 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2951186-428b-4b1b-9e83-2dfbcd906e82" containerName="revoke-keys" Nov 03 03:01:37.414313 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:37.414244 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4" containerName="istio-proxy" Nov 03 03:01:37.414313 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:37.414255 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2951186-428b-4b1b-9e83-2dfbcd906e82" containerName="revoke-keys" Nov 03 03:01:37.419000 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:37.418976 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-f735f4/rate-test-model-f735f4-kserve-6c67f66677-2sqk5" Nov 03 03:01:37.490696 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:37.490666 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e02ba51-abe8-472f-84f8-efa53f1d3b40-model-cache\") pod \"rate-test-model-f735f4-kserve-6c67f66677-2sqk5\" (UID: \"2e02ba51-abe8-472f-84f8-efa53f1d3b40\") " pod="ai-tenant-e2e-shared-a-w2-f735f4/rate-test-model-f735f4-kserve-6c67f66677-2sqk5" Nov 03 03:01:37.490855 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:37.490723 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e02ba51-abe8-472f-84f8-efa53f1d3b40-dshm\") pod \"rate-test-model-f735f4-kserve-6c67f66677-2sqk5\" (UID: \"2e02ba51-abe8-472f-84f8-efa53f1d3b40\") " pod="ai-tenant-e2e-shared-a-w2-f735f4/rate-test-model-f735f4-kserve-6c67f66677-2sqk5" Nov 03 03:01:37.490902 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:37.490851 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rwlw\" (UniqueName: \"kubernetes.io/projected/2e02ba51-abe8-472f-84f8-efa53f1d3b40-kube-api-access-4rwlw\") pod \"rate-test-model-f735f4-kserve-6c67f66677-2sqk5\" (UID: \"2e02ba51-abe8-472f-84f8-efa53f1d3b40\") " pod="ai-tenant-e2e-shared-a-w2-f735f4/rate-test-model-f735f4-kserve-6c67f66677-2sqk5" Nov 03 03:01:37.490902 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:37.490880 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e02ba51-abe8-472f-84f8-efa53f1d3b40-home\") pod \"rate-test-model-f735f4-kserve-6c67f66677-2sqk5\" (UID: \"2e02ba51-abe8-472f-84f8-efa53f1d3b40\") " pod="ai-tenant-e2e-shared-a-w2-f735f4/rate-test-model-f735f4-kserve-6c67f66677-2sqk5" Nov 03 03:01:37.490998 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:37.490978 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e02ba51-abe8-472f-84f8-efa53f1d3b40-tls-certs\") pod \"rate-test-model-f735f4-kserve-6c67f66677-2sqk5\" (UID: \"2e02ba51-abe8-472f-84f8-efa53f1d3b40\") " pod="ai-tenant-e2e-shared-a-w2-f735f4/rate-test-model-f735f4-kserve-6c67f66677-2sqk5" Nov 03 03:01:37.491109 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:37.491092 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e02ba51-abe8-472f-84f8-efa53f1d3b40-tmp-dir\") pod \"rate-test-model-f735f4-kserve-6c67f66677-2sqk5\" (UID: \"2e02ba51-abe8-472f-84f8-efa53f1d3b40\") " pod="ai-tenant-e2e-shared-a-w2-f735f4/rate-test-model-f735f4-kserve-6c67f66677-2sqk5" Nov 03 03:01:37.515774 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:37.515750 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-f735f4\"/\"openshift-service-ca.crt\"" Nov 03 03:01:37.515928 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:37.515797 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-f735f4\"/\"kube-root-ca.crt\"" Nov 03 03:01:37.515928 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:37.515887 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-f735f4\"/\"rate-test-model-f735f4-kserve-self-signed-certs\"" Nov 03 03:01:37.515928 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:37.515900 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-f735f4\"/\"default-dockercfg-n7nmc\"" Nov 03 03:01:37.518145 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:37.518126 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-f735f4/rate-test-model-f735f4-kserve-6c67f66677-2sqk5"] Nov 03 03:01:37.592771 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:37.592740 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e02ba51-abe8-472f-84f8-efa53f1d3b40-tmp-dir\") pod \"rate-test-model-f735f4-kserve-6c67f66677-2sqk5\" (UID: \"2e02ba51-abe8-472f-84f8-efa53f1d3b40\") " pod="ai-tenant-e2e-shared-a-w2-f735f4/rate-test-model-f735f4-kserve-6c67f66677-2sqk5" Nov 03 03:01:37.592989 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:37.592793 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e02ba51-abe8-472f-84f8-efa53f1d3b40-model-cache\") pod \"rate-test-model-f735f4-kserve-6c67f66677-2sqk5\" (UID: \"2e02ba51-abe8-472f-84f8-efa53f1d3b40\") " pod="ai-tenant-e2e-shared-a-w2-f735f4/rate-test-model-f735f4-kserve-6c67f66677-2sqk5" Nov 03 03:01:37.592989 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:37.592825 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e02ba51-abe8-472f-84f8-efa53f1d3b40-dshm\") pod \"rate-test-model-f735f4-kserve-6c67f66677-2sqk5\" (UID: \"2e02ba51-abe8-472f-84f8-efa53f1d3b40\") " pod="ai-tenant-e2e-shared-a-w2-f735f4/rate-test-model-f735f4-kserve-6c67f66677-2sqk5" Nov 03 03:01:37.592989 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:37.592859 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rwlw\" (UniqueName: \"kubernetes.io/projected/2e02ba51-abe8-472f-84f8-efa53f1d3b40-kube-api-access-4rwlw\") pod \"rate-test-model-f735f4-kserve-6c67f66677-2sqk5\" (UID: \"2e02ba51-abe8-472f-84f8-efa53f1d3b40\") " pod="ai-tenant-e2e-shared-a-w2-f735f4/rate-test-model-f735f4-kserve-6c67f66677-2sqk5" Nov 03 03:01:37.592989 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:37.592978 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e02ba51-abe8-472f-84f8-efa53f1d3b40-home\") pod \"rate-test-model-f735f4-kserve-6c67f66677-2sqk5\" (UID: \"2e02ba51-abe8-472f-84f8-efa53f1d3b40\") " pod="ai-tenant-e2e-shared-a-w2-f735f4/rate-test-model-f735f4-kserve-6c67f66677-2sqk5" Nov 03 03:01:37.593241 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:37.593055 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e02ba51-abe8-472f-84f8-efa53f1d3b40-tls-certs\") pod \"rate-test-model-f735f4-kserve-6c67f66677-2sqk5\" (UID: \"2e02ba51-abe8-472f-84f8-efa53f1d3b40\") " pod="ai-tenant-e2e-shared-a-w2-f735f4/rate-test-model-f735f4-kserve-6c67f66677-2sqk5" Nov 03 03:01:37.593241 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:37.593209 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e02ba51-abe8-472f-84f8-efa53f1d3b40-tmp-dir\") pod \"rate-test-model-f735f4-kserve-6c67f66677-2sqk5\" (UID: \"2e02ba51-abe8-472f-84f8-efa53f1d3b40\") " pod="ai-tenant-e2e-shared-a-w2-f735f4/rate-test-model-f735f4-kserve-6c67f66677-2sqk5" Nov 03 03:01:37.593392 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:37.593250 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e02ba51-abe8-472f-84f8-efa53f1d3b40-model-cache\") pod \"rate-test-model-f735f4-kserve-6c67f66677-2sqk5\" (UID: \"2e02ba51-abe8-472f-84f8-efa53f1d3b40\") " pod="ai-tenant-e2e-shared-a-w2-f735f4/rate-test-model-f735f4-kserve-6c67f66677-2sqk5" Nov 03 03:01:37.593392 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:37.593319 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e02ba51-abe8-472f-84f8-efa53f1d3b40-home\") pod \"rate-test-model-f735f4-kserve-6c67f66677-2sqk5\" (UID: \"2e02ba51-abe8-472f-84f8-efa53f1d3b40\") " pod="ai-tenant-e2e-shared-a-w2-f735f4/rate-test-model-f735f4-kserve-6c67f66677-2sqk5" Nov 03 03:01:37.595139 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:37.595112 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e02ba51-abe8-472f-84f8-efa53f1d3b40-dshm\") pod \"rate-test-model-f735f4-kserve-6c67f66677-2sqk5\" (UID: \"2e02ba51-abe8-472f-84f8-efa53f1d3b40\") " pod="ai-tenant-e2e-shared-a-w2-f735f4/rate-test-model-f735f4-kserve-6c67f66677-2sqk5" Nov 03 03:01:37.595551 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:37.595533 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e02ba51-abe8-472f-84f8-efa53f1d3b40-tls-certs\") pod \"rate-test-model-f735f4-kserve-6c67f66677-2sqk5\" (UID: \"2e02ba51-abe8-472f-84f8-efa53f1d3b40\") " pod="ai-tenant-e2e-shared-a-w2-f735f4/rate-test-model-f735f4-kserve-6c67f66677-2sqk5" Nov 03 03:01:37.646411 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:37.646387 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rwlw\" (UniqueName: \"kubernetes.io/projected/2e02ba51-abe8-472f-84f8-efa53f1d3b40-kube-api-access-4rwlw\") pod \"rate-test-model-f735f4-kserve-6c67f66677-2sqk5\" (UID: \"2e02ba51-abe8-472f-84f8-efa53f1d3b40\") " pod="ai-tenant-e2e-shared-a-w2-f735f4/rate-test-model-f735f4-kserve-6c67f66677-2sqk5" Nov 03 03:01:37.731300 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:37.731223 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-f735f4/rate-test-model-f735f4-kserve-6c67f66677-2sqk5" Nov 03 03:01:38.108462 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:38.108429 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-f735f4/rate-test-model-f735f4-kserve-6c67f66677-2sqk5"] Nov 03 03:01:38.114786 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:01:38.114755 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e02ba51_abe8_472f_84f8_efa53f1d3b40.slice/crio-0e2ebbdb5289cab555571a23513f244a667cffa4bdd82d6ee1af0d53b4fe19c9 WatchSource:0}: Error finding container 0e2ebbdb5289cab555571a23513f244a667cffa4bdd82d6ee1af0d53b4fe19c9: Status 404 returned error can't find the container with id 0e2ebbdb5289cab555571a23513f244a667cffa4bdd82d6ee1af0d53b4fe19c9 Nov 03 03:01:38.598877 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:38.598838 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-f735f4/rate-test-model-f735f4-kserve-6c67f66677-2sqk5" event={"ID":"2e02ba51-abe8-472f-84f8-efa53f1d3b40","Type":"ContainerStarted","Data":"47e3e0ffca307aaaf59433e70b6ccb1249f01d46320332d9c697acc3c6c944c0"} Nov 03 03:01:38.599317 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:38.598888 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-f735f4/rate-test-model-f735f4-kserve-6c67f66677-2sqk5" event={"ID":"2e02ba51-abe8-472f-84f8-efa53f1d3b40","Type":"ContainerStarted","Data":"0e2ebbdb5289cab555571a23513f244a667cffa4bdd82d6ee1af0d53b4fe19c9"} Nov 03 03:01:38.708008 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:38.707957 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-f735f4/rate-test-model-f735f4-kserve-6c67f66677-2sqk5" podStartSLOduration=1.707944925 podStartE2EDuration="1.707944925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:01:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:01:38.706162869 +0000 UTC m=+4175.965361024" watchObservedRunningTime="2026-11-03 03:01:38.707944925 +0000 UTC m=+4175.967143080" Nov 03 03:01:38.731756 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:38.731718 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-f735f4/rate-test-model-f735f4-kserve-6c67f66677-2sqk5" Nov 03 03:01:47.731943 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:47.731910 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-f735f4/rate-test-model-f735f4-kserve-6c67f66677-2sqk5" Nov 03 03:01:47.745042 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:47.745015 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-f735f4/rate-test-model-f735f4-kserve-6c67f66677-2sqk5" Nov 03 03:01:48.676388 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:48.676325 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-f735f4/rate-test-model-f735f4-kserve-6c67f66677-2sqk5" Nov 03 03:01:50.592499 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:50.592446 5605 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a0e72e/distinct-2-w3-a0e72e-kserve-59f58c59bf-jz5p8" podUID="70bd98e0-d8ac-47c2-901e-83f301d59a57" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:01:50.763491 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:50.763443 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a0e72e/premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv" podUID="a9b9af44-ab6d-4caf-89af-c986ea25b4c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:01:50.808558 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:50.808506 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-521f1f/simulator-w2-521f1f-kserve-5f47b86f6-mgksf" podUID="6e1b7fda-0ade-4042-ad63-5115bc66b454" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:01:50.915538 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:50.914764 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a0e72e/simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq" podUID="1901f57b-279b-4cbf-bf0c-054abc5786d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Nov 03 03:01:51.052382 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:01:51.051556 5605 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1bccbd/distinct-w1-1bccbd-kserve-5dfc85555f-tkvg6" podUID="dd4a9d44-c63e-4f67-bcac-a4f98ecf1796" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 03:01:53.914280 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:01:53.914243 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:02:05.046545 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:05.046513 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2"] Nov 03 03:02:05.047091 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:05.047073 5605 cpu_manager.go:414] "containerMap: removing container" podUID="c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4" containerName="istio-proxy" Nov 03 03:02:05.047169 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:05.047093 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4" containerName="istio-proxy" Nov 03 03:02:05.047233 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:05.047207 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5f75ae2-27a2-4f9c-a3a4-feddd4fec3d4" containerName="istio-proxy" Nov 03 03:02:05.050738 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:05.050716 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2" Nov 03 03:02:05.104763 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:05.104738 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-f735f4-openshift-default-dockercfg-4w55v\"" Nov 03 03:02:05.173316 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:05.167217 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccc49b02-1ca4-421c-8f34-320fbeb0e0ac-istiod-ca-cert\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2\" (UID: \"ccc49b02-1ca4-421c-8f34-320fbeb0e0ac\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2" Nov 03 03:02:05.173316 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:05.167319 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc49b02-1ca4-421c-8f34-320fbeb0e0ac-credential-socket\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2\" (UID: \"ccc49b02-1ca4-421c-8f34-320fbeb0e0ac\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2" Nov 03 03:02:05.173316 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:05.167397 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccc49b02-1ca4-421c-8f34-320fbeb0e0ac-workload-certs\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2\" (UID: \"ccc49b02-1ca4-421c-8f34-320fbeb0e0ac\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2" Nov 03 03:02:05.173316 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:05.167420 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpz68\" (UniqueName: \"kubernetes.io/projected/ccc49b02-1ca4-421c-8f34-320fbeb0e0ac-kube-api-access-lpz68\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2\" (UID: \"ccc49b02-1ca4-421c-8f34-320fbeb0e0ac\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2" Nov 03 03:02:05.173316 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:05.167465 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccc49b02-1ca4-421c-8f34-320fbeb0e0ac-istio-token\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2\" (UID: \"ccc49b02-1ca4-421c-8f34-320fbeb0e0ac\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2" Nov 03 03:02:05.173316 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:05.167529 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccc49b02-1ca4-421c-8f34-320fbeb0e0ac-istio-podinfo\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2\" (UID: \"ccc49b02-1ca4-421c-8f34-320fbeb0e0ac\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2" Nov 03 03:02:05.173316 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:05.167572 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc49b02-1ca4-421c-8f34-320fbeb0e0ac-workload-socket\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2\" (UID: \"ccc49b02-1ca4-421c-8f34-320fbeb0e0ac\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2" Nov 03 03:02:05.173316 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:05.167610 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccc49b02-1ca4-421c-8f34-320fbeb0e0ac-istio-envoy\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2\" (UID: \"ccc49b02-1ca4-421c-8f34-320fbeb0e0ac\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2" Nov 03 03:02:05.173316 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:05.167670 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccc49b02-1ca4-421c-8f34-320fbeb0e0ac-istio-data\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2\" (UID: \"ccc49b02-1ca4-421c-8f34-320fbeb0e0ac\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2" Nov 03 03:02:05.173800 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:05.173442 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2"] Nov 03 03:02:05.269080 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:05.269042 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccc49b02-1ca4-421c-8f34-320fbeb0e0ac-istio-podinfo\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2\" (UID: \"ccc49b02-1ca4-421c-8f34-320fbeb0e0ac\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2" Nov 03 03:02:05.269269 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:05.269090 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc49b02-1ca4-421c-8f34-320fbeb0e0ac-workload-socket\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2\" (UID: \"ccc49b02-1ca4-421c-8f34-320fbeb0e0ac\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2" Nov 03 03:02:05.269269 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:05.269111 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccc49b02-1ca4-421c-8f34-320fbeb0e0ac-istio-envoy\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2\" (UID: \"ccc49b02-1ca4-421c-8f34-320fbeb0e0ac\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2" Nov 03 03:02:05.269269 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:05.269146 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccc49b02-1ca4-421c-8f34-320fbeb0e0ac-istio-data\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2\" (UID: \"ccc49b02-1ca4-421c-8f34-320fbeb0e0ac\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2" Nov 03 03:02:05.269269 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:05.269187 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccc49b02-1ca4-421c-8f34-320fbeb0e0ac-istiod-ca-cert\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2\" (UID: \"ccc49b02-1ca4-421c-8f34-320fbeb0e0ac\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2" Nov 03 03:02:05.269269 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:05.269252 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc49b02-1ca4-421c-8f34-320fbeb0e0ac-credential-socket\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2\" (UID: \"ccc49b02-1ca4-421c-8f34-320fbeb0e0ac\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2" Nov 03 03:02:05.269598 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:05.269306 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccc49b02-1ca4-421c-8f34-320fbeb0e0ac-workload-certs\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2\" (UID: \"ccc49b02-1ca4-421c-8f34-320fbeb0e0ac\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2" Nov 03 03:02:05.269598 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:05.269432 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpz68\" (UniqueName: \"kubernetes.io/projected/ccc49b02-1ca4-421c-8f34-320fbeb0e0ac-kube-api-access-lpz68\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2\" (UID: \"ccc49b02-1ca4-421c-8f34-320fbeb0e0ac\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2" Nov 03 03:02:05.269598 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:05.269504 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccc49b02-1ca4-421c-8f34-320fbeb0e0ac-istio-token\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2\" (UID: \"ccc49b02-1ca4-421c-8f34-320fbeb0e0ac\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2" Nov 03 03:02:05.269598 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:05.269530 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc49b02-1ca4-421c-8f34-320fbeb0e0ac-workload-socket\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2\" (UID: \"ccc49b02-1ca4-421c-8f34-320fbeb0e0ac\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2" Nov 03 03:02:05.269598 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:05.269577 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccc49b02-1ca4-421c-8f34-320fbeb0e0ac-istio-data\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2\" (UID: \"ccc49b02-1ca4-421c-8f34-320fbeb0e0ac\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2" Nov 03 03:02:05.269885 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:05.269809 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccc49b02-1ca4-421c-8f34-320fbeb0e0ac-workload-certs\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2\" (UID: \"ccc49b02-1ca4-421c-8f34-320fbeb0e0ac\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2" Nov 03 03:02:05.269885 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:05.269870 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc49b02-1ca4-421c-8f34-320fbeb0e0ac-credential-socket\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2\" (UID: \"ccc49b02-1ca4-421c-8f34-320fbeb0e0ac\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2" Nov 03 03:02:05.270045 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:05.270025 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccc49b02-1ca4-421c-8f34-320fbeb0e0ac-istiod-ca-cert\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2\" (UID: \"ccc49b02-1ca4-421c-8f34-320fbeb0e0ac\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2" Nov 03 03:02:05.271609 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:05.271583 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccc49b02-1ca4-421c-8f34-320fbeb0e0ac-istio-envoy\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2\" (UID: \"ccc49b02-1ca4-421c-8f34-320fbeb0e0ac\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2" Nov 03 03:02:05.271896 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:05.271873 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccc49b02-1ca4-421c-8f34-320fbeb0e0ac-istio-podinfo\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2\" (UID: \"ccc49b02-1ca4-421c-8f34-320fbeb0e0ac\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2" Nov 03 03:02:05.323431 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:05.323390 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccc49b02-1ca4-421c-8f34-320fbeb0e0ac-istio-token\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2\" (UID: \"ccc49b02-1ca4-421c-8f34-320fbeb0e0ac\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2" Nov 03 03:02:05.323603 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:05.323387 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpz68\" (UniqueName: \"kubernetes.io/projected/ccc49b02-1ca4-421c-8f34-320fbeb0e0ac-kube-api-access-lpz68\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2\" (UID: \"ccc49b02-1ca4-421c-8f34-320fbeb0e0ac\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2" Nov 03 03:02:05.364731 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:05.364699 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2" Nov 03 03:02:05.786231 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:05.786152 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2"] Nov 03 03:02:05.791672 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:02:05.791643 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccc49b02_1ca4_421c_8f34_320fbeb0e0ac.slice/crio-92a4a59c3d3b7e252db55b69f6b4056db9b068d6bcaef7cd0559a0ffd844f709 WatchSource:0}: Error finding container 92a4a59c3d3b7e252db55b69f6b4056db9b068d6bcaef7cd0559a0ffd844f709: Status 404 returned error can't find the container with id 92a4a59c3d3b7e252db55b69f6b4056db9b068d6bcaef7cd0559a0ffd844f709 Nov 03 03:02:05.794021 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:05.793986 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 03:02:05.794117 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:05.794052 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 03:02:05.794117 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:05.794083 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 03:02:06.784449 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:06.784411 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2" event={"ID":"ccc49b02-1ca4-421c-8f34-320fbeb0e0ac","Type":"ContainerStarted","Data":"1d6ad52f6912003baf1ae6708aa92db4a64e74ba3548d859b3c990514dfb2c7e"} Nov 03 03:02:06.784969 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:06.784489 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2" event={"ID":"ccc49b02-1ca4-421c-8f34-320fbeb0e0ac","Type":"ContainerStarted","Data":"92a4a59c3d3b7e252db55b69f6b4056db9b068d6bcaef7cd0559a0ffd844f709"} Nov 03 03:02:06.949075 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:06.949010 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2" podStartSLOduration=1.948995908 podStartE2EDuration="1.948995908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:02:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:02:06.90137509 +0000 UTC m=+4204.160573241" watchObservedRunningTime="2026-11-03 03:02:06.948995908 +0000 UTC m=+4204.208194061" Nov 03 03:02:07.374383 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:07.371507 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2" Nov 03 03:02:07.374383 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:07.373670 5605 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" start-of-body= Nov 03 03:02:07.374383 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:07.373722 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2" podUID="ccc49b02-1ca4-421c-8f34-320fbeb0e0ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" Nov 03 03:02:08.369674 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:08.369635 5605 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" start-of-body= Nov 03 03:02:08.370295 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:08.370261 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2" podUID="ccc49b02-1ca4-421c-8f34-320fbeb0e0ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" Nov 03 03:02:09.379723 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:09.379686 5605 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" start-of-body= Nov 03 03:02:09.380161 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:09.379748 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2" podUID="ccc49b02-1ca4-421c-8f34-320fbeb0e0ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" Nov 03 03:02:10.374383 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:10.367327 5605 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" start-of-body= Nov 03 03:02:10.374383 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:10.367424 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2" podUID="ccc49b02-1ca4-421c-8f34-320fbeb0e0ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" Nov 03 03:02:11.366828 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:11.366785 5605 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" start-of-body= Nov 03 03:02:11.367469 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:11.367432 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2" podUID="ccc49b02-1ca4-421c-8f34-320fbeb0e0ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" Nov 03 03:02:12.369753 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:12.369723 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2" Nov 03 03:02:12.370168 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:12.369962 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2" Nov 03 03:02:12.370577 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:12.370561 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2" Nov 03 03:02:17.129165 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:17.129132 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:02:23.919398 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:02:23.919343 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:02:28.549421 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:28.549383 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4477d0a8-gp62c"] Nov 03 03:02:28.553205 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:28.553188 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4477d0a8-gp62c" Nov 03 03:02:28.651549 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:28.651506 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:02:28.651549 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:28.651528 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:02:28.651549 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:28.651519 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:02:28.651795 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:28.651575 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 03:02:28.653872 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:28.653850 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4477d0a8-gp62c"] Nov 03 03:02:28.736652 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:28.736610 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/204dc46a-828c-42e4-bf24-36d54a730ff0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4477d0a8-gp62c\" (UID: \"204dc46a-828c-42e4-bf24-36d54a730ff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4477d0a8-gp62c" Nov 03 03:02:28.736831 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:28.736753 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/204dc46a-828c-42e4-bf24-36d54a730ff0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4477d0a8-gp62c\" (UID: \"204dc46a-828c-42e4-bf24-36d54a730ff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4477d0a8-gp62c" Nov 03 03:02:28.736875 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:28.736844 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns7tn\" (UniqueName: \"kubernetes.io/projected/204dc46a-828c-42e4-bf24-36d54a730ff0-kube-api-access-ns7tn\") pod \"maas-api-revoke-sub-4477d0a8-gp62c\" (UID: \"204dc46a-828c-42e4-bf24-36d54a730ff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4477d0a8-gp62c" Nov 03 03:02:28.838508 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:28.838393 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/204dc46a-828c-42e4-bf24-36d54a730ff0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4477d0a8-gp62c\" (UID: \"204dc46a-828c-42e4-bf24-36d54a730ff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4477d0a8-gp62c" Nov 03 03:02:28.838765 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:28.838563 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns7tn\" (UniqueName: \"kubernetes.io/projected/204dc46a-828c-42e4-bf24-36d54a730ff0-kube-api-access-ns7tn\") pod \"maas-api-revoke-sub-4477d0a8-gp62c\" (UID: \"204dc46a-828c-42e4-bf24-36d54a730ff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4477d0a8-gp62c" Nov 03 03:02:28.838765 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:28.838645 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/204dc46a-828c-42e4-bf24-36d54a730ff0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4477d0a8-gp62c\" (UID: \"204dc46a-828c-42e4-bf24-36d54a730ff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4477d0a8-gp62c" Nov 03 03:02:28.839670 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:28.839642 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/204dc46a-828c-42e4-bf24-36d54a730ff0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4477d0a8-gp62c\" (UID: \"204dc46a-828c-42e4-bf24-36d54a730ff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4477d0a8-gp62c" Nov 03 03:02:28.843621 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:28.843591 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/204dc46a-828c-42e4-bf24-36d54a730ff0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4477d0a8-gp62c\" (UID: \"204dc46a-828c-42e4-bf24-36d54a730ff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4477d0a8-gp62c" Nov 03 03:02:28.893654 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:28.893623 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns7tn\" (UniqueName: \"kubernetes.io/projected/204dc46a-828c-42e4-bf24-36d54a730ff0-kube-api-access-ns7tn\") pod \"maas-api-revoke-sub-4477d0a8-gp62c\" (UID: \"204dc46a-828c-42e4-bf24-36d54a730ff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4477d0a8-gp62c" Nov 03 03:02:29.166914 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:29.166828 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4477d0a8-gp62c" Nov 03 03:02:29.534202 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:29.534130 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4477d0a8-gp62c"] Nov 03 03:02:29.539551 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:02:29.539518 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod204dc46a_828c_42e4_bf24_36d54a730ff0.slice/crio-ac3071ff05a653e4c301f26b64121d79a1892589ef5cde9d144fed228d8b1d54 WatchSource:0}: Error finding container ac3071ff05a653e4c301f26b64121d79a1892589ef5cde9d144fed228d8b1d54: Status 404 returned error can't find the container with id ac3071ff05a653e4c301f26b64121d79a1892589ef5cde9d144fed228d8b1d54 Nov 03 03:02:29.909854 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:29.909819 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4477d0a8-gp62c" event={"ID":"204dc46a-828c-42e4-bf24-36d54a730ff0","Type":"ContainerStarted","Data":"9ff9a7045e7e3c8d4413aa8c6c61f569712c22461d6a335418441655a0276a9c"} Nov 03 03:02:29.910284 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:29.909885 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4477d0a8-gp62c" event={"ID":"204dc46a-828c-42e4-bf24-36d54a730ff0","Type":"ContainerStarted","Data":"ac3071ff05a653e4c301f26b64121d79a1892589ef5cde9d144fed228d8b1d54"} Nov 03 03:02:30.017466 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:30.017425 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4477d0a8-gp62c" podStartSLOduration=2.017412834 podStartE2EDuration="2.017412834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:02:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:02:30.016712542 +0000 UTC m=+4227.275910695" watchObservedRunningTime="2026-11-03 03:02:30.017412834 +0000 UTC m=+4227.276611261" Nov 03 03:02:30.916778 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:30.916750 5605 generic.go:356] "Generic (PLEG): container finished" podID="204dc46a-828c-42e4-bf24-36d54a730ff0" containerID="9ff9a7045e7e3c8d4413aa8c6c61f569712c22461d6a335418441655a0276a9c" exitCode=0 Nov 03 03:02:30.917190 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:30.916835 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4477d0a8-gp62c" event={"ID":"204dc46a-828c-42e4-bf24-36d54a730ff0","Type":"ContainerDied","Data":"9ff9a7045e7e3c8d4413aa8c6c61f569712c22461d6a335418441655a0276a9c"} Nov 03 03:02:32.062644 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:32.062622 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4477d0a8-gp62c" Nov 03 03:02:32.067450 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:32.067428 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/204dc46a-828c-42e4-bf24-36d54a730ff0-kube-api-access-ns7tn\" (UniqueName: \"kubernetes.io/projected/204dc46a-828c-42e4-bf24-36d54a730ff0-kube-api-access-ns7tn\") pod \"204dc46a-828c-42e4-bf24-36d54a730ff0\" (UID: \"204dc46a-828c-42e4-bf24-36d54a730ff0\") " Nov 03 03:02:32.067584 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:32.067544 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/204dc46a-828c-42e4-bf24-36d54a730ff0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/204dc46a-828c-42e4-bf24-36d54a730ff0-maas-api-openshift-ca\") pod \"204dc46a-828c-42e4-bf24-36d54a730ff0\" (UID: \"204dc46a-828c-42e4-bf24-36d54a730ff0\") " Nov 03 03:02:32.067659 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:32.067645 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/204dc46a-828c-42e4-bf24-36d54a730ff0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/204dc46a-828c-42e4-bf24-36d54a730ff0-maas-api-xks-ca\") pod \"204dc46a-828c-42e4-bf24-36d54a730ff0\" (UID: \"204dc46a-828c-42e4-bf24-36d54a730ff0\") " Nov 03 03:02:32.067875 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:32.067849 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/204dc46a-828c-42e4-bf24-36d54a730ff0-maas-api-openshift-ca" pod "204dc46a-828c-42e4-bf24-36d54a730ff0" (UID: "204dc46a-828c-42e4-bf24-36d54a730ff0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:02:32.067981 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:32.067924 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/204dc46a-828c-42e4-bf24-36d54a730ff0-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:02:32.069522 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:32.069497 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/204dc46a-828c-42e4-bf24-36d54a730ff0-kube-api-access-ns7tn" pod "204dc46a-828c-42e4-bf24-36d54a730ff0" (UID: "204dc46a-828c-42e4-bf24-36d54a730ff0"). InnerVolumeSpecName "kube-api-access-ns7tn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:02:32.069522 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:32.069511 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/204dc46a-828c-42e4-bf24-36d54a730ff0-maas-api-xks-ca" pod "204dc46a-828c-42e4-bf24-36d54a730ff0" (UID: "204dc46a-828c-42e4-bf24-36d54a730ff0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:02:32.169261 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:32.169231 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/204dc46a-828c-42e4-bf24-36d54a730ff0-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:02:32.169261 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:32.169264 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ns7tn\" (UniqueName: \"kubernetes.io/projected/204dc46a-828c-42e4-bf24-36d54a730ff0-kube-api-access-ns7tn\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:02:32.930154 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:32.930130 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4477d0a8-gp62c" Nov 03 03:02:32.930311 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:32.930128 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4477d0a8-gp62c" event={"ID":"204dc46a-828c-42e4-bf24-36d54a730ff0","Type":"ContainerDied","Data":"ac3071ff05a653e4c301f26b64121d79a1892589ef5cde9d144fed228d8b1d54"} Nov 03 03:02:32.930311 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:32.930261 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac3071ff05a653e4c301f26b64121d79a1892589ef5cde9d144fed228d8b1d54" Nov 03 03:02:49.788684 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:49.788609 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-957b33/rate-test-model-957b33-kserve-7bf748c8f-jjx87"] Nov 03 03:02:49.789176 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:49.789159 5605 cpu_manager.go:414] "containerMap: removing container" podUID="204dc46a-828c-42e4-bf24-36d54a730ff0" containerName="revoke-keys" Nov 03 03:02:49.789222 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:49.789178 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="204dc46a-828c-42e4-bf24-36d54a730ff0" containerName="revoke-keys" Nov 03 03:02:49.789270 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:49.789261 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="204dc46a-828c-42e4-bf24-36d54a730ff0" containerName="revoke-keys" Nov 03 03:02:49.792575 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:49.792555 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-957b33/rate-test-model-957b33-kserve-7bf748c8f-jjx87" Nov 03 03:02:49.847606 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:49.847566 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58aaea68-aa14-4041-a2ee-c8dbb1c19b55-dshm\") pod \"rate-test-model-957b33-kserve-7bf748c8f-jjx87\" (UID: \"58aaea68-aa14-4041-a2ee-c8dbb1c19b55\") " pod="ai-tenant-e2e-shared-b-w2-957b33/rate-test-model-957b33-kserve-7bf748c8f-jjx87" Nov 03 03:02:49.847771 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:49.847682 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58aaea68-aa14-4041-a2ee-c8dbb1c19b55-home\") pod \"rate-test-model-957b33-kserve-7bf748c8f-jjx87\" (UID: \"58aaea68-aa14-4041-a2ee-c8dbb1c19b55\") " pod="ai-tenant-e2e-shared-b-w2-957b33/rate-test-model-957b33-kserve-7bf748c8f-jjx87" Nov 03 03:02:49.847771 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:49.847742 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58aaea68-aa14-4041-a2ee-c8dbb1c19b55-tls-certs\") pod \"rate-test-model-957b33-kserve-7bf748c8f-jjx87\" (UID: \"58aaea68-aa14-4041-a2ee-c8dbb1c19b55\") " pod="ai-tenant-e2e-shared-b-w2-957b33/rate-test-model-957b33-kserve-7bf748c8f-jjx87" Nov 03 03:02:49.847864 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:49.847828 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58aaea68-aa14-4041-a2ee-c8dbb1c19b55-model-cache\") pod \"rate-test-model-957b33-kserve-7bf748c8f-jjx87\" (UID: \"58aaea68-aa14-4041-a2ee-c8dbb1c19b55\") " pod="ai-tenant-e2e-shared-b-w2-957b33/rate-test-model-957b33-kserve-7bf748c8f-jjx87" Nov 03 03:02:49.847920 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:49.847891 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6pp7\" (UniqueName: \"kubernetes.io/projected/58aaea68-aa14-4041-a2ee-c8dbb1c19b55-kube-api-access-q6pp7\") pod \"rate-test-model-957b33-kserve-7bf748c8f-jjx87\" (UID: \"58aaea68-aa14-4041-a2ee-c8dbb1c19b55\") " pod="ai-tenant-e2e-shared-b-w2-957b33/rate-test-model-957b33-kserve-7bf748c8f-jjx87" Nov 03 03:02:49.847920 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:49.847914 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58aaea68-aa14-4041-a2ee-c8dbb1c19b55-tmp-dir\") pod \"rate-test-model-957b33-kserve-7bf748c8f-jjx87\" (UID: \"58aaea68-aa14-4041-a2ee-c8dbb1c19b55\") " pod="ai-tenant-e2e-shared-b-w2-957b33/rate-test-model-957b33-kserve-7bf748c8f-jjx87" Nov 03 03:02:49.890136 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:49.890105 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-957b33\"/\"default-dockercfg-gpnj4\"" Nov 03 03:02:49.890136 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:49.890111 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-957b33\"/\"rate-test-model-957b33-kserve-self-signed-certs\"" Nov 03 03:02:49.890393 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:49.890141 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-957b33\"/\"openshift-service-ca.crt\"" Nov 03 03:02:49.890393 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:49.890238 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-957b33\"/\"kube-root-ca.crt\"" Nov 03 03:02:49.892607 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:49.892581 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-957b33/rate-test-model-957b33-kserve-7bf748c8f-jjx87"] Nov 03 03:02:49.951144 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:49.951103 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58aaea68-aa14-4041-a2ee-c8dbb1c19b55-dshm\") pod \"rate-test-model-957b33-kserve-7bf748c8f-jjx87\" (UID: \"58aaea68-aa14-4041-a2ee-c8dbb1c19b55\") " pod="ai-tenant-e2e-shared-b-w2-957b33/rate-test-model-957b33-kserve-7bf748c8f-jjx87" Nov 03 03:02:49.951390 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:49.951221 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58aaea68-aa14-4041-a2ee-c8dbb1c19b55-home\") pod \"rate-test-model-957b33-kserve-7bf748c8f-jjx87\" (UID: \"58aaea68-aa14-4041-a2ee-c8dbb1c19b55\") " pod="ai-tenant-e2e-shared-b-w2-957b33/rate-test-model-957b33-kserve-7bf748c8f-jjx87" Nov 03 03:02:49.951390 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:49.951344 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58aaea68-aa14-4041-a2ee-c8dbb1c19b55-tls-certs\") pod \"rate-test-model-957b33-kserve-7bf748c8f-jjx87\" (UID: \"58aaea68-aa14-4041-a2ee-c8dbb1c19b55\") " pod="ai-tenant-e2e-shared-b-w2-957b33/rate-test-model-957b33-kserve-7bf748c8f-jjx87" Nov 03 03:02:49.951548 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:49.951491 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58aaea68-aa14-4041-a2ee-c8dbb1c19b55-model-cache\") pod \"rate-test-model-957b33-kserve-7bf748c8f-jjx87\" (UID: \"58aaea68-aa14-4041-a2ee-c8dbb1c19b55\") " pod="ai-tenant-e2e-shared-b-w2-957b33/rate-test-model-957b33-kserve-7bf748c8f-jjx87" Nov 03 03:02:49.951720 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:49.951699 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6pp7\" (UniqueName: \"kubernetes.io/projected/58aaea68-aa14-4041-a2ee-c8dbb1c19b55-kube-api-access-q6pp7\") pod \"rate-test-model-957b33-kserve-7bf748c8f-jjx87\" (UID: \"58aaea68-aa14-4041-a2ee-c8dbb1c19b55\") " pod="ai-tenant-e2e-shared-b-w2-957b33/rate-test-model-957b33-kserve-7bf748c8f-jjx87" Nov 03 03:02:49.951863 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:49.951757 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58aaea68-aa14-4041-a2ee-c8dbb1c19b55-tmp-dir\") pod \"rate-test-model-957b33-kserve-7bf748c8f-jjx87\" (UID: \"58aaea68-aa14-4041-a2ee-c8dbb1c19b55\") " pod="ai-tenant-e2e-shared-b-w2-957b33/rate-test-model-957b33-kserve-7bf748c8f-jjx87" Nov 03 03:02:49.952243 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:49.952211 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58aaea68-aa14-4041-a2ee-c8dbb1c19b55-tmp-dir\") pod \"rate-test-model-957b33-kserve-7bf748c8f-jjx87\" (UID: \"58aaea68-aa14-4041-a2ee-c8dbb1c19b55\") " pod="ai-tenant-e2e-shared-b-w2-957b33/rate-test-model-957b33-kserve-7bf748c8f-jjx87" Nov 03 03:02:49.954506 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:49.953572 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58aaea68-aa14-4041-a2ee-c8dbb1c19b55-home\") pod \"rate-test-model-957b33-kserve-7bf748c8f-jjx87\" (UID: \"58aaea68-aa14-4041-a2ee-c8dbb1c19b55\") " pod="ai-tenant-e2e-shared-b-w2-957b33/rate-test-model-957b33-kserve-7bf748c8f-jjx87" Nov 03 03:02:49.955248 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:49.955223 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58aaea68-aa14-4041-a2ee-c8dbb1c19b55-model-cache\") pod \"rate-test-model-957b33-kserve-7bf748c8f-jjx87\" (UID: \"58aaea68-aa14-4041-a2ee-c8dbb1c19b55\") " pod="ai-tenant-e2e-shared-b-w2-957b33/rate-test-model-957b33-kserve-7bf748c8f-jjx87" Nov 03 03:02:49.957088 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:49.957064 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58aaea68-aa14-4041-a2ee-c8dbb1c19b55-dshm\") pod \"rate-test-model-957b33-kserve-7bf748c8f-jjx87\" (UID: \"58aaea68-aa14-4041-a2ee-c8dbb1c19b55\") " pod="ai-tenant-e2e-shared-b-w2-957b33/rate-test-model-957b33-kserve-7bf748c8f-jjx87" Nov 03 03:02:49.958138 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:49.958112 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58aaea68-aa14-4041-a2ee-c8dbb1c19b55-tls-certs\") pod \"rate-test-model-957b33-kserve-7bf748c8f-jjx87\" (UID: \"58aaea68-aa14-4041-a2ee-c8dbb1c19b55\") " pod="ai-tenant-e2e-shared-b-w2-957b33/rate-test-model-957b33-kserve-7bf748c8f-jjx87" Nov 03 03:02:50.009481 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:50.009452 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6pp7\" (UniqueName: \"kubernetes.io/projected/58aaea68-aa14-4041-a2ee-c8dbb1c19b55-kube-api-access-q6pp7\") pod \"rate-test-model-957b33-kserve-7bf748c8f-jjx87\" (UID: \"58aaea68-aa14-4041-a2ee-c8dbb1c19b55\") " pod="ai-tenant-e2e-shared-b-w2-957b33/rate-test-model-957b33-kserve-7bf748c8f-jjx87" Nov 03 03:02:50.104208 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:50.104182 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-957b33/rate-test-model-957b33-kserve-7bf748c8f-jjx87" Nov 03 03:02:50.483262 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:50.483145 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-957b33/rate-test-model-957b33-kserve-7bf748c8f-jjx87"] Nov 03 03:02:50.489525 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:02:50.489498 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod58aaea68_aa14_4041_a2ee_c8dbb1c19b55.slice/crio-fc38c24ecaf19e0882b8201099544b7ecf7d6daa7310d94e1df3b20a48c7c4f8 WatchSource:0}: Error finding container fc38c24ecaf19e0882b8201099544b7ecf7d6daa7310d94e1df3b20a48c7c4f8: Status 404 returned error can't find the container with id fc38c24ecaf19e0882b8201099544b7ecf7d6daa7310d94e1df3b20a48c7c4f8 Nov 03 03:02:51.027968 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:51.027932 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-957b33/rate-test-model-957b33-kserve-7bf748c8f-jjx87" event={"ID":"58aaea68-aa14-4041-a2ee-c8dbb1c19b55","Type":"ContainerStarted","Data":"14503a3a61315eeee31329d449c65cfdd3884ee7740d1976c8f79cfb653b2a87"} Nov 03 03:02:51.027968 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:51.027972 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-957b33/rate-test-model-957b33-kserve-7bf748c8f-jjx87" event={"ID":"58aaea68-aa14-4041-a2ee-c8dbb1c19b55","Type":"ContainerStarted","Data":"fc38c24ecaf19e0882b8201099544b7ecf7d6daa7310d94e1df3b20a48c7c4f8"} Nov 03 03:02:51.104806 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:51.104781 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-957b33/rate-test-model-957b33-kserve-7bf748c8f-jjx87" Nov 03 03:02:51.135949 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:02:51.135901 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-957b33/rate-test-model-957b33-kserve-7bf748c8f-jjx87" podStartSLOduration=2.135887959 podStartE2EDuration="2.135887959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:02:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:02:51.134793989 +0000 UTC m=+4248.393992143" watchObservedRunningTime="2026-11-03 03:02:51.135887959 +0000 UTC m=+4248.395086112" Nov 03 03:02:53.913077 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:02:53.913038 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:03:00.105207 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:03:00.105171 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-957b33/rate-test-model-957b33-kserve-7bf748c8f-jjx87" Nov 03 03:03:00.117272 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:03:00.117243 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-957b33/rate-test-model-957b33-kserve-7bf748c8f-jjx87" Nov 03 03:03:01.091096 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:03:01.091067 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-957b33/rate-test-model-957b33-kserve-7bf748c8f-jjx87" Nov 03 03:03:02.057319 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:03:02.057290 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:03:05.394569 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:03:05.394516 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1bccbd/distinct-w1-1bccbd-kserve-5dfc85555f-tkvg6" podUID="dd4a9d44-c63e-4f67-bcac-a4f98ecf1796" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 03:03:23.915124 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:03:23.915089 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:03:40.046695 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:03:40.046656 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97dd93e8-kfr9g"] Nov 03 03:03:40.091644 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:03:40.091616 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97dd93e8-kfr9g"] Nov 03 03:03:40.850649 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:03:40.850615 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ad0dc91-1b2d-49bb-81d4-6f50993b7ff9" path="/var/lib/kubelet/pods/0ad0dc91-1b2d-49bb-81d4-6f50993b7ff9/volumes" Nov 03 03:03:53.913019 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:03:53.912979 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:04:10.479829 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:10.479784 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht"] Nov 03 03:04:10.484352 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:10.484322 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht" Nov 03 03:04:10.534209 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:10.534184 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-85e72378-openshift-default-dockercfg-zsnt9\"" Nov 03 03:04:10.583468 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:10.583429 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht"] Nov 03 03:04:10.586031 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:10.586005 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/261be9f1-1325-4210-b5f5-18b81a6b730d-istio-podinfo\") pod \"e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht\" (UID: \"261be9f1-1325-4210-b5f5-18b81a6b730d\") " pod="openshift-ingress/e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht" Nov 03 03:04:10.586183 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:10.586048 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/261be9f1-1325-4210-b5f5-18b81a6b730d-workload-socket\") pod \"e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht\" (UID: \"261be9f1-1325-4210-b5f5-18b81a6b730d\") " pod="openshift-ingress/e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht" Nov 03 03:04:10.586183 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:10.586153 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/261be9f1-1325-4210-b5f5-18b81a6b730d-istiod-ca-cert\") pod \"e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht\" (UID: \"261be9f1-1325-4210-b5f5-18b81a6b730d\") " pod="openshift-ingress/e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht" Nov 03 03:04:10.586183 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:10.586171 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/261be9f1-1325-4210-b5f5-18b81a6b730d-credential-socket\") pod \"e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht\" (UID: \"261be9f1-1325-4210-b5f5-18b81a6b730d\") " pod="openshift-ingress/e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht" Nov 03 03:04:10.586401 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:10.586207 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swrr4\" (UniqueName: \"kubernetes.io/projected/261be9f1-1325-4210-b5f5-18b81a6b730d-kube-api-access-swrr4\") pod \"e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht\" (UID: \"261be9f1-1325-4210-b5f5-18b81a6b730d\") " pod="openshift-ingress/e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht" Nov 03 03:04:10.586401 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:10.586245 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/261be9f1-1325-4210-b5f5-18b81a6b730d-workload-certs\") pod \"e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht\" (UID: \"261be9f1-1325-4210-b5f5-18b81a6b730d\") " pod="openshift-ingress/e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht" Nov 03 03:04:10.586401 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:10.586300 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/261be9f1-1325-4210-b5f5-18b81a6b730d-istio-data\") pod \"e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht\" (UID: \"261be9f1-1325-4210-b5f5-18b81a6b730d\") " pod="openshift-ingress/e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht" Nov 03 03:04:10.586401 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:10.586330 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/261be9f1-1325-4210-b5f5-18b81a6b730d-istio-token\") pod \"e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht\" (UID: \"261be9f1-1325-4210-b5f5-18b81a6b730d\") " pod="openshift-ingress/e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht" Nov 03 03:04:10.586643 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:10.586401 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/261be9f1-1325-4210-b5f5-18b81a6b730d-istio-envoy\") pod \"e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht\" (UID: \"261be9f1-1325-4210-b5f5-18b81a6b730d\") " pod="openshift-ingress/e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht" Nov 03 03:04:10.687170 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:10.687124 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/261be9f1-1325-4210-b5f5-18b81a6b730d-istiod-ca-cert\") pod \"e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht\" (UID: \"261be9f1-1325-4210-b5f5-18b81a6b730d\") " pod="openshift-ingress/e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht" Nov 03 03:04:10.688193 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:10.688170 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/261be9f1-1325-4210-b5f5-18b81a6b730d-credential-socket\") pod \"e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht\" (UID: \"261be9f1-1325-4210-b5f5-18b81a6b730d\") " pod="openshift-ingress/e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht" Nov 03 03:04:10.688419 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:10.688388 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swrr4\" (UniqueName: \"kubernetes.io/projected/261be9f1-1325-4210-b5f5-18b81a6b730d-kube-api-access-swrr4\") pod \"e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht\" (UID: \"261be9f1-1325-4210-b5f5-18b81a6b730d\") " pod="openshift-ingress/e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht" Nov 03 03:04:10.688561 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:10.688546 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/261be9f1-1325-4210-b5f5-18b81a6b730d-workload-certs\") pod \"e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht\" (UID: \"261be9f1-1325-4210-b5f5-18b81a6b730d\") " pod="openshift-ingress/e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht" Nov 03 03:04:10.688677 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:10.688663 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/261be9f1-1325-4210-b5f5-18b81a6b730d-istio-data\") pod \"e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht\" (UID: \"261be9f1-1325-4210-b5f5-18b81a6b730d\") " pod="openshift-ingress/e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht" Nov 03 03:04:10.688792 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:10.688781 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/261be9f1-1325-4210-b5f5-18b81a6b730d-istio-token\") pod \"e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht\" (UID: \"261be9f1-1325-4210-b5f5-18b81a6b730d\") " pod="openshift-ingress/e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht" Nov 03 03:04:10.688895 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:10.688882 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/261be9f1-1325-4210-b5f5-18b81a6b730d-istio-envoy\") pod \"e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht\" (UID: \"261be9f1-1325-4210-b5f5-18b81a6b730d\") " pod="openshift-ingress/e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht" Nov 03 03:04:10.689152 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:10.689135 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/261be9f1-1325-4210-b5f5-18b81a6b730d-istio-podinfo\") pod \"e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht\" (UID: \"261be9f1-1325-4210-b5f5-18b81a6b730d\") " pod="openshift-ingress/e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht" Nov 03 03:04:10.689306 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:10.689286 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/261be9f1-1325-4210-b5f5-18b81a6b730d-workload-socket\") pod \"e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht\" (UID: \"261be9f1-1325-4210-b5f5-18b81a6b730d\") " pod="openshift-ingress/e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht" Nov 03 03:04:10.689858 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:10.689838 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/261be9f1-1325-4210-b5f5-18b81a6b730d-workload-socket\") pod \"e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht\" (UID: \"261be9f1-1325-4210-b5f5-18b81a6b730d\") " pod="openshift-ingress/e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht" Nov 03 03:04:10.690549 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:10.690217 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/261be9f1-1325-4210-b5f5-18b81a6b730d-credential-socket\") pod \"e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht\" (UID: \"261be9f1-1325-4210-b5f5-18b81a6b730d\") " pod="openshift-ingress/e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht" Nov 03 03:04:10.690706 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:10.688113 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/261be9f1-1325-4210-b5f5-18b81a6b730d-istiod-ca-cert\") pod \"e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht\" (UID: \"261be9f1-1325-4210-b5f5-18b81a6b730d\") " pod="openshift-ingress/e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht" Nov 03 03:04:10.690816 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:10.690294 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/261be9f1-1325-4210-b5f5-18b81a6b730d-istio-data\") pod \"e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht\" (UID: \"261be9f1-1325-4210-b5f5-18b81a6b730d\") " pod="openshift-ingress/e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht" Nov 03 03:04:10.690933 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:10.690890 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/261be9f1-1325-4210-b5f5-18b81a6b730d-workload-certs\") pod \"e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht\" (UID: \"261be9f1-1325-4210-b5f5-18b81a6b730d\") " pod="openshift-ingress/e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht" Nov 03 03:04:10.697931 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:10.696122 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/261be9f1-1325-4210-b5f5-18b81a6b730d-istio-envoy\") pod \"e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht\" (UID: \"261be9f1-1325-4210-b5f5-18b81a6b730d\") " pod="openshift-ingress/e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht" Nov 03 03:04:10.699062 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:10.699038 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/261be9f1-1325-4210-b5f5-18b81a6b730d-istio-podinfo\") pod \"e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht\" (UID: \"261be9f1-1325-4210-b5f5-18b81a6b730d\") " pod="openshift-ingress/e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht" Nov 03 03:04:10.746828 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:10.746757 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/261be9f1-1325-4210-b5f5-18b81a6b730d-istio-token\") pod \"e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht\" (UID: \"261be9f1-1325-4210-b5f5-18b81a6b730d\") " pod="openshift-ingress/e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht" Nov 03 03:04:10.747004 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:10.746980 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swrr4\" (UniqueName: \"kubernetes.io/projected/261be9f1-1325-4210-b5f5-18b81a6b730d-kube-api-access-swrr4\") pod \"e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht\" (UID: \"261be9f1-1325-4210-b5f5-18b81a6b730d\") " pod="openshift-ingress/e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht" Nov 03 03:04:10.797605 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:10.797574 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht" Nov 03 03:04:11.238167 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:11.238134 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht"] Nov 03 03:04:11.244295 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:04:11.244266 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod261be9f1_1325_4210_b5f5_18b81a6b730d.slice/crio-c04b80f42267ce0f18cb2b4f1c91799499b7a39e603d755421bb20432b7e369f WatchSource:0}: Error finding container c04b80f42267ce0f18cb2b4f1c91799499b7a39e603d755421bb20432b7e369f: Status 404 returned error can't find the container with id c04b80f42267ce0f18cb2b4f1c91799499b7a39e603d755421bb20432b7e369f Nov 03 03:04:11.246657 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:11.246624 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 03:04:11.246746 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:11.246689 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 03:04:11.246746 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:11.246717 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 03:04:11.529350 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:11.529248 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht" event={"ID":"261be9f1-1325-4210-b5f5-18b81a6b730d","Type":"ContainerStarted","Data":"87ae1fadf136362ee50138f4b289a5ab6bbf76148e2c01cf39bb58ced4c787c4"} Nov 03 03:04:11.529350 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:11.529308 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht" event={"ID":"261be9f1-1325-4210-b5f5-18b81a6b730d","Type":"ContainerStarted","Data":"c04b80f42267ce0f18cb2b4f1c91799499b7a39e603d755421bb20432b7e369f"} Nov 03 03:04:11.640215 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:11.640149 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht" podStartSLOduration=1.640131157 podStartE2EDuration="1.640131157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:04:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:04:11.639572482 +0000 UTC m=+4328.898770637" watchObservedRunningTime="2026-11-03 03:04:11.640131157 +0000 UTC m=+4328.899329314" Nov 03 03:04:11.798266 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:11.798171 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht" Nov 03 03:04:12.803939 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:12.803906 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht" Nov 03 03:04:13.540507 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:13.540473 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht" Nov 03 03:04:13.541323 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:13.541299 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht" Nov 03 03:04:23.917733 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:04:23.917698 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:04:28.770674 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:28.770643 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-9aebe8/rate-test-model-9aebe8-kserve-76d8d8c6c7-bnqlv"] Nov 03 03:04:28.774510 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:28.774490 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-9aebe8/rate-test-model-9aebe8-kserve-76d8d8c6c7-bnqlv" Nov 03 03:04:28.872219 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:28.872180 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-9aebe8\"/\"openshift-service-ca.crt\"" Nov 03 03:04:28.872219 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:28.872203 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-9aebe8\"/\"kube-root-ca.crt\"" Nov 03 03:04:28.872470 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:28.872258 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-9aebe8\"/\"rate-test-model-9aebe8-kserve-self-signed-certs\"" Nov 03 03:04:28.872470 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:28.872282 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-9aebe8\"/\"default-dockercfg-nvn92\"" Nov 03 03:04:28.874526 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:28.874509 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-9aebe8/rate-test-model-9aebe8-kserve-76d8d8c6c7-bnqlv"] Nov 03 03:04:28.900918 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:28.900891 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e54a46d9-0329-4076-9d46-c63d5827f8ea-model-cache\") pod \"rate-test-model-9aebe8-kserve-76d8d8c6c7-bnqlv\" (UID: \"e54a46d9-0329-4076-9d46-c63d5827f8ea\") " pod="ai-tenant-e2e-shared-a-w0-9aebe8/rate-test-model-9aebe8-kserve-76d8d8c6c7-bnqlv" Nov 03 03:04:28.901044 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:28.900928 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e54a46d9-0329-4076-9d46-c63d5827f8ea-dshm\") pod \"rate-test-model-9aebe8-kserve-76d8d8c6c7-bnqlv\" (UID: \"e54a46d9-0329-4076-9d46-c63d5827f8ea\") " pod="ai-tenant-e2e-shared-a-w0-9aebe8/rate-test-model-9aebe8-kserve-76d8d8c6c7-bnqlv" Nov 03 03:04:28.901044 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:28.900956 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e54a46d9-0329-4076-9d46-c63d5827f8ea-tls-certs\") pod \"rate-test-model-9aebe8-kserve-76d8d8c6c7-bnqlv\" (UID: \"e54a46d9-0329-4076-9d46-c63d5827f8ea\") " pod="ai-tenant-e2e-shared-a-w0-9aebe8/rate-test-model-9aebe8-kserve-76d8d8c6c7-bnqlv" Nov 03 03:04:28.901129 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:28.901064 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e54a46d9-0329-4076-9d46-c63d5827f8ea-tmp-dir\") pod \"rate-test-model-9aebe8-kserve-76d8d8c6c7-bnqlv\" (UID: \"e54a46d9-0329-4076-9d46-c63d5827f8ea\") " pod="ai-tenant-e2e-shared-a-w0-9aebe8/rate-test-model-9aebe8-kserve-76d8d8c6c7-bnqlv" Nov 03 03:04:28.901165 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:28.901136 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e54a46d9-0329-4076-9d46-c63d5827f8ea-home\") pod \"rate-test-model-9aebe8-kserve-76d8d8c6c7-bnqlv\" (UID: \"e54a46d9-0329-4076-9d46-c63d5827f8ea\") " pod="ai-tenant-e2e-shared-a-w0-9aebe8/rate-test-model-9aebe8-kserve-76d8d8c6c7-bnqlv" Nov 03 03:04:28.901242 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:28.901213 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gwvc\" (UniqueName: \"kubernetes.io/projected/e54a46d9-0329-4076-9d46-c63d5827f8ea-kube-api-access-2gwvc\") pod \"rate-test-model-9aebe8-kserve-76d8d8c6c7-bnqlv\" (UID: \"e54a46d9-0329-4076-9d46-c63d5827f8ea\") " pod="ai-tenant-e2e-shared-a-w0-9aebe8/rate-test-model-9aebe8-kserve-76d8d8c6c7-bnqlv" Nov 03 03:04:29.002449 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:29.002408 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e54a46d9-0329-4076-9d46-c63d5827f8ea-model-cache\") pod \"rate-test-model-9aebe8-kserve-76d8d8c6c7-bnqlv\" (UID: \"e54a46d9-0329-4076-9d46-c63d5827f8ea\") " pod="ai-tenant-e2e-shared-a-w0-9aebe8/rate-test-model-9aebe8-kserve-76d8d8c6c7-bnqlv" Nov 03 03:04:29.002449 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:29.002454 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e54a46d9-0329-4076-9d46-c63d5827f8ea-dshm\") pod \"rate-test-model-9aebe8-kserve-76d8d8c6c7-bnqlv\" (UID: \"e54a46d9-0329-4076-9d46-c63d5827f8ea\") " pod="ai-tenant-e2e-shared-a-w0-9aebe8/rate-test-model-9aebe8-kserve-76d8d8c6c7-bnqlv" Nov 03 03:04:29.002723 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:29.002483 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e54a46d9-0329-4076-9d46-c63d5827f8ea-tls-certs\") pod \"rate-test-model-9aebe8-kserve-76d8d8c6c7-bnqlv\" (UID: \"e54a46d9-0329-4076-9d46-c63d5827f8ea\") " pod="ai-tenant-e2e-shared-a-w0-9aebe8/rate-test-model-9aebe8-kserve-76d8d8c6c7-bnqlv" Nov 03 03:04:29.002723 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:29.002508 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e54a46d9-0329-4076-9d46-c63d5827f8ea-tmp-dir\") pod \"rate-test-model-9aebe8-kserve-76d8d8c6c7-bnqlv\" (UID: \"e54a46d9-0329-4076-9d46-c63d5827f8ea\") " pod="ai-tenant-e2e-shared-a-w0-9aebe8/rate-test-model-9aebe8-kserve-76d8d8c6c7-bnqlv" Nov 03 03:04:29.002723 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:29.002568 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e54a46d9-0329-4076-9d46-c63d5827f8ea-home\") pod \"rate-test-model-9aebe8-kserve-76d8d8c6c7-bnqlv\" (UID: \"e54a46d9-0329-4076-9d46-c63d5827f8ea\") " pod="ai-tenant-e2e-shared-a-w0-9aebe8/rate-test-model-9aebe8-kserve-76d8d8c6c7-bnqlv" Nov 03 03:04:29.002723 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:29.002613 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gwvc\" (UniqueName: \"kubernetes.io/projected/e54a46d9-0329-4076-9d46-c63d5827f8ea-kube-api-access-2gwvc\") pod \"rate-test-model-9aebe8-kserve-76d8d8c6c7-bnqlv\" (UID: \"e54a46d9-0329-4076-9d46-c63d5827f8ea\") " pod="ai-tenant-e2e-shared-a-w0-9aebe8/rate-test-model-9aebe8-kserve-76d8d8c6c7-bnqlv" Nov 03 03:04:29.002955 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:29.002849 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e54a46d9-0329-4076-9d46-c63d5827f8ea-model-cache\") pod \"rate-test-model-9aebe8-kserve-76d8d8c6c7-bnqlv\" (UID: \"e54a46d9-0329-4076-9d46-c63d5827f8ea\") " pod="ai-tenant-e2e-shared-a-w0-9aebe8/rate-test-model-9aebe8-kserve-76d8d8c6c7-bnqlv" Nov 03 03:04:29.002955 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:29.002932 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e54a46d9-0329-4076-9d46-c63d5827f8ea-tmp-dir\") pod \"rate-test-model-9aebe8-kserve-76d8d8c6c7-bnqlv\" (UID: \"e54a46d9-0329-4076-9d46-c63d5827f8ea\") " pod="ai-tenant-e2e-shared-a-w0-9aebe8/rate-test-model-9aebe8-kserve-76d8d8c6c7-bnqlv" Nov 03 03:04:29.003092 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:29.003072 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e54a46d9-0329-4076-9d46-c63d5827f8ea-home\") pod \"rate-test-model-9aebe8-kserve-76d8d8c6c7-bnqlv\" (UID: \"e54a46d9-0329-4076-9d46-c63d5827f8ea\") " pod="ai-tenant-e2e-shared-a-w0-9aebe8/rate-test-model-9aebe8-kserve-76d8d8c6c7-bnqlv" Nov 03 03:04:29.004978 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:29.004936 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e54a46d9-0329-4076-9d46-c63d5827f8ea-dshm\") pod \"rate-test-model-9aebe8-kserve-76d8d8c6c7-bnqlv\" (UID: \"e54a46d9-0329-4076-9d46-c63d5827f8ea\") " pod="ai-tenant-e2e-shared-a-w0-9aebe8/rate-test-model-9aebe8-kserve-76d8d8c6c7-bnqlv" Nov 03 03:04:29.005142 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:29.005120 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e54a46d9-0329-4076-9d46-c63d5827f8ea-tls-certs\") pod \"rate-test-model-9aebe8-kserve-76d8d8c6c7-bnqlv\" (UID: \"e54a46d9-0329-4076-9d46-c63d5827f8ea\") " pod="ai-tenant-e2e-shared-a-w0-9aebe8/rate-test-model-9aebe8-kserve-76d8d8c6c7-bnqlv" Nov 03 03:04:29.056808 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:29.056781 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gwvc\" (UniqueName: \"kubernetes.io/projected/e54a46d9-0329-4076-9d46-c63d5827f8ea-kube-api-access-2gwvc\") pod \"rate-test-model-9aebe8-kserve-76d8d8c6c7-bnqlv\" (UID: \"e54a46d9-0329-4076-9d46-c63d5827f8ea\") " pod="ai-tenant-e2e-shared-a-w0-9aebe8/rate-test-model-9aebe8-kserve-76d8d8c6c7-bnqlv" Nov 03 03:04:29.087724 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:29.087686 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-9aebe8/rate-test-model-9aebe8-kserve-76d8d8c6c7-bnqlv" Nov 03 03:04:29.465772 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:29.465693 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-9aebe8/rate-test-model-9aebe8-kserve-76d8d8c6c7-bnqlv"] Nov 03 03:04:29.471087 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:04:29.471055 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode54a46d9_0329_4076_9d46_c63d5827f8ea.slice/crio-ff32253792a9336a85cdeb250105b370fa23671958cf0614c3d93a9549422389 WatchSource:0}: Error finding container ff32253792a9336a85cdeb250105b370fa23671958cf0614c3d93a9549422389: Status 404 returned error can't find the container with id ff32253792a9336a85cdeb250105b370fa23671958cf0614c3d93a9549422389 Nov 03 03:04:29.630419 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:29.630387 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-9aebe8/rate-test-model-9aebe8-kserve-76d8d8c6c7-bnqlv" event={"ID":"e54a46d9-0329-4076-9d46-c63d5827f8ea","Type":"ContainerStarted","Data":"b45d67b3a9a08bc6d2f01461f608595e3bab006247418972ffffe0dd9b1f389f"} Nov 03 03:04:29.630588 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:29.630433 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-9aebe8/rate-test-model-9aebe8-kserve-76d8d8c6c7-bnqlv" event={"ID":"e54a46d9-0329-4076-9d46-c63d5827f8ea","Type":"ContainerStarted","Data":"ff32253792a9336a85cdeb250105b370fa23671958cf0614c3d93a9549422389"} Nov 03 03:04:29.737499 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:29.737393 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-9aebe8/rate-test-model-9aebe8-kserve-76d8d8c6c7-bnqlv" podStartSLOduration=1.737352525 podStartE2EDuration="1.737352525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:04:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:04:29.737050397 +0000 UTC m=+4346.996248552" watchObservedRunningTime="2026-11-03 03:04:29.737352525 +0000 UTC m=+4346.996550678" Nov 03 03:04:30.087776 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:30.087744 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-9aebe8/rate-test-model-9aebe8-kserve-76d8d8c6c7-bnqlv" Nov 03 03:04:39.087881 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:39.087842 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-9aebe8/rate-test-model-9aebe8-kserve-76d8d8c6c7-bnqlv" Nov 03 03:04:39.100514 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:39.100485 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-9aebe8/rate-test-model-9aebe8-kserve-76d8d8c6c7-bnqlv" Nov 03 03:04:39.696950 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:39.696921 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-9aebe8/rate-test-model-9aebe8-kserve-76d8d8c6c7-bnqlv" Nov 03 03:04:42.424110 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:04:42.424068 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:04:53.912453 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:04:53.912415 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:05:05.096184 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:05.096151 5605 scope.go:122] "RemoveContainer" containerID="c026d0fec0ed19eb31a2b6ccaa6b996283378375ded849cfe2211bfa81b433bf" Nov 03 03:05:23.915971 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:05:23.915926 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:05:34.775935 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:34.775897 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-3a4683/sub-test-model-3a4683-kserve-656f748dc5-nzhlm"] Nov 03 03:05:34.779550 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:34.779525 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-3a4683/sub-test-model-3a4683-kserve-656f748dc5-nzhlm" Nov 03 03:05:34.872028 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:34.871990 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d37c620a-1824-4c26-b8da-3e44aefcbe6b-model-cache\") pod \"sub-test-model-3a4683-kserve-656f748dc5-nzhlm\" (UID: \"d37c620a-1824-4c26-b8da-3e44aefcbe6b\") " pod="ai-tenant-e2e-shared-a-w1-3a4683/sub-test-model-3a4683-kserve-656f748dc5-nzhlm" Nov 03 03:05:34.872231 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:34.872130 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d37c620a-1824-4c26-b8da-3e44aefcbe6b-tmp-dir\") pod \"sub-test-model-3a4683-kserve-656f748dc5-nzhlm\" (UID: \"d37c620a-1824-4c26-b8da-3e44aefcbe6b\") " pod="ai-tenant-e2e-shared-a-w1-3a4683/sub-test-model-3a4683-kserve-656f748dc5-nzhlm" Nov 03 03:05:34.872231 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:34.872184 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzfqm\" (UniqueName: \"kubernetes.io/projected/d37c620a-1824-4c26-b8da-3e44aefcbe6b-kube-api-access-xzfqm\") pod \"sub-test-model-3a4683-kserve-656f748dc5-nzhlm\" (UID: \"d37c620a-1824-4c26-b8da-3e44aefcbe6b\") " pod="ai-tenant-e2e-shared-a-w1-3a4683/sub-test-model-3a4683-kserve-656f748dc5-nzhlm" Nov 03 03:05:34.872231 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:34.872210 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d37c620a-1824-4c26-b8da-3e44aefcbe6b-dshm\") pod \"sub-test-model-3a4683-kserve-656f748dc5-nzhlm\" (UID: \"d37c620a-1824-4c26-b8da-3e44aefcbe6b\") " pod="ai-tenant-e2e-shared-a-w1-3a4683/sub-test-model-3a4683-kserve-656f748dc5-nzhlm" Nov 03 03:05:34.872377 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:34.872273 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d37c620a-1824-4c26-b8da-3e44aefcbe6b-home\") pod \"sub-test-model-3a4683-kserve-656f748dc5-nzhlm\" (UID: \"d37c620a-1824-4c26-b8da-3e44aefcbe6b\") " pod="ai-tenant-e2e-shared-a-w1-3a4683/sub-test-model-3a4683-kserve-656f748dc5-nzhlm" Nov 03 03:05:34.872434 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:34.872388 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d37c620a-1824-4c26-b8da-3e44aefcbe6b-tls-certs\") pod \"sub-test-model-3a4683-kserve-656f748dc5-nzhlm\" (UID: \"d37c620a-1824-4c26-b8da-3e44aefcbe6b\") " pod="ai-tenant-e2e-shared-a-w1-3a4683/sub-test-model-3a4683-kserve-656f748dc5-nzhlm" Nov 03 03:05:34.876935 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:34.876913 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-3a4683\"/\"default-dockercfg-9qvwg\"" Nov 03 03:05:34.877035 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:34.876967 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-3a4683\"/\"openshift-service-ca.crt\"" Nov 03 03:05:34.877035 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:34.876973 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-3a4683\"/\"kube-root-ca.crt\"" Nov 03 03:05:34.879507 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:34.879487 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-3a4683/sub-test-model-3a4683-kserve-656f748dc5-nzhlm"] Nov 03 03:05:34.931066 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:34.931032 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-3a4683\"/\"sub-test-model-3a4683-kserve-self-signed-certs\"" Nov 03 03:05:34.973540 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:34.973508 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d37c620a-1824-4c26-b8da-3e44aefcbe6b-model-cache\") pod \"sub-test-model-3a4683-kserve-656f748dc5-nzhlm\" (UID: \"d37c620a-1824-4c26-b8da-3e44aefcbe6b\") " pod="ai-tenant-e2e-shared-a-w1-3a4683/sub-test-model-3a4683-kserve-656f748dc5-nzhlm" Nov 03 03:05:34.973739 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:34.973595 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d37c620a-1824-4c26-b8da-3e44aefcbe6b-tmp-dir\") pod \"sub-test-model-3a4683-kserve-656f748dc5-nzhlm\" (UID: \"d37c620a-1824-4c26-b8da-3e44aefcbe6b\") " pod="ai-tenant-e2e-shared-a-w1-3a4683/sub-test-model-3a4683-kserve-656f748dc5-nzhlm" Nov 03 03:05:34.973739 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:34.973628 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzfqm\" (UniqueName: \"kubernetes.io/projected/d37c620a-1824-4c26-b8da-3e44aefcbe6b-kube-api-access-xzfqm\") pod \"sub-test-model-3a4683-kserve-656f748dc5-nzhlm\" (UID: \"d37c620a-1824-4c26-b8da-3e44aefcbe6b\") " pod="ai-tenant-e2e-shared-a-w1-3a4683/sub-test-model-3a4683-kserve-656f748dc5-nzhlm" Nov 03 03:05:34.973739 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:34.973654 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d37c620a-1824-4c26-b8da-3e44aefcbe6b-dshm\") pod \"sub-test-model-3a4683-kserve-656f748dc5-nzhlm\" (UID: \"d37c620a-1824-4c26-b8da-3e44aefcbe6b\") " pod="ai-tenant-e2e-shared-a-w1-3a4683/sub-test-model-3a4683-kserve-656f748dc5-nzhlm" Nov 03 03:05:34.973739 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:34.973701 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d37c620a-1824-4c26-b8da-3e44aefcbe6b-home\") pod \"sub-test-model-3a4683-kserve-656f748dc5-nzhlm\" (UID: \"d37c620a-1824-4c26-b8da-3e44aefcbe6b\") " pod="ai-tenant-e2e-shared-a-w1-3a4683/sub-test-model-3a4683-kserve-656f748dc5-nzhlm" Nov 03 03:05:34.973986 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:34.973778 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d37c620a-1824-4c26-b8da-3e44aefcbe6b-tls-certs\") pod \"sub-test-model-3a4683-kserve-656f748dc5-nzhlm\" (UID: \"d37c620a-1824-4c26-b8da-3e44aefcbe6b\") " pod="ai-tenant-e2e-shared-a-w1-3a4683/sub-test-model-3a4683-kserve-656f748dc5-nzhlm" Nov 03 03:05:34.974051 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:34.974026 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d37c620a-1824-4c26-b8da-3e44aefcbe6b-model-cache\") pod \"sub-test-model-3a4683-kserve-656f748dc5-nzhlm\" (UID: \"d37c620a-1824-4c26-b8da-3e44aefcbe6b\") " pod="ai-tenant-e2e-shared-a-w1-3a4683/sub-test-model-3a4683-kserve-656f748dc5-nzhlm" Nov 03 03:05:34.974155 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:34.974138 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d37c620a-1824-4c26-b8da-3e44aefcbe6b-tmp-dir\") pod \"sub-test-model-3a4683-kserve-656f748dc5-nzhlm\" (UID: \"d37c620a-1824-4c26-b8da-3e44aefcbe6b\") " pod="ai-tenant-e2e-shared-a-w1-3a4683/sub-test-model-3a4683-kserve-656f748dc5-nzhlm" Nov 03 03:05:34.974253 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:34.974218 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d37c620a-1824-4c26-b8da-3e44aefcbe6b-home\") pod \"sub-test-model-3a4683-kserve-656f748dc5-nzhlm\" (UID: \"d37c620a-1824-4c26-b8da-3e44aefcbe6b\") " pod="ai-tenant-e2e-shared-a-w1-3a4683/sub-test-model-3a4683-kserve-656f748dc5-nzhlm" Nov 03 03:05:34.976132 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:34.976107 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d37c620a-1824-4c26-b8da-3e44aefcbe6b-dshm\") pod \"sub-test-model-3a4683-kserve-656f748dc5-nzhlm\" (UID: \"d37c620a-1824-4c26-b8da-3e44aefcbe6b\") " pod="ai-tenant-e2e-shared-a-w1-3a4683/sub-test-model-3a4683-kserve-656f748dc5-nzhlm" Nov 03 03:05:34.976340 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:34.976324 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d37c620a-1824-4c26-b8da-3e44aefcbe6b-tls-certs\") pod \"sub-test-model-3a4683-kserve-656f748dc5-nzhlm\" (UID: \"d37c620a-1824-4c26-b8da-3e44aefcbe6b\") " pod="ai-tenant-e2e-shared-a-w1-3a4683/sub-test-model-3a4683-kserve-656f748dc5-nzhlm" Nov 03 03:05:35.027068 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:35.026994 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzfqm\" (UniqueName: \"kubernetes.io/projected/d37c620a-1824-4c26-b8da-3e44aefcbe6b-kube-api-access-xzfqm\") pod \"sub-test-model-3a4683-kserve-656f748dc5-nzhlm\" (UID: \"d37c620a-1824-4c26-b8da-3e44aefcbe6b\") " pod="ai-tenant-e2e-shared-a-w1-3a4683/sub-test-model-3a4683-kserve-656f748dc5-nzhlm" Nov 03 03:05:35.092383 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:35.092323 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-3a4683/sub-test-model-3a4683-kserve-656f748dc5-nzhlm" Nov 03 03:05:35.466790 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:35.466754 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-3a4683/sub-test-model-3a4683-kserve-656f748dc5-nzhlm"] Nov 03 03:05:35.472913 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:05:35.472882 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd37c620a_1824_4c26_b8da_3e44aefcbe6b.slice/crio-a75ed5cb9d1324f2a27efeab99e926372a1f1a5b8e327340e40e9703d8c46a9a WatchSource:0}: Error finding container a75ed5cb9d1324f2a27efeab99e926372a1f1a5b8e327340e40e9703d8c46a9a: Status 404 returned error can't find the container with id a75ed5cb9d1324f2a27efeab99e926372a1f1a5b8e327340e40e9703d8c46a9a Nov 03 03:05:36.014015 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:36.013979 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-3a4683/sub-test-model-3a4683-kserve-656f748dc5-nzhlm" event={"ID":"d37c620a-1824-4c26-b8da-3e44aefcbe6b","Type":"ContainerStarted","Data":"442939ee02dd9e2774683d5334b47bed5dbd85caa76189b21400a8c4fbf4c5d0"} Nov 03 03:05:36.014448 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:36.014026 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-3a4683/sub-test-model-3a4683-kserve-656f748dc5-nzhlm" event={"ID":"d37c620a-1824-4c26-b8da-3e44aefcbe6b","Type":"ContainerStarted","Data":"a75ed5cb9d1324f2a27efeab99e926372a1f1a5b8e327340e40e9703d8c46a9a"} Nov 03 03:05:36.093271 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:36.093239 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-3a4683/sub-test-model-3a4683-kserve-656f748dc5-nzhlm" Nov 03 03:05:36.121111 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:36.121058 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-3a4683/sub-test-model-3a4683-kserve-656f748dc5-nzhlm" podStartSLOduration=2.121042255 podStartE2EDuration="2.121042255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:05:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:05:36.120440157 +0000 UTC m=+4413.379638313" watchObservedRunningTime="2026-11-03 03:05:36.121042255 +0000 UTC m=+4413.380240410" Nov 03 03:05:41.264048 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:41.263959 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:05:45.093286 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:45.093251 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-3a4683/sub-test-model-3a4683-kserve-656f748dc5-nzhlm" Nov 03 03:05:45.106984 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:45.106955 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-3a4683/sub-test-model-3a4683-kserve-656f748dc5-nzhlm" Nov 03 03:05:46.104425 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:46.104388 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-3a4683/sub-test-model-3a4683-kserve-656f748dc5-nzhlm" Nov 03 03:05:51.044610 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:51.044577 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e003baac-rb25p"] Nov 03 03:05:51.092278 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:51.092242 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e003baac-rb25p"] Nov 03 03:05:51.414890 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:51.414856 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26741169-4jxgn"] Nov 03 03:05:51.418853 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:51.418830 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26741169-4jxgn" Nov 03 03:05:51.471349 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:51.471253 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:05:51.471494 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:51.471351 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:05:51.471494 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:51.471388 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:05:51.471494 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:51.471429 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 03:05:51.472443 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:51.472420 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26741169-4jxgn"] Nov 03 03:05:51.491967 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:51.491942 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56htp\" (UniqueName: \"kubernetes.io/projected/4dfe0768-867a-4f0c-9d33-c9f0e2e947d6-kube-api-access-56htp\") pod \"maas-api-revoke-sub-26741169-4jxgn\" (UID: \"4dfe0768-867a-4f0c-9d33-c9f0e2e947d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26741169-4jxgn" Nov 03 03:05:51.492094 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:51.491980 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dfe0768-867a-4f0c-9d33-c9f0e2e947d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26741169-4jxgn\" (UID: \"4dfe0768-867a-4f0c-9d33-c9f0e2e947d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26741169-4jxgn" Nov 03 03:05:51.492094 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:51.492009 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dfe0768-867a-4f0c-9d33-c9f0e2e947d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26741169-4jxgn\" (UID: \"4dfe0768-867a-4f0c-9d33-c9f0e2e947d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26741169-4jxgn" Nov 03 03:05:51.593710 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:51.593643 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56htp\" (UniqueName: \"kubernetes.io/projected/4dfe0768-867a-4f0c-9d33-c9f0e2e947d6-kube-api-access-56htp\") pod \"maas-api-revoke-sub-26741169-4jxgn\" (UID: \"4dfe0768-867a-4f0c-9d33-c9f0e2e947d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26741169-4jxgn" Nov 03 03:05:51.593710 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:51.593680 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dfe0768-867a-4f0c-9d33-c9f0e2e947d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26741169-4jxgn\" (UID: \"4dfe0768-867a-4f0c-9d33-c9f0e2e947d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26741169-4jxgn" Nov 03 03:05:51.593710 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:51.593707 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dfe0768-867a-4f0c-9d33-c9f0e2e947d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26741169-4jxgn\" (UID: \"4dfe0768-867a-4f0c-9d33-c9f0e2e947d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26741169-4jxgn" Nov 03 03:05:51.594475 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:51.594454 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dfe0768-867a-4f0c-9d33-c9f0e2e947d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26741169-4jxgn\" (UID: \"4dfe0768-867a-4f0c-9d33-c9f0e2e947d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26741169-4jxgn" Nov 03 03:05:51.596083 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:51.596064 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dfe0768-867a-4f0c-9d33-c9f0e2e947d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26741169-4jxgn\" (UID: \"4dfe0768-867a-4f0c-9d33-c9f0e2e947d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26741169-4jxgn" Nov 03 03:05:51.649752 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:51.649722 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56htp\" (UniqueName: \"kubernetes.io/projected/4dfe0768-867a-4f0c-9d33-c9f0e2e947d6-kube-api-access-56htp\") pod \"maas-api-revoke-sub-26741169-4jxgn\" (UID: \"4dfe0768-867a-4f0c-9d33-c9f0e2e947d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26741169-4jxgn" Nov 03 03:05:51.732739 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:51.732713 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26741169-4jxgn" Nov 03 03:05:52.104292 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:52.104251 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26741169-4jxgn"] Nov 03 03:05:52.113623 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:05:52.113596 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dfe0768_867a_4f0c_9d33_c9f0e2e947d6.slice/crio-1678e8d7b59e97ec4538b1a3465c4a39e7d2e546533f5c64deb4782f44212aaa WatchSource:0}: Error finding container 1678e8d7b59e97ec4538b1a3465c4a39e7d2e546533f5c64deb4782f44212aaa: Status 404 returned error can't find the container with id 1678e8d7b59e97ec4538b1a3465c4a39e7d2e546533f5c64deb4782f44212aaa Nov 03 03:05:52.135677 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:52.135645 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26741169-4jxgn" event={"ID":"4dfe0768-867a-4f0c-9d33-c9f0e2e947d6","Type":"ContainerStarted","Data":"1678e8d7b59e97ec4538b1a3465c4a39e7d2e546533f5c64deb4782f44212aaa"} Nov 03 03:05:52.137127 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:52.137104 5605 generic.go:356] "Generic (PLEG): container finished" podID="261be9f1-1325-4210-b5f5-18b81a6b730d" containerID="87ae1fadf136362ee50138f4b289a5ab6bbf76148e2c01cf39bb58ced4c787c4" exitCode=0 Nov 03 03:05:52.137237 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:52.137171 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht" event={"ID":"261be9f1-1325-4210-b5f5-18b81a6b730d","Type":"ContainerDied","Data":"87ae1fadf136362ee50138f4b289a5ab6bbf76148e2c01cf39bb58ced4c787c4"} Nov 03 03:05:52.137599 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:52.137584 5605 scope.go:122] "RemoveContainer" containerID="87ae1fadf136362ee50138f4b289a5ab6bbf76148e2c01cf39bb58ced4c787c4" Nov 03 03:05:52.138998 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:52.138967 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 03:05:52.139099 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:52.139051 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 03:05:52.139166 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:52.139108 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 03:05:52.851545 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:52.851512 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1679ecf5-9e89-40fe-8fee-dbd5b74d3a7a" path="/var/lib/kubelet/pods/1679ecf5-9e89-40fe-8fee-dbd5b74d3a7a/volumes" Nov 03 03:05:53.145282 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:53.145247 5605 generic.go:356] "Generic (PLEG): container finished" podID="4dfe0768-867a-4f0c-9d33-c9f0e2e947d6" containerID="73c34d1382596fb22d42c25a27598a26b869a60417589e0d1d642c5c1f2b8d96" exitCode=0 Nov 03 03:05:53.145743 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:53.145328 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26741169-4jxgn" event={"ID":"4dfe0768-867a-4f0c-9d33-c9f0e2e947d6","Type":"ContainerDied","Data":"73c34d1382596fb22d42c25a27598a26b869a60417589e0d1d642c5c1f2b8d96"} Nov 03 03:05:53.147267 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:53.147243 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht" event={"ID":"261be9f1-1325-4210-b5f5-18b81a6b730d","Type":"ContainerStarted","Data":"f89e2e82cb5c8624ae74f4b6f52a6676ecb08bb16e5f1df670d69764f65d1e00"} Nov 03 03:05:53.798328 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:53.798293 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht" Nov 03 03:05:53.799250 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:53.799225 5605 patch_prober.go:29] interesting pod/e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body= Nov 03 03:05:53.799302 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:53.799288 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht" podUID="261be9f1-1325-4210-b5f5-18b81a6b730d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" Nov 03 03:05:53.920483 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:05:53.920451 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:05:54.375381 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:54.370752 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26741169-4jxgn" Nov 03 03:05:54.432771 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:54.432685 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dfe0768-867a-4f0c-9d33-c9f0e2e947d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dfe0768-867a-4f0c-9d33-c9f0e2e947d6-maas-api-openshift-ca\") pod \"4dfe0768-867a-4f0c-9d33-c9f0e2e947d6\" (UID: \"4dfe0768-867a-4f0c-9d33-c9f0e2e947d6\") " Nov 03 03:05:54.432945 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:54.432858 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dfe0768-867a-4f0c-9d33-c9f0e2e947d6-kube-api-access-56htp\" (UniqueName: \"kubernetes.io/projected/4dfe0768-867a-4f0c-9d33-c9f0e2e947d6-kube-api-access-56htp\") pod \"4dfe0768-867a-4f0c-9d33-c9f0e2e947d6\" (UID: \"4dfe0768-867a-4f0c-9d33-c9f0e2e947d6\") " Nov 03 03:05:54.433006 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:54.432972 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4dfe0768-867a-4f0c-9d33-c9f0e2e947d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dfe0768-867a-4f0c-9d33-c9f0e2e947d6-maas-api-xks-ca\") pod \"4dfe0768-867a-4f0c-9d33-c9f0e2e947d6\" (UID: \"4dfe0768-867a-4f0c-9d33-c9f0e2e947d6\") " Nov 03 03:05:54.433093 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:54.433068 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dfe0768-867a-4f0c-9d33-c9f0e2e947d6-maas-api-openshift-ca" pod "4dfe0768-867a-4f0c-9d33-c9f0e2e947d6" (UID: "4dfe0768-867a-4f0c-9d33-c9f0e2e947d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:05:54.433449 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:54.433417 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dfe0768-867a-4f0c-9d33-c9f0e2e947d6-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:05:54.435050 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:54.435031 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dfe0768-867a-4f0c-9d33-c9f0e2e947d6-kube-api-access-56htp" pod "4dfe0768-867a-4f0c-9d33-c9f0e2e947d6" (UID: "4dfe0768-867a-4f0c-9d33-c9f0e2e947d6"). InnerVolumeSpecName "kube-api-access-56htp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:05:54.435130 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:54.435087 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dfe0768-867a-4f0c-9d33-c9f0e2e947d6-maas-api-xks-ca" pod "4dfe0768-867a-4f0c-9d33-c9f0e2e947d6" (UID: "4dfe0768-867a-4f0c-9d33-c9f0e2e947d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:05:54.534716 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:54.534674 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56htp\" (UniqueName: \"kubernetes.io/projected/4dfe0768-867a-4f0c-9d33-c9f0e2e947d6-kube-api-access-56htp\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:05:54.534716 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:54.534711 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dfe0768-867a-4f0c-9d33-c9f0e2e947d6-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:05:54.798235 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:54.798144 5605 patch_prober.go:29] interesting pod/e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body= Nov 03 03:05:54.798235 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:54.798207 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht" podUID="261be9f1-1325-4210-b5f5-18b81a6b730d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" Nov 03 03:05:55.029157 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:55.029107 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-965a07-64c88dd8c-p8xmd"] Nov 03 03:05:55.185385 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:55.183716 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26741169-4jxgn" event={"ID":"4dfe0768-867a-4f0c-9d33-c9f0e2e947d6","Type":"ContainerDied","Data":"1678e8d7b59e97ec4538b1a3465c4a39e7d2e546533f5c64deb4782f44212aaa"} Nov 03 03:05:55.185385 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:55.183783 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1678e8d7b59e97ec4538b1a3465c4a39e7d2e546533f5c64deb4782f44212aaa" Nov 03 03:05:55.185385 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:55.183860 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26741169-4jxgn" Nov 03 03:05:55.798195 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:55.798156 5605 patch_prober.go:29] interesting pod/e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body= Nov 03 03:05:55.798721 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:55.798225 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht" podUID="261be9f1-1325-4210-b5f5-18b81a6b730d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" Nov 03 03:05:55.934866 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:55.934829 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-ffba43/rate-test-model-ffba43-kserve-78c4969977-mrssx"] Nov 03 03:05:55.935314 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:55.935294 5605 cpu_manager.go:414] "containerMap: removing container" podUID="4dfe0768-867a-4f0c-9d33-c9f0e2e947d6" containerName="revoke-keys" Nov 03 03:05:55.935314 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:55.935308 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dfe0768-867a-4f0c-9d33-c9f0e2e947d6" containerName="revoke-keys" Nov 03 03:05:55.935505 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:55.935437 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dfe0768-867a-4f0c-9d33-c9f0e2e947d6" containerName="revoke-keys" Nov 03 03:05:55.939018 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:55.938997 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-ffba43/rate-test-model-ffba43-kserve-78c4969977-mrssx" Nov 03 03:05:55.989429 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:55.989168 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-ffba43\"/\"default-dockercfg-pgrbk\"" Nov 03 03:05:55.989429 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:55.989222 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-ffba43\"/\"kube-root-ca.crt\"" Nov 03 03:05:55.989429 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:55.989333 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-ffba43\"/\"rate-test-model-ffba43-kserve-self-signed-certs\"" Nov 03 03:05:55.990030 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:55.990004 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-ffba43\"/\"openshift-service-ca.crt\"" Nov 03 03:05:55.992584 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:55.992550 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-ffba43/rate-test-model-ffba43-kserve-78c4969977-mrssx"] Nov 03 03:05:56.080205 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:56.080165 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c7d95c2-f8bc-4b44-b696-4db10b9a9207-tls-certs\") pod \"rate-test-model-ffba43-kserve-78c4969977-mrssx\" (UID: \"7c7d95c2-f8bc-4b44-b696-4db10b9a9207\") " pod="ai-tenant-e2e-shared-b-w0-ffba43/rate-test-model-ffba43-kserve-78c4969977-mrssx" Nov 03 03:05:56.080492 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:56.080345 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c7d95c2-f8bc-4b44-b696-4db10b9a9207-model-cache\") pod \"rate-test-model-ffba43-kserve-78c4969977-mrssx\" (UID: \"7c7d95c2-f8bc-4b44-b696-4db10b9a9207\") " pod="ai-tenant-e2e-shared-b-w0-ffba43/rate-test-model-ffba43-kserve-78c4969977-mrssx" Nov 03 03:05:56.080574 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:56.080494 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9hgk\" (UniqueName: \"kubernetes.io/projected/7c7d95c2-f8bc-4b44-b696-4db10b9a9207-kube-api-access-x9hgk\") pod \"rate-test-model-ffba43-kserve-78c4969977-mrssx\" (UID: \"7c7d95c2-f8bc-4b44-b696-4db10b9a9207\") " pod="ai-tenant-e2e-shared-b-w0-ffba43/rate-test-model-ffba43-kserve-78c4969977-mrssx" Nov 03 03:05:56.080641 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:56.080614 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c7d95c2-f8bc-4b44-b696-4db10b9a9207-tmp-dir\") pod \"rate-test-model-ffba43-kserve-78c4969977-mrssx\" (UID: \"7c7d95c2-f8bc-4b44-b696-4db10b9a9207\") " pod="ai-tenant-e2e-shared-b-w0-ffba43/rate-test-model-ffba43-kserve-78c4969977-mrssx" Nov 03 03:05:56.080684 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:56.080671 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c7d95c2-f8bc-4b44-b696-4db10b9a9207-dshm\") pod \"rate-test-model-ffba43-kserve-78c4969977-mrssx\" (UID: \"7c7d95c2-f8bc-4b44-b696-4db10b9a9207\") " pod="ai-tenant-e2e-shared-b-w0-ffba43/rate-test-model-ffba43-kserve-78c4969977-mrssx" Nov 03 03:05:56.080822 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:56.080801 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c7d95c2-f8bc-4b44-b696-4db10b9a9207-home\") pod \"rate-test-model-ffba43-kserve-78c4969977-mrssx\" (UID: \"7c7d95c2-f8bc-4b44-b696-4db10b9a9207\") " pod="ai-tenant-e2e-shared-b-w0-ffba43/rate-test-model-ffba43-kserve-78c4969977-mrssx" Nov 03 03:05:56.182210 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:56.182169 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c7d95c2-f8bc-4b44-b696-4db10b9a9207-tmp-dir\") pod \"rate-test-model-ffba43-kserve-78c4969977-mrssx\" (UID: \"7c7d95c2-f8bc-4b44-b696-4db10b9a9207\") " pod="ai-tenant-e2e-shared-b-w0-ffba43/rate-test-model-ffba43-kserve-78c4969977-mrssx" Nov 03 03:05:56.183114 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:56.182984 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c7d95c2-f8bc-4b44-b696-4db10b9a9207-tmp-dir\") pod \"rate-test-model-ffba43-kserve-78c4969977-mrssx\" (UID: \"7c7d95c2-f8bc-4b44-b696-4db10b9a9207\") " pod="ai-tenant-e2e-shared-b-w0-ffba43/rate-test-model-ffba43-kserve-78c4969977-mrssx" Nov 03 03:05:56.183327 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:56.183307 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c7d95c2-f8bc-4b44-b696-4db10b9a9207-dshm\") pod \"rate-test-model-ffba43-kserve-78c4969977-mrssx\" (UID: \"7c7d95c2-f8bc-4b44-b696-4db10b9a9207\") " pod="ai-tenant-e2e-shared-b-w0-ffba43/rate-test-model-ffba43-kserve-78c4969977-mrssx" Nov 03 03:05:56.183576 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:56.183559 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c7d95c2-f8bc-4b44-b696-4db10b9a9207-home\") pod \"rate-test-model-ffba43-kserve-78c4969977-mrssx\" (UID: \"7c7d95c2-f8bc-4b44-b696-4db10b9a9207\") " pod="ai-tenant-e2e-shared-b-w0-ffba43/rate-test-model-ffba43-kserve-78c4969977-mrssx" Nov 03 03:05:56.183802 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:56.183786 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c7d95c2-f8bc-4b44-b696-4db10b9a9207-tls-certs\") pod \"rate-test-model-ffba43-kserve-78c4969977-mrssx\" (UID: \"7c7d95c2-f8bc-4b44-b696-4db10b9a9207\") " pod="ai-tenant-e2e-shared-b-w0-ffba43/rate-test-model-ffba43-kserve-78c4969977-mrssx" Nov 03 03:05:56.184007 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:56.183992 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c7d95c2-f8bc-4b44-b696-4db10b9a9207-model-cache\") pod \"rate-test-model-ffba43-kserve-78c4969977-mrssx\" (UID: \"7c7d95c2-f8bc-4b44-b696-4db10b9a9207\") " pod="ai-tenant-e2e-shared-b-w0-ffba43/rate-test-model-ffba43-kserve-78c4969977-mrssx" Nov 03 03:05:56.184427 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:56.183851 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c7d95c2-f8bc-4b44-b696-4db10b9a9207-home\") pod \"rate-test-model-ffba43-kserve-78c4969977-mrssx\" (UID: \"7c7d95c2-f8bc-4b44-b696-4db10b9a9207\") " pod="ai-tenant-e2e-shared-b-w0-ffba43/rate-test-model-ffba43-kserve-78c4969977-mrssx" Nov 03 03:05:56.184427 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:56.184168 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9hgk\" (UniqueName: \"kubernetes.io/projected/7c7d95c2-f8bc-4b44-b696-4db10b9a9207-kube-api-access-x9hgk\") pod \"rate-test-model-ffba43-kserve-78c4969977-mrssx\" (UID: \"7c7d95c2-f8bc-4b44-b696-4db10b9a9207\") " pod="ai-tenant-e2e-shared-b-w0-ffba43/rate-test-model-ffba43-kserve-78c4969977-mrssx" Nov 03 03:05:56.184666 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:56.184622 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c7d95c2-f8bc-4b44-b696-4db10b9a9207-model-cache\") pod \"rate-test-model-ffba43-kserve-78c4969977-mrssx\" (UID: \"7c7d95c2-f8bc-4b44-b696-4db10b9a9207\") " pod="ai-tenant-e2e-shared-b-w0-ffba43/rate-test-model-ffba43-kserve-78c4969977-mrssx" Nov 03 03:05:56.187278 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:56.185693 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c7d95c2-f8bc-4b44-b696-4db10b9a9207-dshm\") pod \"rate-test-model-ffba43-kserve-78c4969977-mrssx\" (UID: \"7c7d95c2-f8bc-4b44-b696-4db10b9a9207\") " pod="ai-tenant-e2e-shared-b-w0-ffba43/rate-test-model-ffba43-kserve-78c4969977-mrssx" Nov 03 03:05:56.187425 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:56.187397 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c7d95c2-f8bc-4b44-b696-4db10b9a9207-tls-certs\") pod \"rate-test-model-ffba43-kserve-78c4969977-mrssx\" (UID: \"7c7d95c2-f8bc-4b44-b696-4db10b9a9207\") " pod="ai-tenant-e2e-shared-b-w0-ffba43/rate-test-model-ffba43-kserve-78c4969977-mrssx" Nov 03 03:05:56.238016 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:56.237982 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9hgk\" (UniqueName: \"kubernetes.io/projected/7c7d95c2-f8bc-4b44-b696-4db10b9a9207-kube-api-access-x9hgk\") pod \"rate-test-model-ffba43-kserve-78c4969977-mrssx\" (UID: \"7c7d95c2-f8bc-4b44-b696-4db10b9a9207\") " pod="ai-tenant-e2e-shared-b-w0-ffba43/rate-test-model-ffba43-kserve-78c4969977-mrssx" Nov 03 03:05:56.251938 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:56.251905 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-ffba43/rate-test-model-ffba43-kserve-78c4969977-mrssx" Nov 03 03:05:56.693437 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:56.692235 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-ffba43/rate-test-model-ffba43-kserve-78c4969977-mrssx"] Nov 03 03:05:56.800436 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:56.798817 5605 patch_prober.go:29] interesting pod/e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body= Nov 03 03:05:56.800436 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:56.798889 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht" podUID="261be9f1-1325-4210-b5f5-18b81a6b730d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" Nov 03 03:05:57.199972 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:57.199939 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-ffba43/rate-test-model-ffba43-kserve-78c4969977-mrssx" event={"ID":"7c7d95c2-f8bc-4b44-b696-4db10b9a9207","Type":"ContainerStarted","Data":"7cb8599912c9d137947e7ec18a20a3508c31d2a68a0d0414efa2406cf45e48b0"} Nov 03 03:05:57.200186 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:57.199979 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-ffba43/rate-test-model-ffba43-kserve-78c4969977-mrssx" event={"ID":"7c7d95c2-f8bc-4b44-b696-4db10b9a9207","Type":"ContainerStarted","Data":"b11ea3e7a1cc6dadc8482564398ad0c73e8475504a752ad406396c5eef1ce183"} Nov 03 03:05:57.252394 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:57.252342 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-ffba43/rate-test-model-ffba43-kserve-78c4969977-mrssx" Nov 03 03:05:57.311203 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:57.311139 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-ffba43/rate-test-model-ffba43-kserve-78c4969977-mrssx" podStartSLOduration=2.311125182 podStartE2EDuration="2.311125182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:05:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:05:57.309204458 +0000 UTC m=+4434.568402612" watchObservedRunningTime="2026-11-03 03:05:57.311125182 +0000 UTC m=+4434.570323400" Nov 03 03:05:57.798201 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:57.798163 5605 patch_prober.go:29] interesting pod/e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body= Nov 03 03:05:57.798432 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:57.798219 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht" podUID="261be9f1-1325-4210-b5f5-18b81a6b730d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" Nov 03 03:05:58.541207 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:58.541167 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht" Nov 03 03:05:58.802771 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:58.802739 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht" Nov 03 03:05:59.212850 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:05:59.212775 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht" Nov 03 03:06:00.029982 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:06:00.029909 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-965a07-64c88dd8c-p8xmd" podUID="18215d27-a924-4c3b-8faa-47fbc0f7cb26" containerName="payload-processing" containerID="cri-o://7a97e6001fb1b40a6f87314f4ea01ffa0326ffabe457d9d3a2d9216fd5be096f" gracePeriod=25 Nov 03 03:06:00.907265 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:06:00.907220 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-965a07-64c88dd8c-p8xmd" podUID="18215d27-a924-4c3b-8faa-47fbc0f7cb26" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:06:02.207640 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:06:02.207604 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c61dc5c-pll6l"] Nov 03 03:06:02.211753 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:06:02.211729 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c61dc5c-pll6l" Nov 03 03:06:02.308541 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:06:02.308501 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:06:02.308541 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:06:02.308526 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:06:02.308541 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:06:02.308526 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 03:06:02.308810 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:06:02.308526 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:06:02.311328 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:06:02.311304 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c61dc5c-pll6l"] Nov 03 03:06:02.334217 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:06:02.334179 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99b91a2c-93fc-4fad-a448-bf5677d892f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c61dc5c-pll6l\" (UID: \"99b91a2c-93fc-4fad-a448-bf5677d892f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c61dc5c-pll6l" Nov 03 03:06:02.334396 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:06:02.334346 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99b91a2c-93fc-4fad-a448-bf5677d892f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c61dc5c-pll6l\" (UID: \"99b91a2c-93fc-4fad-a448-bf5677d892f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c61dc5c-pll6l" Nov 03 03:06:02.334472 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:06:02.334454 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n2gl\" (UniqueName: \"kubernetes.io/projected/99b91a2c-93fc-4fad-a448-bf5677d892f3-kube-api-access-9n2gl\") pod \"maas-api-revoke-sub-9c61dc5c-pll6l\" (UID: \"99b91a2c-93fc-4fad-a448-bf5677d892f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c61dc5c-pll6l" Nov 03 03:06:02.435554 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:06:02.435516 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99b91a2c-93fc-4fad-a448-bf5677d892f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c61dc5c-pll6l\" (UID: \"99b91a2c-93fc-4fad-a448-bf5677d892f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c61dc5c-pll6l" Nov 03 03:06:02.435765 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:06:02.435632 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99b91a2c-93fc-4fad-a448-bf5677d892f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c61dc5c-pll6l\" (UID: \"99b91a2c-93fc-4fad-a448-bf5677d892f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c61dc5c-pll6l" Nov 03 03:06:02.435765 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:06:02.435679 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n2gl\" (UniqueName: \"kubernetes.io/projected/99b91a2c-93fc-4fad-a448-bf5677d892f3-kube-api-access-9n2gl\") pod \"maas-api-revoke-sub-9c61dc5c-pll6l\" (UID: \"99b91a2c-93fc-4fad-a448-bf5677d892f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c61dc5c-pll6l" Nov 03 03:06:02.436442 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:06:02.436415 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99b91a2c-93fc-4fad-a448-bf5677d892f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c61dc5c-pll6l\" (UID: \"99b91a2c-93fc-4fad-a448-bf5677d892f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c61dc5c-pll6l" Nov 03 03:06:02.437919 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:06:02.437898 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99b91a2c-93fc-4fad-a448-bf5677d892f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c61dc5c-pll6l\" (UID: \"99b91a2c-93fc-4fad-a448-bf5677d892f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c61dc5c-pll6l" Nov 03 03:06:02.489158 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:06:02.489082 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n2gl\" (UniqueName: \"kubernetes.io/projected/99b91a2c-93fc-4fad-a448-bf5677d892f3-kube-api-access-9n2gl\") pod \"maas-api-revoke-sub-9c61dc5c-pll6l\" (UID: \"99b91a2c-93fc-4fad-a448-bf5677d892f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c61dc5c-pll6l" Nov 03 03:06:02.526010 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:06:02.525974 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c61dc5c-pll6l" Nov 03 03:06:02.895200 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:06:02.895165 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c61dc5c-pll6l"] Nov 03 03:06:02.902294 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:06:02.902265 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99b91a2c_93fc_4fad_a448_bf5677d892f3.slice/crio-318f6af9a5ef02069a80e3c3770ace08fba0475a0dc78f9eb48e2caede8acb1e WatchSource:0}: Error finding container 318f6af9a5ef02069a80e3c3770ace08fba0475a0dc78f9eb48e2caede8acb1e: Status 404 returned error can't find the container with id 318f6af9a5ef02069a80e3c3770ace08fba0475a0dc78f9eb48e2caede8acb1e Nov 03 03:06:03.236273 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:06:03.236242 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c61dc5c-pll6l" event={"ID":"99b91a2c-93fc-4fad-a448-bf5677d892f3","Type":"ContainerStarted","Data":"f6ec3b850272a6b04e204c56f410b3c6667b307b5dfecd180ed8404032801fb6"} Nov 03 03:06:03.236706 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:06:03.236293 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c61dc5c-pll6l" event={"ID":"99b91a2c-93fc-4fad-a448-bf5677d892f3","Type":"ContainerStarted","Data":"318f6af9a5ef02069a80e3c3770ace08fba0475a0dc78f9eb48e2caede8acb1e"} Nov 03 03:06:03.344506 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:06:03.344452 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c61dc5c-pll6l" podStartSLOduration=1.344437386 podStartE2EDuration="1.344437386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:06:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:06:03.342402869 +0000 UTC m=+4440.601601024" watchObservedRunningTime="2026-11-03 03:06:03.344437386 +0000 UTC m=+4440.603635540" Nov 03 03:06:04.245179 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:06:04.245136 5605 generic.go:356] "Generic (PLEG): container finished" podID="99b91a2c-93fc-4fad-a448-bf5677d892f3" containerID="f6ec3b850272a6b04e204c56f410b3c6667b307b5dfecd180ed8404032801fb6" exitCode=0 Nov 03 03:06:04.245634 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:06:04.245223 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c61dc5c-pll6l" event={"ID":"99b91a2c-93fc-4fad-a448-bf5677d892f3","Type":"ContainerDied","Data":"f6ec3b850272a6b04e204c56f410b3c6667b307b5dfecd180ed8404032801fb6"} Nov 03 03:06:05.117418 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:06:05.117382 5605 scope.go:122] "RemoveContainer" containerID="9b6a81b74b31b0ea401bef5716ed8b8ac5b881c3d22849d24022ea63595fd8fd" Nov 03 03:06:05.408693 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:06:05.408656 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c61dc5c-pll6l" Nov 03 03:06:05.576158 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:06:05.576127 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99b91a2c-93fc-4fad-a448-bf5677d892f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99b91a2c-93fc-4fad-a448-bf5677d892f3-maas-api-xks-ca\") pod \"99b91a2c-93fc-4fad-a448-bf5677d892f3\" (UID: \"99b91a2c-93fc-4fad-a448-bf5677d892f3\") " Nov 03 03:06:05.576342 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:06:05.576264 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99b91a2c-93fc-4fad-a448-bf5677d892f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99b91a2c-93fc-4fad-a448-bf5677d892f3-maas-api-openshift-ca\") pod \"99b91a2c-93fc-4fad-a448-bf5677d892f3\" (UID: \"99b91a2c-93fc-4fad-a448-bf5677d892f3\") " Nov 03 03:06:05.576342 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:06:05.576290 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99b91a2c-93fc-4fad-a448-bf5677d892f3-kube-api-access-9n2gl\" (UniqueName: \"kubernetes.io/projected/99b91a2c-93fc-4fad-a448-bf5677d892f3-kube-api-access-9n2gl\") pod \"99b91a2c-93fc-4fad-a448-bf5677d892f3\" (UID: \"99b91a2c-93fc-4fad-a448-bf5677d892f3\") " Nov 03 03:06:05.576764 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:06:05.576730 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99b91a2c-93fc-4fad-a448-bf5677d892f3-maas-api-openshift-ca" pod "99b91a2c-93fc-4fad-a448-bf5677d892f3" (UID: "99b91a2c-93fc-4fad-a448-bf5677d892f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:06:05.578337 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:06:05.578313 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99b91a2c-93fc-4fad-a448-bf5677d892f3-maas-api-xks-ca" pod "99b91a2c-93fc-4fad-a448-bf5677d892f3" (UID: "99b91a2c-93fc-4fad-a448-bf5677d892f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:06:05.578644 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:06:05.578624 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99b91a2c-93fc-4fad-a448-bf5677d892f3-kube-api-access-9n2gl" pod "99b91a2c-93fc-4fad-a448-bf5677d892f3" (UID: "99b91a2c-93fc-4fad-a448-bf5677d892f3"). InnerVolumeSpecName "kube-api-access-9n2gl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:06:05.678525 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:06:05.678444 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99b91a2c-93fc-4fad-a448-bf5677d892f3-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:06:05.678525 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:06:05.678478 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9n2gl\" (UniqueName: \"kubernetes.io/projected/99b91a2c-93fc-4fad-a448-bf5677d892f3-kube-api-access-9n2gl\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:06:05.678525 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:06:05.678495 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99b91a2c-93fc-4fad-a448-bf5677d892f3-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:06:06.252135 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:06:06.252100 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-ffba43/rate-test-model-ffba43-kserve-78c4969977-mrssx" Nov 03 03:06:06.259305 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:06:06.259273 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c61dc5c-pll6l" Nov 03 03:06:06.259493 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:06:06.259302 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c61dc5c-pll6l" event={"ID":"99b91a2c-93fc-4fad-a448-bf5677d892f3","Type":"ContainerDied","Data":"318f6af9a5ef02069a80e3c3770ace08fba0475a0dc78f9eb48e2caede8acb1e"} Nov 03 03:06:06.259493 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:06:06.259350 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="318f6af9a5ef02069a80e3c3770ace08fba0475a0dc78f9eb48e2caede8acb1e" Nov 03 03:06:06.265524 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:06:06.265504 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-ffba43/rate-test-model-ffba43-kserve-78c4969977-mrssx" Nov 03 03:06:07.263872 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:06:07.263842 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-ffba43/rate-test-model-ffba43-kserve-78c4969977-mrssx" Nov 03 03:06:09.044096 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:06:09.044060 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0ffb465-9zhbg"] Nov 03 03:06:09.065759 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:06:09.065724 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0ffb465-9zhbg"] Nov 03 03:06:10.850099 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:06:10.850064 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2951186-428b-4b1b-9e83-2dfbcd906e82" path="/var/lib/kubelet/pods/f2951186-428b-4b1b-9e83-2dfbcd906e82/volumes" Nov 03 03:06:10.907225 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:06:10.907191 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-965a07-64c88dd8c-p8xmd" podUID="18215d27-a924-4c3b-8faa-47fbc0f7cb26" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:06:14.905653 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:06:14.905600 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:06:20.284479 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:06:20.284452 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-965a07-64c88dd8c-p8xmd" Nov 03 03:06:20.357372 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:06:20.357331 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18215d27-a924-4c3b-8faa-47fbc0f7cb26-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18215d27-a924-4c3b-8faa-47fbc0f7cb26-plugins-config-volume\") pod \"18215d27-a924-4c3b-8faa-47fbc0f7cb26\" (UID: \"18215d27-a924-4c3b-8faa-47fbc0f7cb26\") " Nov 03 03:06:20.357559 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:06:20.357379 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18215d27-a924-4c3b-8faa-47fbc0f7cb26-kube-api-access-qdjw4\" (UniqueName: \"kubernetes.io/projected/18215d27-a924-4c3b-8faa-47fbc0f7cb26-kube-api-access-qdjw4\") pod \"18215d27-a924-4c3b-8faa-47fbc0f7cb26\" (UID: \"18215d27-a924-4c3b-8faa-47fbc0f7cb26\") " Nov 03 03:06:20.357729 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:06:20.357706 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18215d27-a924-4c3b-8faa-47fbc0f7cb26-plugins-config-volume" pod "18215d27-a924-4c3b-8faa-47fbc0f7cb26" (UID: "18215d27-a924-4c3b-8faa-47fbc0f7cb26"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:06:20.359537 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:06:20.359503 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18215d27-a924-4c3b-8faa-47fbc0f7cb26-kube-api-access-qdjw4" pod "18215d27-a924-4c3b-8faa-47fbc0f7cb26" (UID: "18215d27-a924-4c3b-8faa-47fbc0f7cb26"). InnerVolumeSpecName "kube-api-access-qdjw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:06:20.394787 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:06:20.394751 5605 generic.go:356] "Generic (PLEG): container finished" podID="18215d27-a924-4c3b-8faa-47fbc0f7cb26" containerID="7a97e6001fb1b40a6f87314f4ea01ffa0326ffabe457d9d3a2d9216fd5be096f" exitCode=0 Nov 03 03:06:20.395008 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:06:20.394813 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-965a07-64c88dd8c-p8xmd" Nov 03 03:06:20.395008 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:06:20.394815 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-965a07-64c88dd8c-p8xmd" event={"ID":"18215d27-a924-4c3b-8faa-47fbc0f7cb26","Type":"ContainerDied","Data":"7a97e6001fb1b40a6f87314f4ea01ffa0326ffabe457d9d3a2d9216fd5be096f"} Nov 03 03:06:20.395008 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:06:20.394911 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-965a07-64c88dd8c-p8xmd" event={"ID":"18215d27-a924-4c3b-8faa-47fbc0f7cb26","Type":"ContainerDied","Data":"141b652b50032cbef45451f5311534a14bc9f35ef0e8299962351c45f7ee8c8c"} Nov 03 03:06:20.395008 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:06:20.394937 5605 scope.go:122] "RemoveContainer" containerID="7a97e6001fb1b40a6f87314f4ea01ffa0326ffabe457d9d3a2d9216fd5be096f" Nov 03 03:06:20.406820 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:06:20.406804 5605 scope.go:122] "RemoveContainer" containerID="7a97e6001fb1b40a6f87314f4ea01ffa0326ffabe457d9d3a2d9216fd5be096f" Nov 03 03:06:20.407100 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:06:20.407079 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a97e6001fb1b40a6f87314f4ea01ffa0326ffabe457d9d3a2d9216fd5be096f\": container with ID starting with 7a97e6001fb1b40a6f87314f4ea01ffa0326ffabe457d9d3a2d9216fd5be096f not found: ID does not exist" containerID="7a97e6001fb1b40a6f87314f4ea01ffa0326ffabe457d9d3a2d9216fd5be096f" Nov 03 03:06:20.407185 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:06:20.407108 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a97e6001fb1b40a6f87314f4ea01ffa0326ffabe457d9d3a2d9216fd5be096f"} err="failed to get container status \"7a97e6001fb1b40a6f87314f4ea01ffa0326ffabe457d9d3a2d9216fd5be096f\": rpc error: code = NotFound desc = could not find container \"7a97e6001fb1b40a6f87314f4ea01ffa0326ffabe457d9d3a2d9216fd5be096f\": container with ID starting with 7a97e6001fb1b40a6f87314f4ea01ffa0326ffabe457d9d3a2d9216fd5be096f not found: ID does not exist" Nov 03 03:06:20.459224 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:06:20.459197 5605 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18215d27-a924-4c3b-8faa-47fbc0f7cb26-plugins-config-volume\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:06:20.459224 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:06:20.459223 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdjw4\" (UniqueName: \"kubernetes.io/projected/18215d27-a924-4c3b-8faa-47fbc0f7cb26-kube-api-access-qdjw4\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:06:20.550635 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:06:20.550591 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-965a07-64c88dd8c-p8xmd"] Nov 03 03:06:20.555933 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:06:20.555898 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-965a07-64c88dd8c-p8xmd"] Nov 03 03:06:20.850541 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:06:20.850502 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18215d27-a924-4c3b-8faa-47fbc0f7cb26" path="/var/lib/kubelet/pods/18215d27-a924-4c3b-8faa-47fbc0f7cb26/volumes" Nov 03 03:06:23.915639 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:06:23.915589 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:06:50.502242 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:06:50.502210 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-f735f4/rate-test-model-f735f4-kserve-6c67f66677-2sqk5"] Nov 03 03:06:53.917627 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:06:53.917581 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:07:05.141337 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:05.141304 5605 scope.go:122] "RemoveContainer" containerID="286f6f3afe5930f0de710349721f7c08c2b8d112ee3b8945e4cd3a0ba6ff84db" Nov 03 03:07:05.522859 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:05.522764 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-f735f4/rate-test-model-f735f4-kserve-6c67f66677-2sqk5" podUID="2e02ba51-abe8-472f-84f8-efa53f1d3b40" containerName="main" containerID="cri-o://47e3e0ffca307aaaf59433e70b6ccb1249f01d46320332d9c697acc3c6c944c0" gracePeriod=45 Nov 03 03:07:05.658923 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:05.658881 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-f735f4/rate-test-model-f735f4-kserve-6c67f66677-2sqk5" podUID="2e02ba51-abe8-472f-84f8-efa53f1d3b40" containerName="main" probeResult="failure" output="Get \"https://10.134.0.251:8000/ready\": dial tcp 10.134.0.251:8000: connect: connection refused" Nov 03 03:07:05.701116 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:05.701086 5605 generic.go:356] "Generic (PLEG): container finished" podID="2e02ba51-abe8-472f-84f8-efa53f1d3b40" containerID="47e3e0ffca307aaaf59433e70b6ccb1249f01d46320332d9c697acc3c6c944c0" exitCode=0 Nov 03 03:07:05.701249 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:05.701155 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-f735f4/rate-test-model-f735f4-kserve-6c67f66677-2sqk5" event={"ID":"2e02ba51-abe8-472f-84f8-efa53f1d3b40","Type":"ContainerDied","Data":"47e3e0ffca307aaaf59433e70b6ccb1249f01d46320332d9c697acc3c6c944c0"} Nov 03 03:07:05.798641 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:05.798616 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-f735f4/rate-test-model-f735f4-kserve-6c67f66677-2sqk5" Nov 03 03:07:05.925081 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:05.925046 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e02ba51-abe8-472f-84f8-efa53f1d3b40-home\" (UniqueName: \"kubernetes.io/empty-dir/2e02ba51-abe8-472f-84f8-efa53f1d3b40-home\") pod \"2e02ba51-abe8-472f-84f8-efa53f1d3b40\" (UID: \"2e02ba51-abe8-472f-84f8-efa53f1d3b40\") " Nov 03 03:07:05.925254 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:05.925150 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e02ba51-abe8-472f-84f8-efa53f1d3b40-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e02ba51-abe8-472f-84f8-efa53f1d3b40-dshm\") pod \"2e02ba51-abe8-472f-84f8-efa53f1d3b40\" (UID: \"2e02ba51-abe8-472f-84f8-efa53f1d3b40\") " Nov 03 03:07:05.925254 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:05.925200 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e02ba51-abe8-472f-84f8-efa53f1d3b40-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e02ba51-abe8-472f-84f8-efa53f1d3b40-tls-certs\") pod \"2e02ba51-abe8-472f-84f8-efa53f1d3b40\" (UID: \"2e02ba51-abe8-472f-84f8-efa53f1d3b40\") " Nov 03 03:07:05.925254 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:05.925247 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e02ba51-abe8-472f-84f8-efa53f1d3b40-kube-api-access-4rwlw\" (UniqueName: \"kubernetes.io/projected/2e02ba51-abe8-472f-84f8-efa53f1d3b40-kube-api-access-4rwlw\") pod \"2e02ba51-abe8-472f-84f8-efa53f1d3b40\" (UID: \"2e02ba51-abe8-472f-84f8-efa53f1d3b40\") " Nov 03 03:07:05.925915 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:05.925300 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e02ba51-abe8-472f-84f8-efa53f1d3b40-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e02ba51-abe8-472f-84f8-efa53f1d3b40-tmp-dir\") pod \"2e02ba51-abe8-472f-84f8-efa53f1d3b40\" (UID: \"2e02ba51-abe8-472f-84f8-efa53f1d3b40\") " Nov 03 03:07:05.925915 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:05.925306 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e02ba51-abe8-472f-84f8-efa53f1d3b40-home" pod "2e02ba51-abe8-472f-84f8-efa53f1d3b40" (UID: "2e02ba51-abe8-472f-84f8-efa53f1d3b40"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:07:05.925915 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:05.925339 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e02ba51-abe8-472f-84f8-efa53f1d3b40-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e02ba51-abe8-472f-84f8-efa53f1d3b40-model-cache\") pod \"2e02ba51-abe8-472f-84f8-efa53f1d3b40\" (UID: \"2e02ba51-abe8-472f-84f8-efa53f1d3b40\") " Nov 03 03:07:05.925915 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:05.925550 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e02ba51-abe8-472f-84f8-efa53f1d3b40-tmp-dir" pod "2e02ba51-abe8-472f-84f8-efa53f1d3b40" (UID: "2e02ba51-abe8-472f-84f8-efa53f1d3b40"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:07:05.925915 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:05.925814 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e02ba51-abe8-472f-84f8-efa53f1d3b40-model-cache" pod "2e02ba51-abe8-472f-84f8-efa53f1d3b40" (UID: "2e02ba51-abe8-472f-84f8-efa53f1d3b40"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:07:05.926197 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:05.926080 5605 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e02ba51-abe8-472f-84f8-efa53f1d3b40-home\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:07:05.926197 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:05.926102 5605 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e02ba51-abe8-472f-84f8-efa53f1d3b40-tmp-dir\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:07:05.926197 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:05.926116 5605 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e02ba51-abe8-472f-84f8-efa53f1d3b40-model-cache\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:07:05.927639 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:05.927611 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e02ba51-abe8-472f-84f8-efa53f1d3b40-tls-certs" pod "2e02ba51-abe8-472f-84f8-efa53f1d3b40" (UID: "2e02ba51-abe8-472f-84f8-efa53f1d3b40"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:07:05.927788 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:05.927720 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e02ba51-abe8-472f-84f8-efa53f1d3b40-kube-api-access-4rwlw" pod "2e02ba51-abe8-472f-84f8-efa53f1d3b40" (UID: "2e02ba51-abe8-472f-84f8-efa53f1d3b40"). InnerVolumeSpecName "kube-api-access-4rwlw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:07:05.927839 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:05.927789 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e02ba51-abe8-472f-84f8-efa53f1d3b40-dshm" pod "2e02ba51-abe8-472f-84f8-efa53f1d3b40" (UID: "2e02ba51-abe8-472f-84f8-efa53f1d3b40"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:07:06.027487 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:06.027455 5605 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e02ba51-abe8-472f-84f8-efa53f1d3b40-dshm\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:07:06.027487 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:06.027482 5605 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e02ba51-abe8-472f-84f8-efa53f1d3b40-tls-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:07:06.027487 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:06.027492 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rwlw\" (UniqueName: \"kubernetes.io/projected/2e02ba51-abe8-472f-84f8-efa53f1d3b40-kube-api-access-4rwlw\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:07:06.708428 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:06.708388 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-f735f4/rate-test-model-f735f4-kserve-6c67f66677-2sqk5" event={"ID":"2e02ba51-abe8-472f-84f8-efa53f1d3b40","Type":"ContainerDied","Data":"0e2ebbdb5289cab555571a23513f244a667cffa4bdd82d6ee1af0d53b4fe19c9"} Nov 03 03:07:06.708428 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:06.708431 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-f735f4/rate-test-model-f735f4-kserve-6c67f66677-2sqk5" Nov 03 03:07:06.708971 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:06.708443 5605 scope.go:122] "RemoveContainer" containerID="47e3e0ffca307aaaf59433e70b6ccb1249f01d46320332d9c697acc3c6c944c0" Nov 03 03:07:06.862137 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:06.862104 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-f735f4/rate-test-model-f735f4-kserve-6c67f66677-2sqk5"] Nov 03 03:07:06.908677 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:06.908643 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-f735f4/rate-test-model-f735f4-kserve-6c67f66677-2sqk5"] Nov 03 03:07:08.850423 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:08.850384 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e02ba51-abe8-472f-84f8-efa53f1d3b40" path="/var/lib/kubelet/pods/2e02ba51-abe8-472f-84f8-efa53f1d3b40/volumes" Nov 03 03:07:10.851962 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:10.851880 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:07:13.007220 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:13.007175 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-957b33/rate-test-model-957b33-kserve-7bf748c8f-jjx87"] Nov 03 03:07:23.916496 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:07:23.916458 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:07:28.028083 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:28.028042 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-957b33/rate-test-model-957b33-kserve-7bf748c8f-jjx87" podUID="58aaea68-aa14-4041-a2ee-c8dbb1c19b55" containerName="main" containerID="cri-o://14503a3a61315eeee31329d449c65cfdd3884ee7740d1976c8f79cfb653b2a87" gracePeriod=45 Nov 03 03:07:28.081282 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:28.081249 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-957b33/rate-test-model-957b33-kserve-7bf748c8f-jjx87" podUID="58aaea68-aa14-4041-a2ee-c8dbb1c19b55" containerName="main" probeResult="failure" output="Get \"https://10.134.0.254:8000/ready\": dial tcp 10.134.0.254:8000: connect: connection refused" Nov 03 03:07:28.293850 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:28.293792 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-957b33/rate-test-model-957b33-kserve-7bf748c8f-jjx87" Nov 03 03:07:28.416816 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:28.416788 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58aaea68-aa14-4041-a2ee-c8dbb1c19b55-home\" (UniqueName: \"kubernetes.io/empty-dir/58aaea68-aa14-4041-a2ee-c8dbb1c19b55-home\") pod \"58aaea68-aa14-4041-a2ee-c8dbb1c19b55\" (UID: \"58aaea68-aa14-4041-a2ee-c8dbb1c19b55\") " Nov 03 03:07:28.416982 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:28.416884 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58aaea68-aa14-4041-a2ee-c8dbb1c19b55-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58aaea68-aa14-4041-a2ee-c8dbb1c19b55-model-cache\") pod \"58aaea68-aa14-4041-a2ee-c8dbb1c19b55\" (UID: \"58aaea68-aa14-4041-a2ee-c8dbb1c19b55\") " Nov 03 03:07:28.416982 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:28.416926 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58aaea68-aa14-4041-a2ee-c8dbb1c19b55-tls-certs\" (UniqueName: \"kubernetes.io/secret/58aaea68-aa14-4041-a2ee-c8dbb1c19b55-tls-certs\") pod \"58aaea68-aa14-4041-a2ee-c8dbb1c19b55\" (UID: \"58aaea68-aa14-4041-a2ee-c8dbb1c19b55\") " Nov 03 03:07:28.416982 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:28.416961 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58aaea68-aa14-4041-a2ee-c8dbb1c19b55-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58aaea68-aa14-4041-a2ee-c8dbb1c19b55-tmp-dir\") pod \"58aaea68-aa14-4041-a2ee-c8dbb1c19b55\" (UID: \"58aaea68-aa14-4041-a2ee-c8dbb1c19b55\") " Nov 03 03:07:28.417128 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:28.417072 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58aaea68-aa14-4041-a2ee-c8dbb1c19b55-home" pod "58aaea68-aa14-4041-a2ee-c8dbb1c19b55" (UID: "58aaea68-aa14-4041-a2ee-c8dbb1c19b55"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:07:28.417195 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:28.417142 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58aaea68-aa14-4041-a2ee-c8dbb1c19b55-model-cache" pod "58aaea68-aa14-4041-a2ee-c8dbb1c19b55" (UID: "58aaea68-aa14-4041-a2ee-c8dbb1c19b55"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:07:28.417195 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:28.417152 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58aaea68-aa14-4041-a2ee-c8dbb1c19b55-kube-api-access-q6pp7\" (UniqueName: \"kubernetes.io/projected/58aaea68-aa14-4041-a2ee-c8dbb1c19b55-kube-api-access-q6pp7\") pod \"58aaea68-aa14-4041-a2ee-c8dbb1c19b55\" (UID: \"58aaea68-aa14-4041-a2ee-c8dbb1c19b55\") " Nov 03 03:07:28.417313 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:28.417207 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58aaea68-aa14-4041-a2ee-c8dbb1c19b55-tmp-dir" pod "58aaea68-aa14-4041-a2ee-c8dbb1c19b55" (UID: "58aaea68-aa14-4041-a2ee-c8dbb1c19b55"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:07:28.417313 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:28.417215 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58aaea68-aa14-4041-a2ee-c8dbb1c19b55-dshm\" (UniqueName: \"kubernetes.io/empty-dir/58aaea68-aa14-4041-a2ee-c8dbb1c19b55-dshm\") pod \"58aaea68-aa14-4041-a2ee-c8dbb1c19b55\" (UID: \"58aaea68-aa14-4041-a2ee-c8dbb1c19b55\") " Nov 03 03:07:28.417826 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:28.417802 5605 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58aaea68-aa14-4041-a2ee-c8dbb1c19b55-home\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:07:28.417947 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:28.417829 5605 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58aaea68-aa14-4041-a2ee-c8dbb1c19b55-model-cache\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:07:28.417947 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:28.417848 5605 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58aaea68-aa14-4041-a2ee-c8dbb1c19b55-tmp-dir\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:07:28.419077 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:28.419053 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58aaea68-aa14-4041-a2ee-c8dbb1c19b55-tls-certs" pod "58aaea68-aa14-4041-a2ee-c8dbb1c19b55" (UID: "58aaea68-aa14-4041-a2ee-c8dbb1c19b55"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:07:28.419187 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:28.419133 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58aaea68-aa14-4041-a2ee-c8dbb1c19b55-kube-api-access-q6pp7" pod "58aaea68-aa14-4041-a2ee-c8dbb1c19b55" (UID: "58aaea68-aa14-4041-a2ee-c8dbb1c19b55"). InnerVolumeSpecName "kube-api-access-q6pp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:07:28.419262 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:28.419196 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58aaea68-aa14-4041-a2ee-c8dbb1c19b55-dshm" pod "58aaea68-aa14-4041-a2ee-c8dbb1c19b55" (UID: "58aaea68-aa14-4041-a2ee-c8dbb1c19b55"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:07:28.519489 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:28.519463 5605 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58aaea68-aa14-4041-a2ee-c8dbb1c19b55-tls-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:07:28.519489 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:28.519489 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6pp7\" (UniqueName: \"kubernetes.io/projected/58aaea68-aa14-4041-a2ee-c8dbb1c19b55-kube-api-access-q6pp7\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:07:28.519639 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:28.519500 5605 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58aaea68-aa14-4041-a2ee-c8dbb1c19b55-dshm\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:07:28.856111 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:28.856079 5605 generic.go:356] "Generic (PLEG): container finished" podID="58aaea68-aa14-4041-a2ee-c8dbb1c19b55" containerID="14503a3a61315eeee31329d449c65cfdd3884ee7740d1976c8f79cfb653b2a87" exitCode=0 Nov 03 03:07:28.856281 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:28.856126 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-957b33/rate-test-model-957b33-kserve-7bf748c8f-jjx87" Nov 03 03:07:28.856281 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:28.856167 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-957b33/rate-test-model-957b33-kserve-7bf748c8f-jjx87" event={"ID":"58aaea68-aa14-4041-a2ee-c8dbb1c19b55","Type":"ContainerDied","Data":"14503a3a61315eeee31329d449c65cfdd3884ee7740d1976c8f79cfb653b2a87"} Nov 03 03:07:28.856281 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:28.856205 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-957b33/rate-test-model-957b33-kserve-7bf748c8f-jjx87" event={"ID":"58aaea68-aa14-4041-a2ee-c8dbb1c19b55","Type":"ContainerDied","Data":"fc38c24ecaf19e0882b8201099544b7ecf7d6daa7310d94e1df3b20a48c7c4f8"} Nov 03 03:07:28.856281 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:28.856222 5605 scope.go:122] "RemoveContainer" containerID="14503a3a61315eeee31329d449c65cfdd3884ee7740d1976c8f79cfb653b2a87" Nov 03 03:07:28.865944 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:28.865925 5605 scope.go:122] "RemoveContainer" containerID="14503a3a61315eeee31329d449c65cfdd3884ee7740d1976c8f79cfb653b2a87" Nov 03 03:07:28.866212 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:07:28.866190 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14503a3a61315eeee31329d449c65cfdd3884ee7740d1976c8f79cfb653b2a87\": container with ID starting with 14503a3a61315eeee31329d449c65cfdd3884ee7740d1976c8f79cfb653b2a87 not found: ID does not exist" containerID="14503a3a61315eeee31329d449c65cfdd3884ee7740d1976c8f79cfb653b2a87" Nov 03 03:07:28.866276 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:28.866217 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14503a3a61315eeee31329d449c65cfdd3884ee7740d1976c8f79cfb653b2a87"} err="failed to get container status \"14503a3a61315eeee31329d449c65cfdd3884ee7740d1976c8f79cfb653b2a87\": rpc error: code = NotFound desc = could not find container \"14503a3a61315eeee31329d449c65cfdd3884ee7740d1976c8f79cfb653b2a87\": container with ID starting with 14503a3a61315eeee31329d449c65cfdd3884ee7740d1976c8f79cfb653b2a87 not found: ID does not exist" Nov 03 03:07:29.009461 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:29.009432 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-957b33/rate-test-model-957b33-kserve-7bf748c8f-jjx87"] Nov 03 03:07:29.013559 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:29.013537 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-957b33/rate-test-model-957b33-kserve-7bf748c8f-jjx87"] Nov 03 03:07:30.850538 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:30.850502 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58aaea68-aa14-4041-a2ee-c8dbb1c19b55" path="/var/lib/kubelet/pods/58aaea68-aa14-4041-a2ee-c8dbb1c19b55/volumes" Nov 03 03:07:33.086872 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:33.086833 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4477d0a8-gp62c"] Nov 03 03:07:33.089824 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:33.089793 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4477d0a8-gp62c"] Nov 03 03:07:34.851802 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:34.851757 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="204dc46a-828c-42e4-bf24-36d54a730ff0" path="/var/lib/kubelet/pods/204dc46a-828c-42e4-bf24-36d54a730ff0/volumes" Nov 03 03:07:53.928975 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:07:53.928933 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:07:56.180105 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:56.180068 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht"] Nov 03 03:07:56.180636 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:56.180305 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht" podUID="261be9f1-1325-4210-b5f5-18b81a6b730d" containerName="istio-proxy" containerID="cri-o://f89e2e82cb5c8624ae74f4b6f52a6676ecb08bb16e5f1df670d69764f65d1e00" gracePeriod=30 Nov 03 03:07:57.263557 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:57.263520 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1243713b-skfvd"] Nov 03 03:07:57.264287 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:57.264269 5605 cpu_manager.go:414] "containerMap: removing container" podUID="2e02ba51-abe8-472f-84f8-efa53f1d3b40" containerName="main" Nov 03 03:07:57.264337 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:57.264290 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e02ba51-abe8-472f-84f8-efa53f1d3b40" containerName="main" Nov 03 03:07:57.264337 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:57.264335 5605 cpu_manager.go:414] "containerMap: removing container" podUID="58aaea68-aa14-4041-a2ee-c8dbb1c19b55" containerName="main" Nov 03 03:07:57.264446 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:57.264344 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58aaea68-aa14-4041-a2ee-c8dbb1c19b55" containerName="main" Nov 03 03:07:57.264446 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:57.264391 5605 cpu_manager.go:414] "containerMap: removing container" podUID="18215d27-a924-4c3b-8faa-47fbc0f7cb26" containerName="payload-processing" Nov 03 03:07:57.264446 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:57.264402 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18215d27-a924-4c3b-8faa-47fbc0f7cb26" containerName="payload-processing" Nov 03 03:07:57.264446 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:57.264422 5605 cpu_manager.go:414] "containerMap: removing container" podUID="99b91a2c-93fc-4fad-a448-bf5677d892f3" containerName="revoke-keys" Nov 03 03:07:57.264446 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:57.264428 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99b91a2c-93fc-4fad-a448-bf5677d892f3" containerName="revoke-keys" Nov 03 03:07:57.264607 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:57.264534 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="58aaea68-aa14-4041-a2ee-c8dbb1c19b55" containerName="main" Nov 03 03:07:57.264607 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:57.264548 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="18215d27-a924-4c3b-8faa-47fbc0f7cb26" containerName="payload-processing" Nov 03 03:07:57.264607 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:57.264560 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="99b91a2c-93fc-4fad-a448-bf5677d892f3" containerName="revoke-keys" Nov 03 03:07:57.264607 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:57.264574 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e02ba51-abe8-472f-84f8-efa53f1d3b40" containerName="main" Nov 03 03:07:57.267908 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:57.267888 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1243713b-skfvd" Nov 03 03:07:57.318223 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:57.318187 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:07:57.318460 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:57.318282 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:07:57.345174 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:57.345140 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v6bn\" (UniqueName: \"kubernetes.io/projected/1439cf8d-1d92-42ba-80ca-eb5c781fa883-kube-api-access-5v6bn\") pod \"maas-api-revoke-sub-1243713b-skfvd\" (UID: \"1439cf8d-1d92-42ba-80ca-eb5c781fa883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1243713b-skfvd" Nov 03 03:07:57.345393 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:57.345266 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1439cf8d-1d92-42ba-80ca-eb5c781fa883-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1243713b-skfvd\" (UID: \"1439cf8d-1d92-42ba-80ca-eb5c781fa883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1243713b-skfvd" Nov 03 03:07:57.345393 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:57.345311 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1439cf8d-1d92-42ba-80ca-eb5c781fa883-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1243713b-skfvd\" (UID: \"1439cf8d-1d92-42ba-80ca-eb5c781fa883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1243713b-skfvd" Nov 03 03:07:57.364665 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:57.364633 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:07:57.364865 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:57.364689 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 03:07:57.367099 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:57.367080 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1243713b-skfvd"] Nov 03 03:07:57.446911 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:57.446874 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v6bn\" (UniqueName: \"kubernetes.io/projected/1439cf8d-1d92-42ba-80ca-eb5c781fa883-kube-api-access-5v6bn\") pod \"maas-api-revoke-sub-1243713b-skfvd\" (UID: \"1439cf8d-1d92-42ba-80ca-eb5c781fa883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1243713b-skfvd" Nov 03 03:07:57.447114 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:57.447014 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1439cf8d-1d92-42ba-80ca-eb5c781fa883-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1243713b-skfvd\" (UID: \"1439cf8d-1d92-42ba-80ca-eb5c781fa883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1243713b-skfvd" Nov 03 03:07:57.447114 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:57.447055 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1439cf8d-1d92-42ba-80ca-eb5c781fa883-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1243713b-skfvd\" (UID: \"1439cf8d-1d92-42ba-80ca-eb5c781fa883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1243713b-skfvd" Nov 03 03:07:57.447801 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:57.447774 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1439cf8d-1d92-42ba-80ca-eb5c781fa883-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1243713b-skfvd\" (UID: \"1439cf8d-1d92-42ba-80ca-eb5c781fa883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1243713b-skfvd" Nov 03 03:07:57.449616 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:57.449592 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1439cf8d-1d92-42ba-80ca-eb5c781fa883-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1243713b-skfvd\" (UID: \"1439cf8d-1d92-42ba-80ca-eb5c781fa883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1243713b-skfvd" Nov 03 03:07:57.500913 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:57.500879 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v6bn\" (UniqueName: \"kubernetes.io/projected/1439cf8d-1d92-42ba-80ca-eb5c781fa883-kube-api-access-5v6bn\") pod \"maas-api-revoke-sub-1243713b-skfvd\" (UID: \"1439cf8d-1d92-42ba-80ca-eb5c781fa883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1243713b-skfvd" Nov 03 03:07:57.581313 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:57.581278 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1243713b-skfvd" Nov 03 03:07:57.955141 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:57.955053 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1243713b-skfvd"] Nov 03 03:07:57.961919 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:07:57.961893 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1439cf8d_1d92_42ba_80ca_eb5c781fa883.slice/crio-d8dd3c3425c49b6040c48faf8c41ede555c2f565d0b76e959bb3474f9ad1266f WatchSource:0}: Error finding container d8dd3c3425c49b6040c48faf8c41ede555c2f565d0b76e959bb3474f9ad1266f: Status 404 returned error can't find the container with id d8dd3c3425c49b6040c48faf8c41ede555c2f565d0b76e959bb3474f9ad1266f Nov 03 03:07:58.025042 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:58.025000 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1243713b-skfvd" event={"ID":"1439cf8d-1d92-42ba-80ca-eb5c781fa883","Type":"ContainerStarted","Data":"d8dd3c3425c49b6040c48faf8c41ede555c2f565d0b76e959bb3474f9ad1266f"} Nov 03 03:07:59.033478 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:59.033441 5605 generic.go:356] "Generic (PLEG): container finished" podID="1439cf8d-1d92-42ba-80ca-eb5c781fa883" containerID="849a340013366e838d2ac2b3a3bebeb1186590dffe14fa226f3c08fb2973937b" exitCode=0 Nov 03 03:07:59.033902 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:59.033487 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1243713b-skfvd" event={"ID":"1439cf8d-1d92-42ba-80ca-eb5c781fa883","Type":"ContainerDied","Data":"849a340013366e838d2ac2b3a3bebeb1186590dffe14fa226f3c08fb2973937b"} Nov 03 03:07:59.214629 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:59.214542 5605 patch_prober.go:29] interesting pod/e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 03:07:59.214629 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:07:59.214608 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht" podUID="261be9f1-1325-4210-b5f5-18b81a6b730d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 03:08:00.195676 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:00.195649 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1243713b-skfvd" Nov 03 03:08:00.287307 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:00.287270 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1439cf8d-1d92-42ba-80ca-eb5c781fa883-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1439cf8d-1d92-42ba-80ca-eb5c781fa883-maas-api-openshift-ca\") pod \"1439cf8d-1d92-42ba-80ca-eb5c781fa883\" (UID: \"1439cf8d-1d92-42ba-80ca-eb5c781fa883\") " Nov 03 03:08:00.287527 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:00.287375 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1439cf8d-1d92-42ba-80ca-eb5c781fa883-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1439cf8d-1d92-42ba-80ca-eb5c781fa883-maas-api-xks-ca\") pod \"1439cf8d-1d92-42ba-80ca-eb5c781fa883\" (UID: \"1439cf8d-1d92-42ba-80ca-eb5c781fa883\") " Nov 03 03:08:00.287527 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:00.287520 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1439cf8d-1d92-42ba-80ca-eb5c781fa883-kube-api-access-5v6bn\" (UniqueName: \"kubernetes.io/projected/1439cf8d-1d92-42ba-80ca-eb5c781fa883-kube-api-access-5v6bn\") pod \"1439cf8d-1d92-42ba-80ca-eb5c781fa883\" (UID: \"1439cf8d-1d92-42ba-80ca-eb5c781fa883\") " Nov 03 03:08:00.287700 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:00.287674 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1439cf8d-1d92-42ba-80ca-eb5c781fa883-maas-api-openshift-ca" pod "1439cf8d-1d92-42ba-80ca-eb5c781fa883" (UID: "1439cf8d-1d92-42ba-80ca-eb5c781fa883"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:08:00.288054 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:00.288025 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1439cf8d-1d92-42ba-80ca-eb5c781fa883-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:00.289527 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:00.289495 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1439cf8d-1d92-42ba-80ca-eb5c781fa883-maas-api-xks-ca" pod "1439cf8d-1d92-42ba-80ca-eb5c781fa883" (UID: "1439cf8d-1d92-42ba-80ca-eb5c781fa883"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:08:00.289677 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:00.289657 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1439cf8d-1d92-42ba-80ca-eb5c781fa883-kube-api-access-5v6bn" pod "1439cf8d-1d92-42ba-80ca-eb5c781fa883" (UID: "1439cf8d-1d92-42ba-80ca-eb5c781fa883"). InnerVolumeSpecName "kube-api-access-5v6bn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:08:00.389266 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:00.389226 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1439cf8d-1d92-42ba-80ca-eb5c781fa883-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:00.389266 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:00.389261 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5v6bn\" (UniqueName: \"kubernetes.io/projected/1439cf8d-1d92-42ba-80ca-eb5c781fa883-kube-api-access-5v6bn\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:00.671258 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:00.671160 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:08:01.047911 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:01.047821 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1243713b-skfvd" event={"ID":"1439cf8d-1d92-42ba-80ca-eb5c781fa883","Type":"ContainerDied","Data":"d8dd3c3425c49b6040c48faf8c41ede555c2f565d0b76e959bb3474f9ad1266f"} Nov 03 03:08:01.047911 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:01.047888 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8dd3c3425c49b6040c48faf8c41ede555c2f565d0b76e959bb3474f9ad1266f" Nov 03 03:08:01.047911 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:01.047841 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1243713b-skfvd" Nov 03 03:08:01.444084 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:01.444058 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht" Nov 03 03:08:01.505122 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:01.505088 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/261be9f1-1325-4210-b5f5-18b81a6b730d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/261be9f1-1325-4210-b5f5-18b81a6b730d-istiod-ca-cert\") pod \"261be9f1-1325-4210-b5f5-18b81a6b730d\" (UID: \"261be9f1-1325-4210-b5f5-18b81a6b730d\") " Nov 03 03:08:01.505313 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:01.505183 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/261be9f1-1325-4210-b5f5-18b81a6b730d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/261be9f1-1325-4210-b5f5-18b81a6b730d-istio-envoy\") pod \"261be9f1-1325-4210-b5f5-18b81a6b730d\" (UID: \"261be9f1-1325-4210-b5f5-18b81a6b730d\") " Nov 03 03:08:01.505313 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:01.505291 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/261be9f1-1325-4210-b5f5-18b81a6b730d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/261be9f1-1325-4210-b5f5-18b81a6b730d-istio-podinfo\") pod \"261be9f1-1325-4210-b5f5-18b81a6b730d\" (UID: \"261be9f1-1325-4210-b5f5-18b81a6b730d\") " Nov 03 03:08:01.505461 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:01.505330 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/261be9f1-1325-4210-b5f5-18b81a6b730d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/261be9f1-1325-4210-b5f5-18b81a6b730d-istio-data\") pod \"261be9f1-1325-4210-b5f5-18b81a6b730d\" (UID: \"261be9f1-1325-4210-b5f5-18b81a6b730d\") " Nov 03 03:08:01.505529 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:01.505493 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/261be9f1-1325-4210-b5f5-18b81a6b730d-istio-token\" (UniqueName: \"kubernetes.io/projected/261be9f1-1325-4210-b5f5-18b81a6b730d-istio-token\") pod \"261be9f1-1325-4210-b5f5-18b81a6b730d\" (UID: \"261be9f1-1325-4210-b5f5-18b81a6b730d\") " Nov 03 03:08:01.505595 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:01.505527 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/261be9f1-1325-4210-b5f5-18b81a6b730d-kube-api-access-swrr4\" (UniqueName: \"kubernetes.io/projected/261be9f1-1325-4210-b5f5-18b81a6b730d-kube-api-access-swrr4\") pod \"261be9f1-1325-4210-b5f5-18b81a6b730d\" (UID: \"261be9f1-1325-4210-b5f5-18b81a6b730d\") " Nov 03 03:08:01.505595 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:01.505579 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/261be9f1-1325-4210-b5f5-18b81a6b730d-istiod-ca-cert" pod "261be9f1-1325-4210-b5f5-18b81a6b730d" (UID: "261be9f1-1325-4210-b5f5-18b81a6b730d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:08:01.505717 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:01.505681 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/261be9f1-1325-4210-b5f5-18b81a6b730d-istio-data" pod "261be9f1-1325-4210-b5f5-18b81a6b730d" (UID: "261be9f1-1325-4210-b5f5-18b81a6b730d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:01.505717 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:01.505697 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/261be9f1-1325-4210-b5f5-18b81a6b730d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/261be9f1-1325-4210-b5f5-18b81a6b730d-workload-socket\") pod \"261be9f1-1325-4210-b5f5-18b81a6b730d\" (UID: \"261be9f1-1325-4210-b5f5-18b81a6b730d\") " Nov 03 03:08:01.505832 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:01.505743 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/261be9f1-1325-4210-b5f5-18b81a6b730d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/261be9f1-1325-4210-b5f5-18b81a6b730d-workload-certs\") pod \"261be9f1-1325-4210-b5f5-18b81a6b730d\" (UID: \"261be9f1-1325-4210-b5f5-18b81a6b730d\") " Nov 03 03:08:01.505832 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:01.505784 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/261be9f1-1325-4210-b5f5-18b81a6b730d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/261be9f1-1325-4210-b5f5-18b81a6b730d-credential-socket\") pod \"261be9f1-1325-4210-b5f5-18b81a6b730d\" (UID: \"261be9f1-1325-4210-b5f5-18b81a6b730d\") " Nov 03 03:08:01.506141 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:01.506113 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/261be9f1-1325-4210-b5f5-18b81a6b730d-credential-socket" pod "261be9f1-1325-4210-b5f5-18b81a6b730d" (UID: "261be9f1-1325-4210-b5f5-18b81a6b730d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:01.506141 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:01.506132 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/261be9f1-1325-4210-b5f5-18b81a6b730d-workload-certs" pod "261be9f1-1325-4210-b5f5-18b81a6b730d" (UID: "261be9f1-1325-4210-b5f5-18b81a6b730d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:01.507910 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:01.506228 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/261be9f1-1325-4210-b5f5-18b81a6b730d-workload-socket" pod "261be9f1-1325-4210-b5f5-18b81a6b730d" (UID: "261be9f1-1325-4210-b5f5-18b81a6b730d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:01.507910 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:01.506501 5605 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/261be9f1-1325-4210-b5f5-18b81a6b730d-workload-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:01.507910 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:01.506526 5605 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/261be9f1-1325-4210-b5f5-18b81a6b730d-workload-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:01.507910 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:01.506541 5605 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/261be9f1-1325-4210-b5f5-18b81a6b730d-credential-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:01.507910 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:01.506556 5605 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/261be9f1-1325-4210-b5f5-18b81a6b730d-istiod-ca-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:01.507910 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:01.506570 5605 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/261be9f1-1325-4210-b5f5-18b81a6b730d-istio-data\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:01.507910 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:01.507901 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/261be9f1-1325-4210-b5f5-18b81a6b730d-istio-envoy" pod "261be9f1-1325-4210-b5f5-18b81a6b730d" (UID: "261be9f1-1325-4210-b5f5-18b81a6b730d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:01.508194 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:01.508095 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/261be9f1-1325-4210-b5f5-18b81a6b730d-istio-podinfo" pod "261be9f1-1325-4210-b5f5-18b81a6b730d" (UID: "261be9f1-1325-4210-b5f5-18b81a6b730d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:08:01.508194 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:01.508135 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/261be9f1-1325-4210-b5f5-18b81a6b730d-istio-token" pod "261be9f1-1325-4210-b5f5-18b81a6b730d" (UID: "261be9f1-1325-4210-b5f5-18b81a6b730d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:08:01.508194 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:01.508158 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/261be9f1-1325-4210-b5f5-18b81a6b730d-kube-api-access-swrr4" pod "261be9f1-1325-4210-b5f5-18b81a6b730d" (UID: "261be9f1-1325-4210-b5f5-18b81a6b730d"). InnerVolumeSpecName "kube-api-access-swrr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:08:01.607914 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:01.607837 5605 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/261be9f1-1325-4210-b5f5-18b81a6b730d-istio-envoy\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:01.607914 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:01.607866 5605 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/261be9f1-1325-4210-b5f5-18b81a6b730d-istio-podinfo\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:01.607914 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:01.607875 5605 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/261be9f1-1325-4210-b5f5-18b81a6b730d-istio-token\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:01.607914 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:01.607885 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swrr4\" (UniqueName: \"kubernetes.io/projected/261be9f1-1325-4210-b5f5-18b81a6b730d-kube-api-access-swrr4\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:01.621665 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:01.621627 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c7d758/distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5"] Nov 03 03:08:02.055429 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:02.055395 5605 generic.go:356] "Generic (PLEG): container finished" podID="261be9f1-1325-4210-b5f5-18b81a6b730d" containerID="f89e2e82cb5c8624ae74f4b6f52a6676ecb08bb16e5f1df670d69764f65d1e00" exitCode=0 Nov 03 03:08:02.055619 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:02.055448 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht" Nov 03 03:08:02.055619 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:02.055479 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht" event={"ID":"261be9f1-1325-4210-b5f5-18b81a6b730d","Type":"ContainerDied","Data":"f89e2e82cb5c8624ae74f4b6f52a6676ecb08bb16e5f1df670d69764f65d1e00"} Nov 03 03:08:02.055619 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:02.055538 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht" event={"ID":"261be9f1-1325-4210-b5f5-18b81a6b730d","Type":"ContainerDied","Data":"c04b80f42267ce0f18cb2b4f1c91799499b7a39e603d755421bb20432b7e369f"} Nov 03 03:08:02.055619 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:02.055561 5605 scope.go:122] "RemoveContainer" containerID="f89e2e82cb5c8624ae74f4b6f52a6676ecb08bb16e5f1df670d69764f65d1e00" Nov 03 03:08:02.066101 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:02.066055 5605 scope.go:122] "RemoveContainer" containerID="87ae1fadf136362ee50138f4b289a5ab6bbf76148e2c01cf39bb58ced4c787c4" Nov 03 03:08:02.075026 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:02.075009 5605 scope.go:122] "RemoveContainer" containerID="f89e2e82cb5c8624ae74f4b6f52a6676ecb08bb16e5f1df670d69764f65d1e00" Nov 03 03:08:02.075273 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:08:02.075255 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f89e2e82cb5c8624ae74f4b6f52a6676ecb08bb16e5f1df670d69764f65d1e00\": container with ID starting with f89e2e82cb5c8624ae74f4b6f52a6676ecb08bb16e5f1df670d69764f65d1e00 not found: ID does not exist" containerID="f89e2e82cb5c8624ae74f4b6f52a6676ecb08bb16e5f1df670d69764f65d1e00" Nov 03 03:08:02.075332 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:02.075287 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f89e2e82cb5c8624ae74f4b6f52a6676ecb08bb16e5f1df670d69764f65d1e00"} err="failed to get container status \"f89e2e82cb5c8624ae74f4b6f52a6676ecb08bb16e5f1df670d69764f65d1e00\": rpc error: code = NotFound desc = could not find container \"f89e2e82cb5c8624ae74f4b6f52a6676ecb08bb16e5f1df670d69764f65d1e00\": container with ID starting with f89e2e82cb5c8624ae74f4b6f52a6676ecb08bb16e5f1df670d69764f65d1e00 not found: ID does not exist" Nov 03 03:08:02.075332 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:02.075306 5605 scope.go:122] "RemoveContainer" containerID="87ae1fadf136362ee50138f4b289a5ab6bbf76148e2c01cf39bb58ced4c787c4" Nov 03 03:08:02.075542 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:08:02.075523 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87ae1fadf136362ee50138f4b289a5ab6bbf76148e2c01cf39bb58ced4c787c4\": container with ID starting with 87ae1fadf136362ee50138f4b289a5ab6bbf76148e2c01cf39bb58ced4c787c4 not found: ID does not exist" containerID="87ae1fadf136362ee50138f4b289a5ab6bbf76148e2c01cf39bb58ced4c787c4" Nov 03 03:08:02.075579 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:02.075550 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87ae1fadf136362ee50138f4b289a5ab6bbf76148e2c01cf39bb58ced4c787c4"} err="failed to get container status \"87ae1fadf136362ee50138f4b289a5ab6bbf76148e2c01cf39bb58ced4c787c4\": rpc error: code = NotFound desc = could not find container \"87ae1fadf136362ee50138f4b289a5ab6bbf76148e2c01cf39bb58ced4c787c4\": container with ID starting with 87ae1fadf136362ee50138f4b289a5ab6bbf76148e2c01cf39bb58ced4c787c4 not found: ID does not exist" Nov 03 03:08:02.210400 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:02.210366 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht"] Nov 03 03:08:02.257153 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:02.257119 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-85e72378-openshift-default-7b79c67b5-nrlht"] Nov 03 03:08:02.850105 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:02.850070 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="261be9f1-1325-4210-b5f5-18b81a6b730d" path="/var/lib/kubelet/pods/261be9f1-1325-4210-b5f5-18b81a6b730d/volumes" Nov 03 03:08:05.167820 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:05.167784 5605 scope.go:122] "RemoveContainer" containerID="a721dd2dd0ccf7b9b91eaad7ad2d74a3579f485e172e4015b80ad624d4ffc964" Nov 03 03:08:08.251592 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:08.251560 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dc834fd-hh4fr"] Nov 03 03:08:08.252047 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:08.252033 5605 cpu_manager.go:414] "containerMap: removing container" podUID="1439cf8d-1d92-42ba-80ca-eb5c781fa883" containerName="revoke-keys" Nov 03 03:08:08.252094 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:08.252048 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1439cf8d-1d92-42ba-80ca-eb5c781fa883" containerName="revoke-keys" Nov 03 03:08:08.252094 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:08.252068 5605 cpu_manager.go:414] "containerMap: removing container" podUID="261be9f1-1325-4210-b5f5-18b81a6b730d" containerName="istio-proxy" Nov 03 03:08:08.252094 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:08.252076 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="261be9f1-1325-4210-b5f5-18b81a6b730d" containerName="istio-proxy" Nov 03 03:08:08.252190 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:08.252095 5605 cpu_manager.go:414] "containerMap: removing container" podUID="261be9f1-1325-4210-b5f5-18b81a6b730d" containerName="istio-proxy" Nov 03 03:08:08.252190 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:08.252101 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="261be9f1-1325-4210-b5f5-18b81a6b730d" containerName="istio-proxy" Nov 03 03:08:08.252248 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:08.252199 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="1439cf8d-1d92-42ba-80ca-eb5c781fa883" containerName="revoke-keys" Nov 03 03:08:08.252248 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:08.252211 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="261be9f1-1325-4210-b5f5-18b81a6b730d" containerName="istio-proxy" Nov 03 03:08:08.252248 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:08.252221 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="261be9f1-1325-4210-b5f5-18b81a6b730d" containerName="istio-proxy" Nov 03 03:08:08.257124 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:08.257098 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc834fd-hh4fr" Nov 03 03:08:08.310029 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:08.310005 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dc834fd-hh4fr"] Nov 03 03:08:08.353854 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:08.353827 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:08:08.353854 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:08.353827 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:08:08.354042 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:08.353836 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:08:08.354042 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:08.353883 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 03:08:08.387175 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:08.387145 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8a5223d-e4dd-4c99-8042-825c94b85ba4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9dc834fd-hh4fr\" (UID: \"e8a5223d-e4dd-4c99-8042-825c94b85ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc834fd-hh4fr" Nov 03 03:08:08.387282 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:08.387185 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8a5223d-e4dd-4c99-8042-825c94b85ba4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9dc834fd-hh4fr\" (UID: \"e8a5223d-e4dd-4c99-8042-825c94b85ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc834fd-hh4fr" Nov 03 03:08:08.387345 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:08.387329 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbpnm\" (UniqueName: \"kubernetes.io/projected/e8a5223d-e4dd-4c99-8042-825c94b85ba4-kube-api-access-xbpnm\") pod \"maas-api-revoke-sub-9dc834fd-hh4fr\" (UID: \"e8a5223d-e4dd-4c99-8042-825c94b85ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc834fd-hh4fr" Nov 03 03:08:08.487994 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:08.487969 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8a5223d-e4dd-4c99-8042-825c94b85ba4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9dc834fd-hh4fr\" (UID: \"e8a5223d-e4dd-4c99-8042-825c94b85ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc834fd-hh4fr" Nov 03 03:08:08.488125 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:08.488015 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8a5223d-e4dd-4c99-8042-825c94b85ba4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9dc834fd-hh4fr\" (UID: \"e8a5223d-e4dd-4c99-8042-825c94b85ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc834fd-hh4fr" Nov 03 03:08:08.488125 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:08.488078 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbpnm\" (UniqueName: \"kubernetes.io/projected/e8a5223d-e4dd-4c99-8042-825c94b85ba4-kube-api-access-xbpnm\") pod \"maas-api-revoke-sub-9dc834fd-hh4fr\" (UID: \"e8a5223d-e4dd-4c99-8042-825c94b85ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc834fd-hh4fr" Nov 03 03:08:08.488696 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:08.488673 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8a5223d-e4dd-4c99-8042-825c94b85ba4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9dc834fd-hh4fr\" (UID: \"e8a5223d-e4dd-4c99-8042-825c94b85ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc834fd-hh4fr" Nov 03 03:08:08.490517 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:08.490492 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8a5223d-e4dd-4c99-8042-825c94b85ba4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9dc834fd-hh4fr\" (UID: \"e8a5223d-e4dd-4c99-8042-825c94b85ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc834fd-hh4fr" Nov 03 03:08:08.548687 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:08.548663 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbpnm\" (UniqueName: \"kubernetes.io/projected/e8a5223d-e4dd-4c99-8042-825c94b85ba4-kube-api-access-xbpnm\") pod \"maas-api-revoke-sub-9dc834fd-hh4fr\" (UID: \"e8a5223d-e4dd-4c99-8042-825c94b85ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc834fd-hh4fr" Nov 03 03:08:08.570460 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:08.570437 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc834fd-hh4fr" Nov 03 03:08:08.946349 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:08.946275 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dc834fd-hh4fr"] Nov 03 03:08:08.952046 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:08:08.952020 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8a5223d_e4dd_4c99_8042_825c94b85ba4.slice/crio-128c84871076e6fed72aa9115e9d1848ed8baf3760674f9e7fd4b59652fe08b4 WatchSource:0}: Error finding container 128c84871076e6fed72aa9115e9d1848ed8baf3760674f9e7fd4b59652fe08b4: Status 404 returned error can't find the container with id 128c84871076e6fed72aa9115e9d1848ed8baf3760674f9e7fd4b59652fe08b4 Nov 03 03:08:09.098172 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:09.098141 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc834fd-hh4fr" event={"ID":"e8a5223d-e4dd-4c99-8042-825c94b85ba4","Type":"ContainerStarted","Data":"128c84871076e6fed72aa9115e9d1848ed8baf3760674f9e7fd4b59652fe08b4"} Nov 03 03:08:10.106137 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:10.106046 5605 generic.go:356] "Generic (PLEG): container finished" podID="e8a5223d-e4dd-4c99-8042-825c94b85ba4" containerID="57d83a43860b48d30eff86b838c613ff51c7a59efe27bdb7acb6a8997c7e0288" exitCode=0 Nov 03 03:08:10.106607 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:10.106132 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc834fd-hh4fr" event={"ID":"e8a5223d-e4dd-4c99-8042-825c94b85ba4","Type":"ContainerDied","Data":"57d83a43860b48d30eff86b838c613ff51c7a59efe27bdb7acb6a8997c7e0288"} Nov 03 03:08:11.259004 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:11.258977 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc834fd-hh4fr" Nov 03 03:08:11.420271 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:11.420185 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8a5223d-e4dd-4c99-8042-825c94b85ba4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8a5223d-e4dd-4c99-8042-825c94b85ba4-maas-api-xks-ca\") pod \"e8a5223d-e4dd-4c99-8042-825c94b85ba4\" (UID: \"e8a5223d-e4dd-4c99-8042-825c94b85ba4\") " Nov 03 03:08:11.420271 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:11.420253 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8a5223d-e4dd-4c99-8042-825c94b85ba4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8a5223d-e4dd-4c99-8042-825c94b85ba4-maas-api-openshift-ca\") pod \"e8a5223d-e4dd-4c99-8042-825c94b85ba4\" (UID: \"e8a5223d-e4dd-4c99-8042-825c94b85ba4\") " Nov 03 03:08:11.420526 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:11.420419 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8a5223d-e4dd-4c99-8042-825c94b85ba4-kube-api-access-xbpnm\" (UniqueName: \"kubernetes.io/projected/e8a5223d-e4dd-4c99-8042-825c94b85ba4-kube-api-access-xbpnm\") pod \"e8a5223d-e4dd-4c99-8042-825c94b85ba4\" (UID: \"e8a5223d-e4dd-4c99-8042-825c94b85ba4\") " Nov 03 03:08:11.420710 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:11.420678 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8a5223d-e4dd-4c99-8042-825c94b85ba4-maas-api-openshift-ca" pod "e8a5223d-e4dd-4c99-8042-825c94b85ba4" (UID: "e8a5223d-e4dd-4c99-8042-825c94b85ba4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:08:11.420888 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:11.420873 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8a5223d-e4dd-4c99-8042-825c94b85ba4-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:11.422423 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:11.422394 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8a5223d-e4dd-4c99-8042-825c94b85ba4-maas-api-xks-ca" pod "e8a5223d-e4dd-4c99-8042-825c94b85ba4" (UID: "e8a5223d-e4dd-4c99-8042-825c94b85ba4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:08:11.422598 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:11.422584 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8a5223d-e4dd-4c99-8042-825c94b85ba4-kube-api-access-xbpnm" pod "e8a5223d-e4dd-4c99-8042-825c94b85ba4" (UID: "e8a5223d-e4dd-4c99-8042-825c94b85ba4"). InnerVolumeSpecName "kube-api-access-xbpnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:08:11.522534 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:11.522495 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbpnm\" (UniqueName: \"kubernetes.io/projected/e8a5223d-e4dd-4c99-8042-825c94b85ba4-kube-api-access-xbpnm\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:11.522534 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:11.522527 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8a5223d-e4dd-4c99-8042-825c94b85ba4-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:12.122368 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:12.122324 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc834fd-hh4fr" event={"ID":"e8a5223d-e4dd-4c99-8042-825c94b85ba4","Type":"ContainerDied","Data":"128c84871076e6fed72aa9115e9d1848ed8baf3760674f9e7fd4b59652fe08b4"} Nov 03 03:08:12.122553 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:12.122377 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc834fd-hh4fr" Nov 03 03:08:12.122553 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:12.122391 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="128c84871076e6fed72aa9115e9d1848ed8baf3760674f9e7fd4b59652fe08b4" Nov 03 03:08:12.599254 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:12.599216 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-a0e72e/distinct-2-w3-a0e72e-kserve-59f58c59bf-jz5p8"] Nov 03 03:08:12.609969 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:12.609933 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-a0e72e/distinct-w3-a0e72e-kserve-fb499cc4-2dfxc"] Nov 03 03:08:12.646289 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:12.646250 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-a0e72e/premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv"] Nov 03 03:08:12.648575 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:12.648547 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-a0e72e/simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq"] Nov 03 03:08:13.598627 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:13.598594 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa18948d-pmch6"] Nov 03 03:08:13.599079 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:13.599066 5605 cpu_manager.go:414] "containerMap: removing container" podUID="e8a5223d-e4dd-4c99-8042-825c94b85ba4" containerName="revoke-keys" Nov 03 03:08:13.599129 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:13.599080 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8a5223d-e4dd-4c99-8042-825c94b85ba4" containerName="revoke-keys" Nov 03 03:08:13.599194 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:13.599185 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8a5223d-e4dd-4c99-8042-825c94b85ba4" containerName="revoke-keys" Nov 03 03:08:13.602697 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:13.602673 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa18948d-pmch6" Nov 03 03:08:13.657256 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:13.657218 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:08:13.657256 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:13.657249 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:08:13.657535 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:13.657488 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 03:08:13.657535 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:13.657496 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:08:13.663726 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:13.663697 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa18948d-pmch6"] Nov 03 03:08:13.749768 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:13.749731 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbg56\" (UniqueName: \"kubernetes.io/projected/f58b31f9-f1f8-49c9-bd84-dfc5230ae8ea-kube-api-access-jbg56\") pod \"maas-api-revoke-sub-aa18948d-pmch6\" (UID: \"f58b31f9-f1f8-49c9-bd84-dfc5230ae8ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa18948d-pmch6" Nov 03 03:08:13.749964 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:13.749854 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f58b31f9-f1f8-49c9-bd84-dfc5230ae8ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa18948d-pmch6\" (UID: \"f58b31f9-f1f8-49c9-bd84-dfc5230ae8ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa18948d-pmch6" Nov 03 03:08:13.749964 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:13.749934 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f58b31f9-f1f8-49c9-bd84-dfc5230ae8ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa18948d-pmch6\" (UID: \"f58b31f9-f1f8-49c9-bd84-dfc5230ae8ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa18948d-pmch6" Nov 03 03:08:13.851218 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:13.851116 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f58b31f9-f1f8-49c9-bd84-dfc5230ae8ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa18948d-pmch6\" (UID: \"f58b31f9-f1f8-49c9-bd84-dfc5230ae8ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa18948d-pmch6" Nov 03 03:08:13.851218 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:13.851197 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f58b31f9-f1f8-49c9-bd84-dfc5230ae8ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa18948d-pmch6\" (UID: \"f58b31f9-f1f8-49c9-bd84-dfc5230ae8ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa18948d-pmch6" Nov 03 03:08:13.851479 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:13.851264 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbg56\" (UniqueName: \"kubernetes.io/projected/f58b31f9-f1f8-49c9-bd84-dfc5230ae8ea-kube-api-access-jbg56\") pod \"maas-api-revoke-sub-aa18948d-pmch6\" (UID: \"f58b31f9-f1f8-49c9-bd84-dfc5230ae8ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa18948d-pmch6" Nov 03 03:08:13.851970 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:13.851943 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f58b31f9-f1f8-49c9-bd84-dfc5230ae8ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa18948d-pmch6\" (UID: \"f58b31f9-f1f8-49c9-bd84-dfc5230ae8ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa18948d-pmch6" Nov 03 03:08:13.853722 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:13.853696 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f58b31f9-f1f8-49c9-bd84-dfc5230ae8ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa18948d-pmch6\" (UID: \"f58b31f9-f1f8-49c9-bd84-dfc5230ae8ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa18948d-pmch6" Nov 03 03:08:13.905864 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:13.905819 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbg56\" (UniqueName: \"kubernetes.io/projected/f58b31f9-f1f8-49c9-bd84-dfc5230ae8ea-kube-api-access-jbg56\") pod \"maas-api-revoke-sub-aa18948d-pmch6\" (UID: \"f58b31f9-f1f8-49c9-bd84-dfc5230ae8ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa18948d-pmch6" Nov 03 03:08:13.915347 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:13.915285 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa18948d-pmch6" Nov 03 03:08:14.286935 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:14.286847 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa18948d-pmch6"] Nov 03 03:08:14.292002 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:08:14.291973 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf58b31f9_f1f8_49c9_bd84_dfc5230ae8ea.slice/crio-b1f24e0bb000455451143b07e2dbd094dbdef7b7b2cd018172577259c4d6ddf4 WatchSource:0}: Error finding container b1f24e0bb000455451143b07e2dbd094dbdef7b7b2cd018172577259c4d6ddf4: Status 404 returned error can't find the container with id b1f24e0bb000455451143b07e2dbd094dbdef7b7b2cd018172577259c4d6ddf4 Nov 03 03:08:15.140754 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:15.140718 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa18948d-pmch6" event={"ID":"f58b31f9-f1f8-49c9-bd84-dfc5230ae8ea","Type":"ContainerStarted","Data":"1148f4805e7c8510e14cf6ffd3b3d352578f13a0810e32413f37f03a51acaf74"} Nov 03 03:08:15.141136 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:15.140781 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa18948d-pmch6" event={"ID":"f58b31f9-f1f8-49c9-bd84-dfc5230ae8ea","Type":"ContainerStarted","Data":"b1f24e0bb000455451143b07e2dbd094dbdef7b7b2cd018172577259c4d6ddf4"} Nov 03 03:08:15.248721 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:15.248671 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa18948d-pmch6" podStartSLOduration=2.2486571 podStartE2EDuration="2.2486571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:08:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:08:15.247490061 +0000 UTC m=+4572.506688216" watchObservedRunningTime="2026-11-03 03:08:15.2486571 +0000 UTC m=+4572.507855254" Nov 03 03:08:15.273517 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:15.273491 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c7d758-d578b44bc-cccmm"] Nov 03 03:08:16.147796 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:16.147759 5605 generic.go:356] "Generic (PLEG): container finished" podID="f58b31f9-f1f8-49c9-bd84-dfc5230ae8ea" containerID="1148f4805e7c8510e14cf6ffd3b3d352578f13a0810e32413f37f03a51acaf74" exitCode=0 Nov 03 03:08:16.148197 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:16.147840 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa18948d-pmch6" event={"ID":"f58b31f9-f1f8-49c9-bd84-dfc5230ae8ea","Type":"ContainerDied","Data":"1148f4805e7c8510e14cf6ffd3b3d352578f13a0810e32413f37f03a51acaf74"} Nov 03 03:08:16.642312 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:16.642273 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-c7d758/distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5" podUID="fffc3fbb-097a-4a5d-82c8-b1e221d0e710" containerName="main" containerID="cri-o://ede419b3651cd05f584bfbde949fdae0a9bf930d245e968f6a29a25f8e9e0bb9" gracePeriod=45 Nov 03 03:08:16.923697 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:16.923675 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c7d758/distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5" Nov 03 03:08:17.088991 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:17.088960 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fffc3fbb-097a-4a5d-82c8-b1e221d0e710-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fffc3fbb-097a-4a5d-82c8-b1e221d0e710-model-cache\") pod \"fffc3fbb-097a-4a5d-82c8-b1e221d0e710\" (UID: \"fffc3fbb-097a-4a5d-82c8-b1e221d0e710\") " Nov 03 03:08:17.089168 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:17.089019 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fffc3fbb-097a-4a5d-82c8-b1e221d0e710-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fffc3fbb-097a-4a5d-82c8-b1e221d0e710-tmp-dir\") pod \"fffc3fbb-097a-4a5d-82c8-b1e221d0e710\" (UID: \"fffc3fbb-097a-4a5d-82c8-b1e221d0e710\") " Nov 03 03:08:17.089168 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:17.089106 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fffc3fbb-097a-4a5d-82c8-b1e221d0e710-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fffc3fbb-097a-4a5d-82c8-b1e221d0e710-dshm\") pod \"fffc3fbb-097a-4a5d-82c8-b1e221d0e710\" (UID: \"fffc3fbb-097a-4a5d-82c8-b1e221d0e710\") " Nov 03 03:08:17.089264 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:17.089178 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fffc3fbb-097a-4a5d-82c8-b1e221d0e710-kube-api-access-srb5t\" (UniqueName: \"kubernetes.io/projected/fffc3fbb-097a-4a5d-82c8-b1e221d0e710-kube-api-access-srb5t\") pod \"fffc3fbb-097a-4a5d-82c8-b1e221d0e710\" (UID: \"fffc3fbb-097a-4a5d-82c8-b1e221d0e710\") " Nov 03 03:08:17.089264 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:17.089199 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fffc3fbb-097a-4a5d-82c8-b1e221d0e710-model-cache" pod "fffc3fbb-097a-4a5d-82c8-b1e221d0e710" (UID: "fffc3fbb-097a-4a5d-82c8-b1e221d0e710"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:17.089264 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:17.089238 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fffc3fbb-097a-4a5d-82c8-b1e221d0e710-home\" (UniqueName: \"kubernetes.io/empty-dir/fffc3fbb-097a-4a5d-82c8-b1e221d0e710-home\") pod \"fffc3fbb-097a-4a5d-82c8-b1e221d0e710\" (UID: \"fffc3fbb-097a-4a5d-82c8-b1e221d0e710\") " Nov 03 03:08:17.089394 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:17.089281 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fffc3fbb-097a-4a5d-82c8-b1e221d0e710-tls-certs\" (UniqueName: \"kubernetes.io/secret/fffc3fbb-097a-4a5d-82c8-b1e221d0e710-tls-certs\") pod \"fffc3fbb-097a-4a5d-82c8-b1e221d0e710\" (UID: \"fffc3fbb-097a-4a5d-82c8-b1e221d0e710\") " Nov 03 03:08:17.089394 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:17.089281 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fffc3fbb-097a-4a5d-82c8-b1e221d0e710-tmp-dir" pod "fffc3fbb-097a-4a5d-82c8-b1e221d0e710" (UID: "fffc3fbb-097a-4a5d-82c8-b1e221d0e710"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:17.089518 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:17.089499 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fffc3fbb-097a-4a5d-82c8-b1e221d0e710-home" pod "fffc3fbb-097a-4a5d-82c8-b1e221d0e710" (UID: "fffc3fbb-097a-4a5d-82c8-b1e221d0e710"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:17.089723 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:17.089702 5605 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fffc3fbb-097a-4a5d-82c8-b1e221d0e710-home\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:17.089723 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:17.089724 5605 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fffc3fbb-097a-4a5d-82c8-b1e221d0e710-model-cache\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:17.089873 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:17.089733 5605 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fffc3fbb-097a-4a5d-82c8-b1e221d0e710-tmp-dir\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:17.091336 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:17.091312 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fffc3fbb-097a-4a5d-82c8-b1e221d0e710-dshm" pod "fffc3fbb-097a-4a5d-82c8-b1e221d0e710" (UID: "fffc3fbb-097a-4a5d-82c8-b1e221d0e710"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:17.091541 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:17.091519 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fffc3fbb-097a-4a5d-82c8-b1e221d0e710-tls-certs" pod "fffc3fbb-097a-4a5d-82c8-b1e221d0e710" (UID: "fffc3fbb-097a-4a5d-82c8-b1e221d0e710"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:08:17.091686 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:17.091667 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fffc3fbb-097a-4a5d-82c8-b1e221d0e710-kube-api-access-srb5t" pod "fffc3fbb-097a-4a5d-82c8-b1e221d0e710" (UID: "fffc3fbb-097a-4a5d-82c8-b1e221d0e710"). InnerVolumeSpecName "kube-api-access-srb5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:08:17.155543 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:17.155455 5605 generic.go:356] "Generic (PLEG): container finished" podID="fffc3fbb-097a-4a5d-82c8-b1e221d0e710" containerID="ede419b3651cd05f584bfbde949fdae0a9bf930d245e968f6a29a25f8e9e0bb9" exitCode=0 Nov 03 03:08:17.155543 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:17.155504 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c7d758/distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5" Nov 03 03:08:17.156028 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:17.155530 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c7d758/distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5" event={"ID":"fffc3fbb-097a-4a5d-82c8-b1e221d0e710","Type":"ContainerDied","Data":"ede419b3651cd05f584bfbde949fdae0a9bf930d245e968f6a29a25f8e9e0bb9"} Nov 03 03:08:17.156028 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:17.155579 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c7d758/distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5" event={"ID":"fffc3fbb-097a-4a5d-82c8-b1e221d0e710","Type":"ContainerDied","Data":"2b818c481602eb05f3a43c56b5b6249b70f08800abaca27f3c6a31530756249d"} Nov 03 03:08:17.156028 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:17.155596 5605 scope.go:122] "RemoveContainer" containerID="ede419b3651cd05f584bfbde949fdae0a9bf930d245e968f6a29a25f8e9e0bb9" Nov 03 03:08:17.174121 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:17.174097 5605 scope.go:122] "RemoveContainer" containerID="ede419b3651cd05f584bfbde949fdae0a9bf930d245e968f6a29a25f8e9e0bb9" Nov 03 03:08:17.174533 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:08:17.174503 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ede419b3651cd05f584bfbde949fdae0a9bf930d245e968f6a29a25f8e9e0bb9\": container with ID starting with ede419b3651cd05f584bfbde949fdae0a9bf930d245e968f6a29a25f8e9e0bb9 not found: ID does not exist" containerID="ede419b3651cd05f584bfbde949fdae0a9bf930d245e968f6a29a25f8e9e0bb9" Nov 03 03:08:17.174661 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:17.174544 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ede419b3651cd05f584bfbde949fdae0a9bf930d245e968f6a29a25f8e9e0bb9"} err="failed to get container status \"ede419b3651cd05f584bfbde949fdae0a9bf930d245e968f6a29a25f8e9e0bb9\": rpc error: code = NotFound desc = could not find container \"ede419b3651cd05f584bfbde949fdae0a9bf930d245e968f6a29a25f8e9e0bb9\": container with ID starting with ede419b3651cd05f584bfbde949fdae0a9bf930d245e968f6a29a25f8e9e0bb9 not found: ID does not exist" Nov 03 03:08:17.191695 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:17.191667 5605 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fffc3fbb-097a-4a5d-82c8-b1e221d0e710-dshm\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:17.191862 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:17.191842 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srb5t\" (UniqueName: \"kubernetes.io/projected/fffc3fbb-097a-4a5d-82c8-b1e221d0e710-kube-api-access-srb5t\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:17.191862 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:17.191863 5605 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fffc3fbb-097a-4a5d-82c8-b1e221d0e710-tls-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:17.312521 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:17.312412 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c7d758/distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5"] Nov 03 03:08:17.315585 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:17.315558 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-c7d758/distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5"] Nov 03 03:08:17.333473 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:17.333449 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa18948d-pmch6" Nov 03 03:08:17.496260 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:17.496187 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f58b31f9-f1f8-49c9-bd84-dfc5230ae8ea-kube-api-access-jbg56\" (UniqueName: \"kubernetes.io/projected/f58b31f9-f1f8-49c9-bd84-dfc5230ae8ea-kube-api-access-jbg56\") pod \"f58b31f9-f1f8-49c9-bd84-dfc5230ae8ea\" (UID: \"f58b31f9-f1f8-49c9-bd84-dfc5230ae8ea\") " Nov 03 03:08:17.496432 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:17.496307 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f58b31f9-f1f8-49c9-bd84-dfc5230ae8ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f58b31f9-f1f8-49c9-bd84-dfc5230ae8ea-maas-api-xks-ca\") pod \"f58b31f9-f1f8-49c9-bd84-dfc5230ae8ea\" (UID: \"f58b31f9-f1f8-49c9-bd84-dfc5230ae8ea\") " Nov 03 03:08:17.496493 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:17.496475 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f58b31f9-f1f8-49c9-bd84-dfc5230ae8ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f58b31f9-f1f8-49c9-bd84-dfc5230ae8ea-maas-api-openshift-ca\") pod \"f58b31f9-f1f8-49c9-bd84-dfc5230ae8ea\" (UID: \"f58b31f9-f1f8-49c9-bd84-dfc5230ae8ea\") " Nov 03 03:08:17.496944 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:17.496913 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f58b31f9-f1f8-49c9-bd84-dfc5230ae8ea-maas-api-openshift-ca" pod "f58b31f9-f1f8-49c9-bd84-dfc5230ae8ea" (UID: "f58b31f9-f1f8-49c9-bd84-dfc5230ae8ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:08:17.498510 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:17.498485 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f58b31f9-f1f8-49c9-bd84-dfc5230ae8ea-maas-api-xks-ca" pod "f58b31f9-f1f8-49c9-bd84-dfc5230ae8ea" (UID: "f58b31f9-f1f8-49c9-bd84-dfc5230ae8ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:08:17.498597 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:17.498542 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f58b31f9-f1f8-49c9-bd84-dfc5230ae8ea-kube-api-access-jbg56" pod "f58b31f9-f1f8-49c9-bd84-dfc5230ae8ea" (UID: "f58b31f9-f1f8-49c9-bd84-dfc5230ae8ea"). InnerVolumeSpecName "kube-api-access-jbg56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:08:17.597902 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:17.597868 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f58b31f9-f1f8-49c9-bd84-dfc5230ae8ea-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:17.597902 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:17.597902 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbg56\" (UniqueName: \"kubernetes.io/projected/f58b31f9-f1f8-49c9-bd84-dfc5230ae8ea-kube-api-access-jbg56\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:17.598048 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:17.597915 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f58b31f9-f1f8-49c9-bd84-dfc5230ae8ea-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:17.898507 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:17.898470 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/distinct-2-w4-c7d758-kserve-6f94867f84-mw4j5" podUID="fffc3fbb-097a-4a5d-82c8-b1e221d0e710" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:08:18.164520 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:18.164426 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa18948d-pmch6" Nov 03 03:08:18.164520 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:18.164453 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa18948d-pmch6" event={"ID":"f58b31f9-f1f8-49c9-bd84-dfc5230ae8ea","Type":"ContainerDied","Data":"b1f24e0bb000455451143b07e2dbd094dbdef7b7b2cd018172577259c4d6ddf4"} Nov 03 03:08:18.165049 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:18.164528 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1f24e0bb000455451143b07e2dbd094dbdef7b7b2cd018172577259c4d6ddf4" Nov 03 03:08:18.850445 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:18.850403 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fffc3fbb-097a-4a5d-82c8-b1e221d0e710" path="/var/lib/kubelet/pods/fffc3fbb-097a-4a5d-82c8-b1e221d0e710/volumes" Nov 03 03:08:20.273767 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:20.273709 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-c7d758-d578b44bc-cccmm" podUID="e2eff9e7-e011-46ce-a9e1-306ca1bf5893" containerName="payload-processing" containerID="cri-o://1ac18cacaa2401eeb837d21b4ebd191b0603a700887195bcb3343877b4b6bd03" gracePeriod=25 Nov 03 03:08:23.916703 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:08:23.916655 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:08:24.589852 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:24.589811 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-c7d758-d578b44bc-cccmm" podUID="e2eff9e7-e011-46ce-a9e1-306ca1bf5893" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:08:25.538381 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:25.538334 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-a0e72e-64b874b665-4k4v9"] Nov 03 03:08:25.765838 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:25.765802 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af76fd17-lhvd7"] Nov 03 03:08:25.766684 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:25.766657 5605 cpu_manager.go:414] "containerMap: removing container" podUID="fffc3fbb-097a-4a5d-82c8-b1e221d0e710" containerName="main" Nov 03 03:08:25.766684 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:25.766682 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fffc3fbb-097a-4a5d-82c8-b1e221d0e710" containerName="main" Nov 03 03:08:25.766959 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:25.766714 5605 cpu_manager.go:414] "containerMap: removing container" podUID="f58b31f9-f1f8-49c9-bd84-dfc5230ae8ea" containerName="revoke-keys" Nov 03 03:08:25.766959 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:25.766722 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f58b31f9-f1f8-49c9-bd84-dfc5230ae8ea" containerName="revoke-keys" Nov 03 03:08:25.767090 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:25.766971 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="fffc3fbb-097a-4a5d-82c8-b1e221d0e710" containerName="main" Nov 03 03:08:25.767090 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:25.766996 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="f58b31f9-f1f8-49c9-bd84-dfc5230ae8ea" containerName="revoke-keys" Nov 03 03:08:25.770775 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:25.770754 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af76fd17-lhvd7" Nov 03 03:08:25.795149 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:25.795091 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b9772d3-3595-4ac3-b747-d7cbaae23c08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af76fd17-lhvd7\" (UID: \"6b9772d3-3595-4ac3-b747-d7cbaae23c08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af76fd17-lhvd7" Nov 03 03:08:25.795149 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:25.795139 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j5gp\" (UniqueName: \"kubernetes.io/projected/6b9772d3-3595-4ac3-b747-d7cbaae23c08-kube-api-access-4j5gp\") pod \"maas-api-revoke-sub-af76fd17-lhvd7\" (UID: \"6b9772d3-3595-4ac3-b747-d7cbaae23c08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af76fd17-lhvd7" Nov 03 03:08:25.795295 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:25.795217 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b9772d3-3595-4ac3-b747-d7cbaae23c08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af76fd17-lhvd7\" (UID: \"6b9772d3-3595-4ac3-b747-d7cbaae23c08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af76fd17-lhvd7" Nov 03 03:08:25.820673 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:25.820642 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:08:25.820933 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:25.820912 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:08:25.821047 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:25.820954 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 03:08:25.822368 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:25.822333 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af76fd17-lhvd7"] Nov 03 03:08:25.823463 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:25.823436 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:08:25.896026 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:25.895998 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b9772d3-3595-4ac3-b747-d7cbaae23c08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af76fd17-lhvd7\" (UID: \"6b9772d3-3595-4ac3-b747-d7cbaae23c08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af76fd17-lhvd7" Nov 03 03:08:25.896185 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:25.896057 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j5gp\" (UniqueName: \"kubernetes.io/projected/6b9772d3-3595-4ac3-b747-d7cbaae23c08-kube-api-access-4j5gp\") pod \"maas-api-revoke-sub-af76fd17-lhvd7\" (UID: \"6b9772d3-3595-4ac3-b747-d7cbaae23c08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af76fd17-lhvd7" Nov 03 03:08:25.896185 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:25.896112 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b9772d3-3595-4ac3-b747-d7cbaae23c08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af76fd17-lhvd7\" (UID: \"6b9772d3-3595-4ac3-b747-d7cbaae23c08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af76fd17-lhvd7" Nov 03 03:08:25.896813 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:25.896792 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b9772d3-3595-4ac3-b747-d7cbaae23c08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af76fd17-lhvd7\" (UID: \"6b9772d3-3595-4ac3-b747-d7cbaae23c08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af76fd17-lhvd7" Nov 03 03:08:25.898403 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:25.898385 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b9772d3-3595-4ac3-b747-d7cbaae23c08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af76fd17-lhvd7\" (UID: \"6b9772d3-3595-4ac3-b747-d7cbaae23c08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af76fd17-lhvd7" Nov 03 03:08:25.969580 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:25.956766 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j5gp\" (UniqueName: \"kubernetes.io/projected/6b9772d3-3595-4ac3-b747-d7cbaae23c08-kube-api-access-4j5gp\") pod \"maas-api-revoke-sub-af76fd17-lhvd7\" (UID: \"6b9772d3-3595-4ac3-b747-d7cbaae23c08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af76fd17-lhvd7" Nov 03 03:08:26.085948 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:26.085913 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af76fd17-lhvd7" Nov 03 03:08:26.474782 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:26.474694 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af76fd17-lhvd7"] Nov 03 03:08:26.481743 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:08:26.481716 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b9772d3_3595_4ac3_b747_d7cbaae23c08.slice/crio-0c8d1163f348c6f255309fbdc915cd293751c0134211aeabc3f52a1f57761b50 WatchSource:0}: Error finding container 0c8d1163f348c6f255309fbdc915cd293751c0134211aeabc3f52a1f57761b50: Status 404 returned error can't find the container with id 0c8d1163f348c6f255309fbdc915cd293751c0134211aeabc3f52a1f57761b50 Nov 03 03:08:27.223187 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:27.223150 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af76fd17-lhvd7" event={"ID":"6b9772d3-3595-4ac3-b747-d7cbaae23c08","Type":"ContainerStarted","Data":"114ebee09aab92b2815ce5c7232f54ae0e5aaf077122ae644bc50db162642b7b"} Nov 03 03:08:27.223738 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:27.223202 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af76fd17-lhvd7" event={"ID":"6b9772d3-3595-4ac3-b747-d7cbaae23c08","Type":"ContainerStarted","Data":"0c8d1163f348c6f255309fbdc915cd293751c0134211aeabc3f52a1f57761b50"} Nov 03 03:08:27.337530 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:27.337474 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af76fd17-lhvd7" podStartSLOduration=2.33745995 podStartE2EDuration="2.33745995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:08:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:08:27.336528283 +0000 UTC m=+4584.595726439" watchObservedRunningTime="2026-11-03 03:08:27.33745995 +0000 UTC m=+4584.596658103" Nov 03 03:08:27.620310 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:27.620272 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-a0e72e/distinct-2-w3-a0e72e-kserve-59f58c59bf-jz5p8" podUID="70bd98e0-d8ac-47c2-901e-83f301d59a57" containerName="main" containerID="cri-o://81d3f5c16c98c27bdb8dfd4a2e963704733d3775640740eec7d25422c78230c9" gracePeriod=45 Nov 03 03:08:27.631888 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:27.631840 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-a0e72e/distinct-w3-a0e72e-kserve-fb499cc4-2dfxc" podUID="308bca3b-9bd5-4724-9b95-acdbc978aa10" containerName="main" containerID="cri-o://1aa09c6571bd9571b9f8b05e504c098aab7c3adeae9acd73269599620cb9d477" gracePeriod=45 Nov 03 03:08:27.668017 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:27.667969 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-a0e72e/premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv" podUID="a9b9af44-ab6d-4caf-89af-c986ea25b4c9" containerName="main" containerID="cri-o://9221409c6e9a68a85e610d4a12f79d98e717f4cc7a63ba3d6f60d794b1f21b67" gracePeriod=45 Nov 03 03:08:27.669086 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:27.669061 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-a0e72e/simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq" podUID="1901f57b-279b-4cbf-bf0c-054abc5786d4" containerName="main" containerID="cri-o://83278823364d3ba13b2c0f63ab403df213efbd2f7cd1e611afa52b2c2130f5d8" gracePeriod=45 Nov 03 03:08:27.760559 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:27.760529 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a0e72e/premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv" podUID="a9b9af44-ab6d-4caf-89af-c986ea25b4c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": dial tcp 10.134.0.40:8000: connect: connection refused" Nov 03 03:08:27.958698 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:27.958369 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a0e72e/distinct-w3-a0e72e-kserve-fb499cc4-2dfxc" Nov 03 03:08:28.010929 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.010708 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a0e72e/distinct-2-w3-a0e72e-kserve-59f58c59bf-jz5p8" Nov 03 03:08:28.014867 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.014816 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a0e72e/simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq" Nov 03 03:08:28.017519 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.017501 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a0e72e/premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv" Nov 03 03:08:28.022705 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.022683 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/308bca3b-9bd5-4724-9b95-acdbc978aa10-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/308bca3b-9bd5-4724-9b95-acdbc978aa10-tmp-dir\") pod \"308bca3b-9bd5-4724-9b95-acdbc978aa10\" (UID: \"308bca3b-9bd5-4724-9b95-acdbc978aa10\") " Nov 03 03:08:28.022814 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.022764 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/308bca3b-9bd5-4724-9b95-acdbc978aa10-home\" (UniqueName: \"kubernetes.io/empty-dir/308bca3b-9bd5-4724-9b95-acdbc978aa10-home\") pod \"308bca3b-9bd5-4724-9b95-acdbc978aa10\" (UID: \"308bca3b-9bd5-4724-9b95-acdbc978aa10\") " Nov 03 03:08:28.022932 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.022914 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/308bca3b-9bd5-4724-9b95-acdbc978aa10-kube-api-access-wx6w7\" (UniqueName: \"kubernetes.io/projected/308bca3b-9bd5-4724-9b95-acdbc978aa10-kube-api-access-wx6w7\") pod \"308bca3b-9bd5-4724-9b95-acdbc978aa10\" (UID: \"308bca3b-9bd5-4724-9b95-acdbc978aa10\") " Nov 03 03:08:28.023000 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.022947 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/308bca3b-9bd5-4724-9b95-acdbc978aa10-dshm\" (UniqueName: \"kubernetes.io/empty-dir/308bca3b-9bd5-4724-9b95-acdbc978aa10-dshm\") pod \"308bca3b-9bd5-4724-9b95-acdbc978aa10\" (UID: \"308bca3b-9bd5-4724-9b95-acdbc978aa10\") " Nov 03 03:08:28.023000 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.022931 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/308bca3b-9bd5-4724-9b95-acdbc978aa10-tmp-dir" pod "308bca3b-9bd5-4724-9b95-acdbc978aa10" (UID: "308bca3b-9bd5-4724-9b95-acdbc978aa10"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:28.023128 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.023053 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/308bca3b-9bd5-4724-9b95-acdbc978aa10-tls-certs\" (UniqueName: \"kubernetes.io/secret/308bca3b-9bd5-4724-9b95-acdbc978aa10-tls-certs\") pod \"308bca3b-9bd5-4724-9b95-acdbc978aa10\" (UID: \"308bca3b-9bd5-4724-9b95-acdbc978aa10\") " Nov 03 03:08:28.023128 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.023080 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/308bca3b-9bd5-4724-9b95-acdbc978aa10-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/308bca3b-9bd5-4724-9b95-acdbc978aa10-model-cache\") pod \"308bca3b-9bd5-4724-9b95-acdbc978aa10\" (UID: \"308bca3b-9bd5-4724-9b95-acdbc978aa10\") " Nov 03 03:08:28.023553 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.023535 5605 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/308bca3b-9bd5-4724-9b95-acdbc978aa10-tmp-dir\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:28.023669 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.023554 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/308bca3b-9bd5-4724-9b95-acdbc978aa10-home" pod "308bca3b-9bd5-4724-9b95-acdbc978aa10" (UID: "308bca3b-9bd5-4724-9b95-acdbc978aa10"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:28.023835 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.023811 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/308bca3b-9bd5-4724-9b95-acdbc978aa10-model-cache" pod "308bca3b-9bd5-4724-9b95-acdbc978aa10" (UID: "308bca3b-9bd5-4724-9b95-acdbc978aa10"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:28.025702 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.025675 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/308bca3b-9bd5-4724-9b95-acdbc978aa10-dshm" pod "308bca3b-9bd5-4724-9b95-acdbc978aa10" (UID: "308bca3b-9bd5-4724-9b95-acdbc978aa10"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:28.025782 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.025705 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/308bca3b-9bd5-4724-9b95-acdbc978aa10-tls-certs" pod "308bca3b-9bd5-4724-9b95-acdbc978aa10" (UID: "308bca3b-9bd5-4724-9b95-acdbc978aa10"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:08:28.025839 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.025782 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/308bca3b-9bd5-4724-9b95-acdbc978aa10-kube-api-access-wx6w7" pod "308bca3b-9bd5-4724-9b95-acdbc978aa10" (UID: "308bca3b-9bd5-4724-9b95-acdbc978aa10"). InnerVolumeSpecName "kube-api-access-wx6w7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:08:28.125038 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.125000 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9b9af44-ab6d-4caf-89af-c986ea25b4c9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9b9af44-ab6d-4caf-89af-c986ea25b4c9-model-cache\") pod \"a9b9af44-ab6d-4caf-89af-c986ea25b4c9\" (UID: \"a9b9af44-ab6d-4caf-89af-c986ea25b4c9\") " Nov 03 03:08:28.125207 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.125061 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70bd98e0-d8ac-47c2-901e-83f301d59a57-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70bd98e0-d8ac-47c2-901e-83f301d59a57-model-cache\") pod \"70bd98e0-d8ac-47c2-901e-83f301d59a57\" (UID: \"70bd98e0-d8ac-47c2-901e-83f301d59a57\") " Nov 03 03:08:28.125207 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.125102 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9b9af44-ab6d-4caf-89af-c986ea25b4c9-home\" (UniqueName: \"kubernetes.io/empty-dir/a9b9af44-ab6d-4caf-89af-c986ea25b4c9-home\") pod \"a9b9af44-ab6d-4caf-89af-c986ea25b4c9\" (UID: \"a9b9af44-ab6d-4caf-89af-c986ea25b4c9\") " Nov 03 03:08:28.125207 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.125149 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70bd98e0-d8ac-47c2-901e-83f301d59a57-kube-api-access-h9dcn\" (UniqueName: \"kubernetes.io/projected/70bd98e0-d8ac-47c2-901e-83f301d59a57-kube-api-access-h9dcn\") pod \"70bd98e0-d8ac-47c2-901e-83f301d59a57\" (UID: \"70bd98e0-d8ac-47c2-901e-83f301d59a57\") " Nov 03 03:08:28.125207 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.125192 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70bd98e0-d8ac-47c2-901e-83f301d59a57-home\" (UniqueName: \"kubernetes.io/empty-dir/70bd98e0-d8ac-47c2-901e-83f301d59a57-home\") pod \"70bd98e0-d8ac-47c2-901e-83f301d59a57\" (UID: \"70bd98e0-d8ac-47c2-901e-83f301d59a57\") " Nov 03 03:08:28.125454 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.125236 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9b9af44-ab6d-4caf-89af-c986ea25b4c9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9b9af44-ab6d-4caf-89af-c986ea25b4c9-dshm\") pod \"a9b9af44-ab6d-4caf-89af-c986ea25b4c9\" (UID: \"a9b9af44-ab6d-4caf-89af-c986ea25b4c9\") " Nov 03 03:08:28.125454 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.125255 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9b9af44-ab6d-4caf-89af-c986ea25b4c9-model-cache" pod "a9b9af44-ab6d-4caf-89af-c986ea25b4c9" (UID: "a9b9af44-ab6d-4caf-89af-c986ea25b4c9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:28.125454 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.125275 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1901f57b-279b-4cbf-bf0c-054abc5786d4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1901f57b-279b-4cbf-bf0c-054abc5786d4-dshm\") pod \"1901f57b-279b-4cbf-bf0c-054abc5786d4\" (UID: \"1901f57b-279b-4cbf-bf0c-054abc5786d4\") " Nov 03 03:08:28.125454 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.125311 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70bd98e0-d8ac-47c2-901e-83f301d59a57-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70bd98e0-d8ac-47c2-901e-83f301d59a57-tmp-dir\") pod \"70bd98e0-d8ac-47c2-901e-83f301d59a57\" (UID: \"70bd98e0-d8ac-47c2-901e-83f301d59a57\") " Nov 03 03:08:28.125454 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.125391 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9b9af44-ab6d-4caf-89af-c986ea25b4c9-tls-certs\" (UniqueName: \"kubernetes.io/secret/a9b9af44-ab6d-4caf-89af-c986ea25b4c9-tls-certs\") pod \"a9b9af44-ab6d-4caf-89af-c986ea25b4c9\" (UID: \"a9b9af44-ab6d-4caf-89af-c986ea25b4c9\") " Nov 03 03:08:28.125454 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.125430 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1901f57b-279b-4cbf-bf0c-054abc5786d4-tls-certs\" (UniqueName: \"kubernetes.io/secret/1901f57b-279b-4cbf-bf0c-054abc5786d4-tls-certs\") pod \"1901f57b-279b-4cbf-bf0c-054abc5786d4\" (UID: \"1901f57b-279b-4cbf-bf0c-054abc5786d4\") " Nov 03 03:08:28.125770 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.125468 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9b9af44-ab6d-4caf-89af-c986ea25b4c9-kube-api-access-mfpdh\" (UniqueName: \"kubernetes.io/projected/a9b9af44-ab6d-4caf-89af-c986ea25b4c9-kube-api-access-mfpdh\") pod \"a9b9af44-ab6d-4caf-89af-c986ea25b4c9\" (UID: \"a9b9af44-ab6d-4caf-89af-c986ea25b4c9\") " Nov 03 03:08:28.125770 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.125407 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70bd98e0-d8ac-47c2-901e-83f301d59a57-model-cache" pod "70bd98e0-d8ac-47c2-901e-83f301d59a57" (UID: "70bd98e0-d8ac-47c2-901e-83f301d59a57"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:28.125770 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.125468 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9b9af44-ab6d-4caf-89af-c986ea25b4c9-home" pod "a9b9af44-ab6d-4caf-89af-c986ea25b4c9" (UID: "a9b9af44-ab6d-4caf-89af-c986ea25b4c9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:28.125770 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.125534 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1901f57b-279b-4cbf-bf0c-054abc5786d4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1901f57b-279b-4cbf-bf0c-054abc5786d4-model-cache\") pod \"1901f57b-279b-4cbf-bf0c-054abc5786d4\" (UID: \"1901f57b-279b-4cbf-bf0c-054abc5786d4\") " Nov 03 03:08:28.125770 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.125565 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1901f57b-279b-4cbf-bf0c-054abc5786d4-kube-api-access-stlkh\" (UniqueName: \"kubernetes.io/projected/1901f57b-279b-4cbf-bf0c-054abc5786d4-kube-api-access-stlkh\") pod \"1901f57b-279b-4cbf-bf0c-054abc5786d4\" (UID: \"1901f57b-279b-4cbf-bf0c-054abc5786d4\") " Nov 03 03:08:28.125770 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.125604 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1901f57b-279b-4cbf-bf0c-054abc5786d4-home\" (UniqueName: \"kubernetes.io/empty-dir/1901f57b-279b-4cbf-bf0c-054abc5786d4-home\") pod \"1901f57b-279b-4cbf-bf0c-054abc5786d4\" (UID: \"1901f57b-279b-4cbf-bf0c-054abc5786d4\") " Nov 03 03:08:28.125770 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.125608 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70bd98e0-d8ac-47c2-901e-83f301d59a57-home" pod "70bd98e0-d8ac-47c2-901e-83f301d59a57" (UID: "70bd98e0-d8ac-47c2-901e-83f301d59a57"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:28.125770 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.125646 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1901f57b-279b-4cbf-bf0c-054abc5786d4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1901f57b-279b-4cbf-bf0c-054abc5786d4-tmp-dir\") pod \"1901f57b-279b-4cbf-bf0c-054abc5786d4\" (UID: \"1901f57b-279b-4cbf-bf0c-054abc5786d4\") " Nov 03 03:08:28.125770 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.125697 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70bd98e0-d8ac-47c2-901e-83f301d59a57-dshm\" (UniqueName: \"kubernetes.io/empty-dir/70bd98e0-d8ac-47c2-901e-83f301d59a57-dshm\") pod \"70bd98e0-d8ac-47c2-901e-83f301d59a57\" (UID: \"70bd98e0-d8ac-47c2-901e-83f301d59a57\") " Nov 03 03:08:28.126241 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.125782 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9b9af44-ab6d-4caf-89af-c986ea25b4c9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9b9af44-ab6d-4caf-89af-c986ea25b4c9-tmp-dir\") pod \"a9b9af44-ab6d-4caf-89af-c986ea25b4c9\" (UID: \"a9b9af44-ab6d-4caf-89af-c986ea25b4c9\") " Nov 03 03:08:28.126241 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.125791 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1901f57b-279b-4cbf-bf0c-054abc5786d4-model-cache" pod "1901f57b-279b-4cbf-bf0c-054abc5786d4" (UID: "1901f57b-279b-4cbf-bf0c-054abc5786d4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:28.126241 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.125848 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70bd98e0-d8ac-47c2-901e-83f301d59a57-tls-certs\" (UniqueName: \"kubernetes.io/secret/70bd98e0-d8ac-47c2-901e-83f301d59a57-tls-certs\") pod \"70bd98e0-d8ac-47c2-901e-83f301d59a57\" (UID: \"70bd98e0-d8ac-47c2-901e-83f301d59a57\") " Nov 03 03:08:28.126241 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.125996 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1901f57b-279b-4cbf-bf0c-054abc5786d4-home" pod "1901f57b-279b-4cbf-bf0c-054abc5786d4" (UID: "1901f57b-279b-4cbf-bf0c-054abc5786d4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:28.126464 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.126302 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1901f57b-279b-4cbf-bf0c-054abc5786d4-tmp-dir" pod "1901f57b-279b-4cbf-bf0c-054abc5786d4" (UID: "1901f57b-279b-4cbf-bf0c-054abc5786d4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:28.126542 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.126517 5605 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1901f57b-279b-4cbf-bf0c-054abc5786d4-model-cache\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:28.126598 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.126544 5605 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1901f57b-279b-4cbf-bf0c-054abc5786d4-home\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:28.126598 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.126560 5605 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1901f57b-279b-4cbf-bf0c-054abc5786d4-tmp-dir\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:28.126598 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.126576 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wx6w7\" (UniqueName: \"kubernetes.io/projected/308bca3b-9bd5-4724-9b95-acdbc978aa10-kube-api-access-wx6w7\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:28.126598 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.126591 5605 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/308bca3b-9bd5-4724-9b95-acdbc978aa10-dshm\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:28.127140 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.126605 5605 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9b9af44-ab6d-4caf-89af-c986ea25b4c9-model-cache\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:28.127140 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.126618 5605 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70bd98e0-d8ac-47c2-901e-83f301d59a57-model-cache\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:28.127140 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.126631 5605 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9b9af44-ab6d-4caf-89af-c986ea25b4c9-home\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:28.127140 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.126643 5605 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70bd98e0-d8ac-47c2-901e-83f301d59a57-home\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:28.127140 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.126657 5605 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/308bca3b-9bd5-4724-9b95-acdbc978aa10-tls-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:28.127140 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.126712 5605 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/308bca3b-9bd5-4724-9b95-acdbc978aa10-model-cache\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:28.127140 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.126730 5605 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/308bca3b-9bd5-4724-9b95-acdbc978aa10-home\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:28.127140 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.126839 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9b9af44-ab6d-4caf-89af-c986ea25b4c9-tmp-dir" pod "a9b9af44-ab6d-4caf-89af-c986ea25b4c9" (UID: "a9b9af44-ab6d-4caf-89af-c986ea25b4c9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:28.127140 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.126959 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70bd98e0-d8ac-47c2-901e-83f301d59a57-tmp-dir" pod "70bd98e0-d8ac-47c2-901e-83f301d59a57" (UID: "70bd98e0-d8ac-47c2-901e-83f301d59a57"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:28.128586 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.128560 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1901f57b-279b-4cbf-bf0c-054abc5786d4-kube-api-access-stlkh" pod "1901f57b-279b-4cbf-bf0c-054abc5786d4" (UID: "1901f57b-279b-4cbf-bf0c-054abc5786d4"). InnerVolumeSpecName "kube-api-access-stlkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:08:28.128738 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.128714 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1901f57b-279b-4cbf-bf0c-054abc5786d4-dshm" pod "1901f57b-279b-4cbf-bf0c-054abc5786d4" (UID: "1901f57b-279b-4cbf-bf0c-054abc5786d4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:28.128938 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.128921 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70bd98e0-d8ac-47c2-901e-83f301d59a57-dshm" pod "70bd98e0-d8ac-47c2-901e-83f301d59a57" (UID: "70bd98e0-d8ac-47c2-901e-83f301d59a57"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:28.129018 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.128960 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9b9af44-ab6d-4caf-89af-c986ea25b4c9-tls-certs" pod "a9b9af44-ab6d-4caf-89af-c986ea25b4c9" (UID: "a9b9af44-ab6d-4caf-89af-c986ea25b4c9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:08:28.129526 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.129493 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1901f57b-279b-4cbf-bf0c-054abc5786d4-tls-certs" pod "1901f57b-279b-4cbf-bf0c-054abc5786d4" (UID: "1901f57b-279b-4cbf-bf0c-054abc5786d4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:08:28.129861 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.129803 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9b9af44-ab6d-4caf-89af-c986ea25b4c9-dshm" pod "a9b9af44-ab6d-4caf-89af-c986ea25b4c9" (UID: "a9b9af44-ab6d-4caf-89af-c986ea25b4c9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:28.129952 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.129922 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70bd98e0-d8ac-47c2-901e-83f301d59a57-kube-api-access-h9dcn" pod "70bd98e0-d8ac-47c2-901e-83f301d59a57" (UID: "70bd98e0-d8ac-47c2-901e-83f301d59a57"). InnerVolumeSpecName "kube-api-access-h9dcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:08:28.129997 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.129947 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9b9af44-ab6d-4caf-89af-c986ea25b4c9-kube-api-access-mfpdh" pod "a9b9af44-ab6d-4caf-89af-c986ea25b4c9" (UID: "a9b9af44-ab6d-4caf-89af-c986ea25b4c9"). InnerVolumeSpecName "kube-api-access-mfpdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:08:28.130633 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.130616 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70bd98e0-d8ac-47c2-901e-83f301d59a57-tls-certs" pod "70bd98e0-d8ac-47c2-901e-83f301d59a57" (UID: "70bd98e0-d8ac-47c2-901e-83f301d59a57"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:08:28.228212 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.228140 5605 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70bd98e0-d8ac-47c2-901e-83f301d59a57-dshm\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:28.228212 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.228169 5605 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9b9af44-ab6d-4caf-89af-c986ea25b4c9-tmp-dir\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:28.228212 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.228180 5605 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70bd98e0-d8ac-47c2-901e-83f301d59a57-tls-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:28.228212 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.228189 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9dcn\" (UniqueName: \"kubernetes.io/projected/70bd98e0-d8ac-47c2-901e-83f301d59a57-kube-api-access-h9dcn\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:28.228212 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.228197 5605 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9b9af44-ab6d-4caf-89af-c986ea25b4c9-dshm\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:28.228212 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.228207 5605 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1901f57b-279b-4cbf-bf0c-054abc5786d4-dshm\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:28.228212 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.228214 5605 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70bd98e0-d8ac-47c2-901e-83f301d59a57-tmp-dir\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:28.228915 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.228229 5605 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9b9af44-ab6d-4caf-89af-c986ea25b4c9-tls-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:28.228915 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.228237 5605 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1901f57b-279b-4cbf-bf0c-054abc5786d4-tls-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:28.228915 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.228247 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfpdh\" (UniqueName: \"kubernetes.io/projected/a9b9af44-ab6d-4caf-89af-c986ea25b4c9-kube-api-access-mfpdh\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:28.228915 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.228257 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stlkh\" (UniqueName: \"kubernetes.io/projected/1901f57b-279b-4cbf-bf0c-054abc5786d4-kube-api-access-stlkh\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:28.233993 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.233951 5605 generic.go:356] "Generic (PLEG): container finished" podID="6b9772d3-3595-4ac3-b747-d7cbaae23c08" containerID="114ebee09aab92b2815ce5c7232f54ae0e5aaf077122ae644bc50db162642b7b" exitCode=0 Nov 03 03:08:28.234134 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.234044 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af76fd17-lhvd7" event={"ID":"6b9772d3-3595-4ac3-b747-d7cbaae23c08","Type":"ContainerDied","Data":"114ebee09aab92b2815ce5c7232f54ae0e5aaf077122ae644bc50db162642b7b"} Nov 03 03:08:28.235778 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.235750 5605 generic.go:356] "Generic (PLEG): container finished" podID="70bd98e0-d8ac-47c2-901e-83f301d59a57" containerID="81d3f5c16c98c27bdb8dfd4a2e963704733d3775640740eec7d25422c78230c9" exitCode=0 Nov 03 03:08:28.235909 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.235790 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a0e72e/distinct-2-w3-a0e72e-kserve-59f58c59bf-jz5p8" Nov 03 03:08:28.235909 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.235830 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a0e72e/distinct-2-w3-a0e72e-kserve-59f58c59bf-jz5p8" event={"ID":"70bd98e0-d8ac-47c2-901e-83f301d59a57","Type":"ContainerDied","Data":"81d3f5c16c98c27bdb8dfd4a2e963704733d3775640740eec7d25422c78230c9"} Nov 03 03:08:28.235909 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.235873 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a0e72e/distinct-2-w3-a0e72e-kserve-59f58c59bf-jz5p8" event={"ID":"70bd98e0-d8ac-47c2-901e-83f301d59a57","Type":"ContainerDied","Data":"1fe42bab462c3b011ebc822c0a5a50a303d8d41e8e5a40d2640df82c94211b90"} Nov 03 03:08:28.235909 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.235895 5605 scope.go:122] "RemoveContainer" containerID="81d3f5c16c98c27bdb8dfd4a2e963704733d3775640740eec7d25422c78230c9" Nov 03 03:08:28.237333 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.237299 5605 generic.go:356] "Generic (PLEG): container finished" podID="308bca3b-9bd5-4724-9b95-acdbc978aa10" containerID="1aa09c6571bd9571b9f8b05e504c098aab7c3adeae9acd73269599620cb9d477" exitCode=0 Nov 03 03:08:28.237461 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.237341 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a0e72e/distinct-w3-a0e72e-kserve-fb499cc4-2dfxc" Nov 03 03:08:28.237672 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.237337 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a0e72e/distinct-w3-a0e72e-kserve-fb499cc4-2dfxc" event={"ID":"308bca3b-9bd5-4724-9b95-acdbc978aa10","Type":"ContainerDied","Data":"1aa09c6571bd9571b9f8b05e504c098aab7c3adeae9acd73269599620cb9d477"} Nov 03 03:08:28.237672 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.237579 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a0e72e/distinct-w3-a0e72e-kserve-fb499cc4-2dfxc" event={"ID":"308bca3b-9bd5-4724-9b95-acdbc978aa10","Type":"ContainerDied","Data":"1a8528c05512f2b4fab4bab9ec5aaab81ef8be9d968ec01b1deb2806c911771a"} Nov 03 03:08:28.240466 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.240440 5605 generic.go:356] "Generic (PLEG): container finished" podID="a9b9af44-ab6d-4caf-89af-c986ea25b4c9" containerID="9221409c6e9a68a85e610d4a12f79d98e717f4cc7a63ba3d6f60d794b1f21b67" exitCode=0 Nov 03 03:08:28.240588 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.240483 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a0e72e/premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv" Nov 03 03:08:28.240588 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.240540 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a0e72e/premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv" event={"ID":"a9b9af44-ab6d-4caf-89af-c986ea25b4c9","Type":"ContainerDied","Data":"9221409c6e9a68a85e610d4a12f79d98e717f4cc7a63ba3d6f60d794b1f21b67"} Nov 03 03:08:28.240588 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.240570 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a0e72e/premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv" event={"ID":"a9b9af44-ab6d-4caf-89af-c986ea25b4c9","Type":"ContainerDied","Data":"0392f35f9cb3eca01159741a9db4ac42c87ec836e0048f986a9983ea98ce8468"} Nov 03 03:08:28.242207 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.242184 5605 generic.go:356] "Generic (PLEG): container finished" podID="1901f57b-279b-4cbf-bf0c-054abc5786d4" containerID="83278823364d3ba13b2c0f63ab403df213efbd2f7cd1e611afa52b2c2130f5d8" exitCode=0 Nov 03 03:08:28.242308 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.242224 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a0e72e/simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq" event={"ID":"1901f57b-279b-4cbf-bf0c-054abc5786d4","Type":"ContainerDied","Data":"83278823364d3ba13b2c0f63ab403df213efbd2f7cd1e611afa52b2c2130f5d8"} Nov 03 03:08:28.242308 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.242249 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a0e72e/simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq" event={"ID":"1901f57b-279b-4cbf-bf0c-054abc5786d4","Type":"ContainerDied","Data":"4bc681f89142799b7b29b94db7fdbd3d38e04d9b10fd2be959740e529a878426"} Nov 03 03:08:28.242308 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.242302 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a0e72e/simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq" Nov 03 03:08:28.254599 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.254581 5605 scope.go:122] "RemoveContainer" containerID="81d3f5c16c98c27bdb8dfd4a2e963704733d3775640740eec7d25422c78230c9" Nov 03 03:08:28.266613 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:08:28.266585 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81d3f5c16c98c27bdb8dfd4a2e963704733d3775640740eec7d25422c78230c9\": container with ID starting with 81d3f5c16c98c27bdb8dfd4a2e963704733d3775640740eec7d25422c78230c9 not found: ID does not exist" containerID="81d3f5c16c98c27bdb8dfd4a2e963704733d3775640740eec7d25422c78230c9" Nov 03 03:08:28.266706 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.266624 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81d3f5c16c98c27bdb8dfd4a2e963704733d3775640740eec7d25422c78230c9"} err="failed to get container status \"81d3f5c16c98c27bdb8dfd4a2e963704733d3775640740eec7d25422c78230c9\": rpc error: code = NotFound desc = could not find container \"81d3f5c16c98c27bdb8dfd4a2e963704733d3775640740eec7d25422c78230c9\": container with ID starting with 81d3f5c16c98c27bdb8dfd4a2e963704733d3775640740eec7d25422c78230c9 not found: ID does not exist" Nov 03 03:08:28.266706 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.266646 5605 scope.go:122] "RemoveContainer" containerID="1aa09c6571bd9571b9f8b05e504c098aab7c3adeae9acd73269599620cb9d477" Nov 03 03:08:28.275876 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.275857 5605 scope.go:122] "RemoveContainer" containerID="1aa09c6571bd9571b9f8b05e504c098aab7c3adeae9acd73269599620cb9d477" Nov 03 03:08:28.276138 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:08:28.276119 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1aa09c6571bd9571b9f8b05e504c098aab7c3adeae9acd73269599620cb9d477\": container with ID starting with 1aa09c6571bd9571b9f8b05e504c098aab7c3adeae9acd73269599620cb9d477 not found: ID does not exist" containerID="1aa09c6571bd9571b9f8b05e504c098aab7c3adeae9acd73269599620cb9d477" Nov 03 03:08:28.276190 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.276145 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1aa09c6571bd9571b9f8b05e504c098aab7c3adeae9acd73269599620cb9d477"} err="failed to get container status \"1aa09c6571bd9571b9f8b05e504c098aab7c3adeae9acd73269599620cb9d477\": rpc error: code = NotFound desc = could not find container \"1aa09c6571bd9571b9f8b05e504c098aab7c3adeae9acd73269599620cb9d477\": container with ID starting with 1aa09c6571bd9571b9f8b05e504c098aab7c3adeae9acd73269599620cb9d477 not found: ID does not exist" Nov 03 03:08:28.276190 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.276169 5605 scope.go:122] "RemoveContainer" containerID="9221409c6e9a68a85e610d4a12f79d98e717f4cc7a63ba3d6f60d794b1f21b67" Nov 03 03:08:28.285095 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.285071 5605 scope.go:122] "RemoveContainer" containerID="9221409c6e9a68a85e610d4a12f79d98e717f4cc7a63ba3d6f60d794b1f21b67" Nov 03 03:08:28.285382 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:08:28.285344 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9221409c6e9a68a85e610d4a12f79d98e717f4cc7a63ba3d6f60d794b1f21b67\": container with ID starting with 9221409c6e9a68a85e610d4a12f79d98e717f4cc7a63ba3d6f60d794b1f21b67 not found: ID does not exist" containerID="9221409c6e9a68a85e610d4a12f79d98e717f4cc7a63ba3d6f60d794b1f21b67" Nov 03 03:08:28.285442 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.285392 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9221409c6e9a68a85e610d4a12f79d98e717f4cc7a63ba3d6f60d794b1f21b67"} err="failed to get container status \"9221409c6e9a68a85e610d4a12f79d98e717f4cc7a63ba3d6f60d794b1f21b67\": rpc error: code = NotFound desc = could not find container \"9221409c6e9a68a85e610d4a12f79d98e717f4cc7a63ba3d6f60d794b1f21b67\": container with ID starting with 9221409c6e9a68a85e610d4a12f79d98e717f4cc7a63ba3d6f60d794b1f21b67 not found: ID does not exist" Nov 03 03:08:28.285442 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.285412 5605 scope.go:122] "RemoveContainer" containerID="83278823364d3ba13b2c0f63ab403df213efbd2f7cd1e611afa52b2c2130f5d8" Nov 03 03:08:28.294735 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.294720 5605 scope.go:122] "RemoveContainer" containerID="83278823364d3ba13b2c0f63ab403df213efbd2f7cd1e611afa52b2c2130f5d8" Nov 03 03:08:28.295004 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:08:28.294987 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83278823364d3ba13b2c0f63ab403df213efbd2f7cd1e611afa52b2c2130f5d8\": container with ID starting with 83278823364d3ba13b2c0f63ab403df213efbd2f7cd1e611afa52b2c2130f5d8 not found: ID does not exist" containerID="83278823364d3ba13b2c0f63ab403df213efbd2f7cd1e611afa52b2c2130f5d8" Nov 03 03:08:28.295054 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.295010 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83278823364d3ba13b2c0f63ab403df213efbd2f7cd1e611afa52b2c2130f5d8"} err="failed to get container status \"83278823364d3ba13b2c0f63ab403df213efbd2f7cd1e611afa52b2c2130f5d8\": rpc error: code = NotFound desc = could not find container \"83278823364d3ba13b2c0f63ab403df213efbd2f7cd1e611afa52b2c2130f5d8\": container with ID starting with 83278823364d3ba13b2c0f63ab403df213efbd2f7cd1e611afa52b2c2130f5d8 not found: ID does not exist" Nov 03 03:08:28.629604 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.629569 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-a0e72e/premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv"] Nov 03 03:08:28.632204 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.632179 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-a0e72e/premium-w3-a0e72e-kserve-5c5c9cd58c-8szzv"] Nov 03 03:08:28.695804 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.695779 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jgcjh"] Nov 03 03:08:28.696004 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.695983 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jgcjh" podUID="80871945-cbe2-40b4-8715-472c062fc27f" containerName="istio-proxy" containerID="cri-o://ec8627709f986d040ae797ce67bac125af0f73f36320ee6e8309cfc60c2e5de9" gracePeriod=30 Nov 03 03:08:28.702809 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.702785 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7dn2l"] Nov 03 03:08:28.703020 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.702988 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7dn2l" podUID="21200be4-59d6-4955-8d62-5394f4de28df" containerName="istio-proxy" containerID="cri-o://c7d79248d1b67b2fcf22e4ea304961bf1019daae99df51f335af48dd78b637ca" gracePeriod=30 Nov 03 03:08:28.705044 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.704990 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-644zf"] Nov 03 03:08:28.705394 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.705344 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-644zf" podUID="0b3468eb-1b4e-4727-910f-d975527c8644" containerName="istio-proxy" containerID="cri-o://8681bb515ff81651c2615f5062da8424e8fcd1ea05484a48046bf79e4fa44060" gracePeriod=30 Nov 03 03:08:28.707128 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.707106 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-flj4q"] Nov 03 03:08:28.707303 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.707283 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-flj4q" podUID="d76cc067-c62a-40e3-90d2-6435dc8cf655" containerName="istio-proxy" containerID="cri-o://cb3e796b406e3abaeef2192c6926a294dec1b9b8d73476089aa9b133a1dd7a24" gracePeriod=30 Nov 03 03:08:28.738922 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.738897 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-a0e72e/distinct-w3-a0e72e-kserve-fb499cc4-2dfxc"] Nov 03 03:08:28.743035 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.743017 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-a0e72e/distinct-w3-a0e72e-kserve-fb499cc4-2dfxc"] Nov 03 03:08:28.851211 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.851171 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="308bca3b-9bd5-4724-9b95-acdbc978aa10" path="/var/lib/kubelet/pods/308bca3b-9bd5-4724-9b95-acdbc978aa10/volumes" Nov 03 03:08:28.851677 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.851658 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9b9af44-ab6d-4caf-89af-c986ea25b4c9" path="/var/lib/kubelet/pods/a9b9af44-ab6d-4caf-89af-c986ea25b4c9/volumes" Nov 03 03:08:28.854711 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.854686 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-a0e72e/distinct-2-w3-a0e72e-kserve-59f58c59bf-jz5p8"] Nov 03 03:08:28.857078 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.857051 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-a0e72e/distinct-2-w3-a0e72e-kserve-59f58c59bf-jz5p8"] Nov 03 03:08:28.904181 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.904117 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a0e72e/simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq" podUID="1901f57b-279b-4cbf-bf0c-054abc5786d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:08:28.936621 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.936595 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a0e72e/distinct-2-w3-a0e72e-kserve-59f58c59bf-jz5p8" podUID="70bd98e0-d8ac-47c2-901e-83f301d59a57" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Nov 03 03:08:28.969585 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.969566 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-a0e72e/simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq"] Nov 03 03:08:28.977209 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:28.977183 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-a0e72e/simulator-w3-a0e72e-kserve-7498ff9dc4-wgtdq"] Nov 03 03:08:29.399894 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:29.399870 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af76fd17-lhvd7" Nov 03 03:08:29.439104 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:29.439080 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b9772d3-3595-4ac3-b747-d7cbaae23c08-kube-api-access-4j5gp\" (UniqueName: \"kubernetes.io/projected/6b9772d3-3595-4ac3-b747-d7cbaae23c08-kube-api-access-4j5gp\") pod \"6b9772d3-3595-4ac3-b747-d7cbaae23c08\" (UID: \"6b9772d3-3595-4ac3-b747-d7cbaae23c08\") " Nov 03 03:08:29.439228 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:29.439121 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b9772d3-3595-4ac3-b747-d7cbaae23c08-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b9772d3-3595-4ac3-b747-d7cbaae23c08-maas-api-openshift-ca\") pod \"6b9772d3-3595-4ac3-b747-d7cbaae23c08\" (UID: \"6b9772d3-3595-4ac3-b747-d7cbaae23c08\") " Nov 03 03:08:29.439228 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:29.439182 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b9772d3-3595-4ac3-b747-d7cbaae23c08-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b9772d3-3595-4ac3-b747-d7cbaae23c08-maas-api-xks-ca\") pod \"6b9772d3-3595-4ac3-b747-d7cbaae23c08\" (UID: \"6b9772d3-3595-4ac3-b747-d7cbaae23c08\") " Nov 03 03:08:29.439576 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:29.439553 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b9772d3-3595-4ac3-b747-d7cbaae23c08-maas-api-openshift-ca" pod "6b9772d3-3595-4ac3-b747-d7cbaae23c08" (UID: "6b9772d3-3595-4ac3-b747-d7cbaae23c08"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:08:29.439661 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:29.439643 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b9772d3-3595-4ac3-b747-d7cbaae23c08-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:29.441127 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:29.441107 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b9772d3-3595-4ac3-b747-d7cbaae23c08-maas-api-xks-ca" pod "6b9772d3-3595-4ac3-b747-d7cbaae23c08" (UID: "6b9772d3-3595-4ac3-b747-d7cbaae23c08"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:08:29.441127 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:29.441115 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b9772d3-3595-4ac3-b747-d7cbaae23c08-kube-api-access-4j5gp" pod "6b9772d3-3595-4ac3-b747-d7cbaae23c08" (UID: "6b9772d3-3595-4ac3-b747-d7cbaae23c08"). InnerVolumeSpecName "kube-api-access-4j5gp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:08:29.540946 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:29.540888 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4j5gp\" (UniqueName: \"kubernetes.io/projected/6b9772d3-3595-4ac3-b747-d7cbaae23c08-kube-api-access-4j5gp\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:29.540946 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:29.540913 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b9772d3-3595-4ac3-b747-d7cbaae23c08-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:30.258958 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:30.258919 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af76fd17-lhvd7" Nov 03 03:08:30.258958 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:30.258946 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af76fd17-lhvd7" event={"ID":"6b9772d3-3595-4ac3-b747-d7cbaae23c08","Type":"ContainerDied","Data":"0c8d1163f348c6f255309fbdc915cd293751c0134211aeabc3f52a1f57761b50"} Nov 03 03:08:30.259177 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:30.258995 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c8d1163f348c6f255309fbdc915cd293751c0134211aeabc3f52a1f57761b50" Nov 03 03:08:30.538930 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:30.538853 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-a0e72e-64b874b665-4k4v9" podUID="5c524f2c-3641-4719-b906-2434a016e677" containerName="payload-processing" containerID="cri-o://fb3e3dd8a330b2f38c9c8bd5d2ac7a81845b0246797e3ea2f62dcc5d68681d56" gracePeriod=25 Nov 03 03:08:30.850456 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:30.850417 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1901f57b-279b-4cbf-bf0c-054abc5786d4" path="/var/lib/kubelet/pods/1901f57b-279b-4cbf-bf0c-054abc5786d4/volumes" Nov 03 03:08:30.850867 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:30.850853 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70bd98e0-d8ac-47c2-901e-83f301d59a57" path="/var/lib/kubelet/pods/70bd98e0-d8ac-47c2-901e-83f301d59a57/volumes" Nov 03 03:08:31.139719 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:31.139630 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-7dn2l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 03:08:31.139719 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:31.139692 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7dn2l" podUID="21200be4-59d6-4955-8d62-5394f4de28df" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 03:08:31.175527 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:31.175492 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-jgcjh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 03:08:31.175704 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:31.175551 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jgcjh" podUID="80871945-cbe2-40b4-8715-472c062fc27f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 03:08:31.372006 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:31.371966 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-644zf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 03:08:31.372163 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:31.372029 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-644zf" podUID="0b3468eb-1b4e-4727-910f-d975527c8644" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 03:08:31.464560 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:31.464471 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-a0e72e-64b874b665-4k4v9" podUID="5c524f2c-3641-4719-b906-2434a016e677" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:08:32.124728 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:32.124696 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-flj4q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 03:08:32.125133 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:32.124752 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-flj4q" podUID="d76cc067-c62a-40e3-90d2-6435dc8cf655" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 03:08:33.969808 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:33.969742 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7dn2l" Nov 03 03:08:34.003033 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.003003 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jgcjh" Nov 03 03:08:34.042031 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.042005 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-flj4q" Nov 03 03:08:34.056966 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.056942 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-644zf" Nov 03 03:08:34.088245 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.088218 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80871945-cbe2-40b4-8715-472c062fc27f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80871945-cbe2-40b4-8715-472c062fc27f-istio-data\") pod \"80871945-cbe2-40b4-8715-472c062fc27f\" (UID: \"80871945-cbe2-40b4-8715-472c062fc27f\") " Nov 03 03:08:34.088443 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.088251 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21200be4-59d6-4955-8d62-5394f4de28df-istio-token\" (UniqueName: \"kubernetes.io/projected/21200be4-59d6-4955-8d62-5394f4de28df-istio-token\") pod \"21200be4-59d6-4955-8d62-5394f4de28df\" (UID: \"21200be4-59d6-4955-8d62-5394f4de28df\") " Nov 03 03:08:34.088443 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.088281 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21200be4-59d6-4955-8d62-5394f4de28df-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21200be4-59d6-4955-8d62-5394f4de28df-workload-socket\") pod \"21200be4-59d6-4955-8d62-5394f4de28df\" (UID: \"21200be4-59d6-4955-8d62-5394f4de28df\") " Nov 03 03:08:34.088443 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.088322 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21200be4-59d6-4955-8d62-5394f4de28df-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21200be4-59d6-4955-8d62-5394f4de28df-istio-data\") pod \"21200be4-59d6-4955-8d62-5394f4de28df\" (UID: \"21200be4-59d6-4955-8d62-5394f4de28df\") " Nov 03 03:08:34.088443 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.088342 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80871945-cbe2-40b4-8715-472c062fc27f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80871945-cbe2-40b4-8715-472c062fc27f-credential-socket\") pod \"80871945-cbe2-40b4-8715-472c062fc27f\" (UID: \"80871945-cbe2-40b4-8715-472c062fc27f\") " Nov 03 03:08:34.088443 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.088401 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80871945-cbe2-40b4-8715-472c062fc27f-istio-token\" (UniqueName: \"kubernetes.io/projected/80871945-cbe2-40b4-8715-472c062fc27f-istio-token\") pod \"80871945-cbe2-40b4-8715-472c062fc27f\" (UID: \"80871945-cbe2-40b4-8715-472c062fc27f\") " Nov 03 03:08:34.088443 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.088432 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21200be4-59d6-4955-8d62-5394f4de28df-kube-api-access-749g6\" (UniqueName: \"kubernetes.io/projected/21200be4-59d6-4955-8d62-5394f4de28df-kube-api-access-749g6\") pod \"21200be4-59d6-4955-8d62-5394f4de28df\" (UID: \"21200be4-59d6-4955-8d62-5394f4de28df\") " Nov 03 03:08:34.088812 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.088468 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d76cc067-c62a-40e3-90d2-6435dc8cf655-kube-api-access-62qbl\" (UniqueName: \"kubernetes.io/projected/d76cc067-c62a-40e3-90d2-6435dc8cf655-kube-api-access-62qbl\") pod \"d76cc067-c62a-40e3-90d2-6435dc8cf655\" (UID: \"d76cc067-c62a-40e3-90d2-6435dc8cf655\") " Nov 03 03:08:34.088812 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.088533 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d76cc067-c62a-40e3-90d2-6435dc8cf655-istio-token\" (UniqueName: \"kubernetes.io/projected/d76cc067-c62a-40e3-90d2-6435dc8cf655-istio-token\") pod \"d76cc067-c62a-40e3-90d2-6435dc8cf655\" (UID: \"d76cc067-c62a-40e3-90d2-6435dc8cf655\") " Nov 03 03:08:34.088812 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.088565 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80871945-cbe2-40b4-8715-472c062fc27f-istio-data" pod "80871945-cbe2-40b4-8715-472c062fc27f" (UID: "80871945-cbe2-40b4-8715-472c062fc27f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:34.088812 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.088575 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21200be4-59d6-4955-8d62-5394f4de28df-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21200be4-59d6-4955-8d62-5394f4de28df-istio-envoy\") pod \"21200be4-59d6-4955-8d62-5394f4de28df\" (UID: \"21200be4-59d6-4955-8d62-5394f4de28df\") " Nov 03 03:08:34.088812 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.088633 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80871945-cbe2-40b4-8715-472c062fc27f-credential-socket" pod "80871945-cbe2-40b4-8715-472c062fc27f" (UID: "80871945-cbe2-40b4-8715-472c062fc27f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:34.088812 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.088667 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80871945-cbe2-40b4-8715-472c062fc27f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80871945-cbe2-40b4-8715-472c062fc27f-workload-socket\") pod \"80871945-cbe2-40b4-8715-472c062fc27f\" (UID: \"80871945-cbe2-40b4-8715-472c062fc27f\") " Nov 03 03:08:34.088812 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.088671 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21200be4-59d6-4955-8d62-5394f4de28df-istio-data" pod "21200be4-59d6-4955-8d62-5394f4de28df" (UID: "21200be4-59d6-4955-8d62-5394f4de28df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:34.088812 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.088715 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d76cc067-c62a-40e3-90d2-6435dc8cf655-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d76cc067-c62a-40e3-90d2-6435dc8cf655-istio-data\") pod \"d76cc067-c62a-40e3-90d2-6435dc8cf655\" (UID: \"d76cc067-c62a-40e3-90d2-6435dc8cf655\") " Nov 03 03:08:34.088812 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.088751 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d76cc067-c62a-40e3-90d2-6435dc8cf655-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d76cc067-c62a-40e3-90d2-6435dc8cf655-workload-certs\") pod \"d76cc067-c62a-40e3-90d2-6435dc8cf655\" (UID: \"d76cc067-c62a-40e3-90d2-6435dc8cf655\") " Nov 03 03:08:34.088812 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.088813 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80871945-cbe2-40b4-8715-472c062fc27f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80871945-cbe2-40b4-8715-472c062fc27f-istio-podinfo\") pod \"80871945-cbe2-40b4-8715-472c062fc27f\" (UID: \"80871945-cbe2-40b4-8715-472c062fc27f\") " Nov 03 03:08:34.089337 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.088842 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d76cc067-c62a-40e3-90d2-6435dc8cf655-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d76cc067-c62a-40e3-90d2-6435dc8cf655-istiod-ca-cert\") pod \"d76cc067-c62a-40e3-90d2-6435dc8cf655\" (UID: \"d76cc067-c62a-40e3-90d2-6435dc8cf655\") " Nov 03 03:08:34.089337 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.088882 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d76cc067-c62a-40e3-90d2-6435dc8cf655-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d76cc067-c62a-40e3-90d2-6435dc8cf655-credential-socket\") pod \"d76cc067-c62a-40e3-90d2-6435dc8cf655\" (UID: \"d76cc067-c62a-40e3-90d2-6435dc8cf655\") " Nov 03 03:08:34.089337 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.088916 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80871945-cbe2-40b4-8715-472c062fc27f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80871945-cbe2-40b4-8715-472c062fc27f-istio-envoy\") pod \"80871945-cbe2-40b4-8715-472c062fc27f\" (UID: \"80871945-cbe2-40b4-8715-472c062fc27f\") " Nov 03 03:08:34.089337 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.088925 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21200be4-59d6-4955-8d62-5394f4de28df-workload-socket" pod "21200be4-59d6-4955-8d62-5394f4de28df" (UID: "21200be4-59d6-4955-8d62-5394f4de28df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:34.089337 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.088971 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d76cc067-c62a-40e3-90d2-6435dc8cf655-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d76cc067-c62a-40e3-90d2-6435dc8cf655-istio-envoy\") pod \"d76cc067-c62a-40e3-90d2-6435dc8cf655\" (UID: \"d76cc067-c62a-40e3-90d2-6435dc8cf655\") " Nov 03 03:08:34.089337 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.089003 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80871945-cbe2-40b4-8715-472c062fc27f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80871945-cbe2-40b4-8715-472c062fc27f-istiod-ca-cert\") pod \"80871945-cbe2-40b4-8715-472c062fc27f\" (UID: \"80871945-cbe2-40b4-8715-472c062fc27f\") " Nov 03 03:08:34.089337 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.089047 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/21200be4-59d6-4955-8d62-5394f4de28df-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21200be4-59d6-4955-8d62-5394f4de28df-istio-podinfo\") pod \"21200be4-59d6-4955-8d62-5394f4de28df\" (UID: \"21200be4-59d6-4955-8d62-5394f4de28df\") " Nov 03 03:08:34.089337 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.089080 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21200be4-59d6-4955-8d62-5394f4de28df-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21200be4-59d6-4955-8d62-5394f4de28df-istiod-ca-cert\") pod \"21200be4-59d6-4955-8d62-5394f4de28df\" (UID: \"21200be4-59d6-4955-8d62-5394f4de28df\") " Nov 03 03:08:34.089337 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.089118 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d76cc067-c62a-40e3-90d2-6435dc8cf655-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d76cc067-c62a-40e3-90d2-6435dc8cf655-istio-podinfo\") pod \"d76cc067-c62a-40e3-90d2-6435dc8cf655\" (UID: \"d76cc067-c62a-40e3-90d2-6435dc8cf655\") " Nov 03 03:08:34.089337 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.089143 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d76cc067-c62a-40e3-90d2-6435dc8cf655-workload-certs" pod "d76cc067-c62a-40e3-90d2-6435dc8cf655" (UID: "d76cc067-c62a-40e3-90d2-6435dc8cf655"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:34.089337 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.089172 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80871945-cbe2-40b4-8715-472c062fc27f-kube-api-access-2trql\" (UniqueName: \"kubernetes.io/projected/80871945-cbe2-40b4-8715-472c062fc27f-kube-api-access-2trql\") pod \"80871945-cbe2-40b4-8715-472c062fc27f\" (UID: \"80871945-cbe2-40b4-8715-472c062fc27f\") " Nov 03 03:08:34.089337 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.089218 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80871945-cbe2-40b4-8715-472c062fc27f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80871945-cbe2-40b4-8715-472c062fc27f-workload-certs\") pod \"80871945-cbe2-40b4-8715-472c062fc27f\" (UID: \"80871945-cbe2-40b4-8715-472c062fc27f\") " Nov 03 03:08:34.089337 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.089248 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d76cc067-c62a-40e3-90d2-6435dc8cf655-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d76cc067-c62a-40e3-90d2-6435dc8cf655-workload-socket\") pod \"d76cc067-c62a-40e3-90d2-6435dc8cf655\" (UID: \"d76cc067-c62a-40e3-90d2-6435dc8cf655\") " Nov 03 03:08:34.089337 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.089300 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21200be4-59d6-4955-8d62-5394f4de28df-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21200be4-59d6-4955-8d62-5394f4de28df-credential-socket\") pod \"21200be4-59d6-4955-8d62-5394f4de28df\" (UID: \"21200be4-59d6-4955-8d62-5394f4de28df\") " Nov 03 03:08:34.089337 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.089338 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21200be4-59d6-4955-8d62-5394f4de28df-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21200be4-59d6-4955-8d62-5394f4de28df-workload-certs\") pod \"21200be4-59d6-4955-8d62-5394f4de28df\" (UID: \"21200be4-59d6-4955-8d62-5394f4de28df\") " Nov 03 03:08:34.090320 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.089541 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80871945-cbe2-40b4-8715-472c062fc27f-workload-socket" pod "80871945-cbe2-40b4-8715-472c062fc27f" (UID: "80871945-cbe2-40b4-8715-472c062fc27f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:34.090320 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.089823 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80871945-cbe2-40b4-8715-472c062fc27f-workload-certs" pod "80871945-cbe2-40b4-8715-472c062fc27f" (UID: "80871945-cbe2-40b4-8715-472c062fc27f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:34.090320 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.089917 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80871945-cbe2-40b4-8715-472c062fc27f-istiod-ca-cert" pod "80871945-cbe2-40b4-8715-472c062fc27f" (UID: "80871945-cbe2-40b4-8715-472c062fc27f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:08:34.090320 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.089924 5605 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d76cc067-c62a-40e3-90d2-6435dc8cf655-workload-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:34.090320 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.089960 5605 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80871945-cbe2-40b4-8715-472c062fc27f-workload-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:34.090320 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.089981 5605 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80871945-cbe2-40b4-8715-472c062fc27f-istio-data\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:34.090320 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.089996 5605 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21200be4-59d6-4955-8d62-5394f4de28df-workload-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:34.090320 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.090011 5605 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21200be4-59d6-4955-8d62-5394f4de28df-istio-data\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:34.090320 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.090026 5605 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80871945-cbe2-40b4-8715-472c062fc27f-credential-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:34.090320 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.090041 5605 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80871945-cbe2-40b4-8715-472c062fc27f-workload-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:34.090320 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.090138 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d76cc067-c62a-40e3-90d2-6435dc8cf655-istio-data" pod "d76cc067-c62a-40e3-90d2-6435dc8cf655" (UID: "d76cc067-c62a-40e3-90d2-6435dc8cf655"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:34.090320 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.090150 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d76cc067-c62a-40e3-90d2-6435dc8cf655-workload-socket" pod "d76cc067-c62a-40e3-90d2-6435dc8cf655" (UID: "d76cc067-c62a-40e3-90d2-6435dc8cf655"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:34.091122 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.090352 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21200be4-59d6-4955-8d62-5394f4de28df-credential-socket" pod "21200be4-59d6-4955-8d62-5394f4de28df" (UID: "21200be4-59d6-4955-8d62-5394f4de28df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:34.091122 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.090561 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21200be4-59d6-4955-8d62-5394f4de28df-workload-certs" pod "21200be4-59d6-4955-8d62-5394f4de28df" (UID: "21200be4-59d6-4955-8d62-5394f4de28df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:34.091122 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.091094 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d76cc067-c62a-40e3-90d2-6435dc8cf655-istiod-ca-cert" pod "d76cc067-c62a-40e3-90d2-6435dc8cf655" (UID: "d76cc067-c62a-40e3-90d2-6435dc8cf655"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:08:34.091293 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.091154 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d76cc067-c62a-40e3-90d2-6435dc8cf655-credential-socket" pod "d76cc067-c62a-40e3-90d2-6435dc8cf655" (UID: "d76cc067-c62a-40e3-90d2-6435dc8cf655"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:34.091293 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.091218 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21200be4-59d6-4955-8d62-5394f4de28df-istiod-ca-cert" pod "21200be4-59d6-4955-8d62-5394f4de28df" (UID: "21200be4-59d6-4955-8d62-5394f4de28df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:08:34.091895 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.091858 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21200be4-59d6-4955-8d62-5394f4de28df-istio-token" pod "21200be4-59d6-4955-8d62-5394f4de28df" (UID: "21200be4-59d6-4955-8d62-5394f4de28df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:08:34.092886 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.092582 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21200be4-59d6-4955-8d62-5394f4de28df-istio-envoy" pod "21200be4-59d6-4955-8d62-5394f4de28df" (UID: "21200be4-59d6-4955-8d62-5394f4de28df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:34.092886 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.092712 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d76cc067-c62a-40e3-90d2-6435dc8cf655-istio-token" pod "d76cc067-c62a-40e3-90d2-6435dc8cf655" (UID: "d76cc067-c62a-40e3-90d2-6435dc8cf655"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:08:34.092886 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.092744 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80871945-cbe2-40b4-8715-472c062fc27f-istio-token" pod "80871945-cbe2-40b4-8715-472c062fc27f" (UID: "80871945-cbe2-40b4-8715-472c062fc27f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:08:34.092886 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.092776 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d76cc067-c62a-40e3-90d2-6435dc8cf655-kube-api-access-62qbl" pod "d76cc067-c62a-40e3-90d2-6435dc8cf655" (UID: "d76cc067-c62a-40e3-90d2-6435dc8cf655"). InnerVolumeSpecName "kube-api-access-62qbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:08:34.092886 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.092854 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21200be4-59d6-4955-8d62-5394f4de28df-kube-api-access-749g6" pod "21200be4-59d6-4955-8d62-5394f4de28df" (UID: "21200be4-59d6-4955-8d62-5394f4de28df"). InnerVolumeSpecName "kube-api-access-749g6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:08:34.093247 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.092926 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80871945-cbe2-40b4-8715-472c062fc27f-istio-podinfo" pod "80871945-cbe2-40b4-8715-472c062fc27f" (UID: "80871945-cbe2-40b4-8715-472c062fc27f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:08:34.093315 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.093270 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d76cc067-c62a-40e3-90d2-6435dc8cf655-istio-podinfo" pod "d76cc067-c62a-40e3-90d2-6435dc8cf655" (UID: "d76cc067-c62a-40e3-90d2-6435dc8cf655"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:08:34.093315 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.093278 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21200be4-59d6-4955-8d62-5394f4de28df-istio-podinfo" pod "21200be4-59d6-4955-8d62-5394f4de28df" (UID: "21200be4-59d6-4955-8d62-5394f4de28df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:08:34.093476 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.093454 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d76cc067-c62a-40e3-90d2-6435dc8cf655-istio-envoy" pod "d76cc067-c62a-40e3-90d2-6435dc8cf655" (UID: "d76cc067-c62a-40e3-90d2-6435dc8cf655"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:34.093550 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.093535 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80871945-cbe2-40b4-8715-472c062fc27f-istio-envoy" pod "80871945-cbe2-40b4-8715-472c062fc27f" (UID: "80871945-cbe2-40b4-8715-472c062fc27f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:34.094044 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.094020 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80871945-cbe2-40b4-8715-472c062fc27f-kube-api-access-2trql" pod "80871945-cbe2-40b4-8715-472c062fc27f" (UID: "80871945-cbe2-40b4-8715-472c062fc27f"). InnerVolumeSpecName "kube-api-access-2trql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:08:34.190659 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.190628 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b3468eb-1b4e-4727-910f-d975527c8644-istio-token\" (UniqueName: \"kubernetes.io/projected/0b3468eb-1b4e-4727-910f-d975527c8644-istio-token\") pod \"0b3468eb-1b4e-4727-910f-d975527c8644\" (UID: \"0b3468eb-1b4e-4727-910f-d975527c8644\") " Nov 03 03:08:34.190865 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.190666 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b3468eb-1b4e-4727-910f-d975527c8644-kube-api-access-mcsz6\" (UniqueName: \"kubernetes.io/projected/0b3468eb-1b4e-4727-910f-d975527c8644-kube-api-access-mcsz6\") pod \"0b3468eb-1b4e-4727-910f-d975527c8644\" (UID: \"0b3468eb-1b4e-4727-910f-d975527c8644\") " Nov 03 03:08:34.190865 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.190689 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b3468eb-1b4e-4727-910f-d975527c8644-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b3468eb-1b4e-4727-910f-d975527c8644-credential-socket\") pod \"0b3468eb-1b4e-4727-910f-d975527c8644\" (UID: \"0b3468eb-1b4e-4727-910f-d975527c8644\") " Nov 03 03:08:34.190865 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.190810 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b3468eb-1b4e-4727-910f-d975527c8644-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b3468eb-1b4e-4727-910f-d975527c8644-istiod-ca-cert\") pod \"0b3468eb-1b4e-4727-910f-d975527c8644\" (UID: \"0b3468eb-1b4e-4727-910f-d975527c8644\") " Nov 03 03:08:34.191050 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.190915 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b3468eb-1b4e-4727-910f-d975527c8644-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b3468eb-1b4e-4727-910f-d975527c8644-workload-certs\") pod \"0b3468eb-1b4e-4727-910f-d975527c8644\" (UID: \"0b3468eb-1b4e-4727-910f-d975527c8644\") " Nov 03 03:08:34.191050 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.190977 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b3468eb-1b4e-4727-910f-d975527c8644-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b3468eb-1b4e-4727-910f-d975527c8644-istio-data\") pod \"0b3468eb-1b4e-4727-910f-d975527c8644\" (UID: \"0b3468eb-1b4e-4727-910f-d975527c8644\") " Nov 03 03:08:34.191050 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.191019 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0b3468eb-1b4e-4727-910f-d975527c8644-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b3468eb-1b4e-4727-910f-d975527c8644-istio-podinfo\") pod \"0b3468eb-1b4e-4727-910f-d975527c8644\" (UID: \"0b3468eb-1b4e-4727-910f-d975527c8644\") " Nov 03 03:08:34.191050 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.191031 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b3468eb-1b4e-4727-910f-d975527c8644-credential-socket" pod "0b3468eb-1b4e-4727-910f-d975527c8644" (UID: "0b3468eb-1b4e-4727-910f-d975527c8644"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:34.191283 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.191056 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b3468eb-1b4e-4727-910f-d975527c8644-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b3468eb-1b4e-4727-910f-d975527c8644-workload-socket\") pod \"0b3468eb-1b4e-4727-910f-d975527c8644\" (UID: \"0b3468eb-1b4e-4727-910f-d975527c8644\") " Nov 03 03:08:34.191283 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.191101 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b3468eb-1b4e-4727-910f-d975527c8644-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b3468eb-1b4e-4727-910f-d975527c8644-istio-envoy\") pod \"0b3468eb-1b4e-4727-910f-d975527c8644\" (UID: \"0b3468eb-1b4e-4727-910f-d975527c8644\") " Nov 03 03:08:34.191283 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.191210 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b3468eb-1b4e-4727-910f-d975527c8644-istio-data" pod "0b3468eb-1b4e-4727-910f-d975527c8644" (UID: "0b3468eb-1b4e-4727-910f-d975527c8644"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:34.191283 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.191230 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b3468eb-1b4e-4727-910f-d975527c8644-istiod-ca-cert" pod "0b3468eb-1b4e-4727-910f-d975527c8644" (UID: "0b3468eb-1b4e-4727-910f-d975527c8644"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:08:34.191522 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.191319 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b3468eb-1b4e-4727-910f-d975527c8644-workload-socket" pod "0b3468eb-1b4e-4727-910f-d975527c8644" (UID: "0b3468eb-1b4e-4727-910f-d975527c8644"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:34.191522 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.191408 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b3468eb-1b4e-4727-910f-d975527c8644-workload-certs" pod "0b3468eb-1b4e-4727-910f-d975527c8644" (UID: "0b3468eb-1b4e-4727-910f-d975527c8644"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:34.191671 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.191650 5605 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d76cc067-c62a-40e3-90d2-6435dc8cf655-istio-envoy\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:34.191738 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.191678 5605 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80871945-cbe2-40b4-8715-472c062fc27f-istiod-ca-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:34.191738 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.191695 5605 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21200be4-59d6-4955-8d62-5394f4de28df-istio-podinfo\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:34.191738 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.191711 5605 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21200be4-59d6-4955-8d62-5394f4de28df-istiod-ca-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:34.191738 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.191729 5605 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b3468eb-1b4e-4727-910f-d975527c8644-credential-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:34.191960 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.191743 5605 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d76cc067-c62a-40e3-90d2-6435dc8cf655-istio-podinfo\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:34.191960 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.191759 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2trql\" (UniqueName: \"kubernetes.io/projected/80871945-cbe2-40b4-8715-472c062fc27f-kube-api-access-2trql\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:34.191960 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.191776 5605 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d76cc067-c62a-40e3-90d2-6435dc8cf655-workload-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:34.191960 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.191792 5605 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21200be4-59d6-4955-8d62-5394f4de28df-credential-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:34.191960 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.191807 5605 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21200be4-59d6-4955-8d62-5394f4de28df-workload-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:34.191960 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.191820 5605 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b3468eb-1b4e-4727-910f-d975527c8644-istiod-ca-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:34.191960 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.191834 5605 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21200be4-59d6-4955-8d62-5394f4de28df-istio-token\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:34.191960 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.191846 5605 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b3468eb-1b4e-4727-910f-d975527c8644-workload-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:34.191960 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.191859 5605 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80871945-cbe2-40b4-8715-472c062fc27f-istio-token\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:34.191960 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.191873 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-749g6\" (UniqueName: \"kubernetes.io/projected/21200be4-59d6-4955-8d62-5394f4de28df-kube-api-access-749g6\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:34.191960 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.191887 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62qbl\" (UniqueName: \"kubernetes.io/projected/d76cc067-c62a-40e3-90d2-6435dc8cf655-kube-api-access-62qbl\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:34.191960 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.191900 5605 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b3468eb-1b4e-4727-910f-d975527c8644-istio-data\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:34.191960 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.191914 5605 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b3468eb-1b4e-4727-910f-d975527c8644-workload-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:34.191960 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.191928 5605 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d76cc067-c62a-40e3-90d2-6435dc8cf655-istio-token\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:34.191960 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.191943 5605 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21200be4-59d6-4955-8d62-5394f4de28df-istio-envoy\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:34.191960 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.191955 5605 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d76cc067-c62a-40e3-90d2-6435dc8cf655-istio-data\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:34.192822 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.191969 5605 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80871945-cbe2-40b4-8715-472c062fc27f-istio-podinfo\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:34.192822 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.191984 5605 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d76cc067-c62a-40e3-90d2-6435dc8cf655-istiod-ca-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:34.192822 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.191998 5605 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d76cc067-c62a-40e3-90d2-6435dc8cf655-credential-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:34.192822 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.192011 5605 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80871945-cbe2-40b4-8715-472c062fc27f-istio-envoy\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:34.193277 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.193251 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b3468eb-1b4e-4727-910f-d975527c8644-kube-api-access-mcsz6" pod "0b3468eb-1b4e-4727-910f-d975527c8644" (UID: "0b3468eb-1b4e-4727-910f-d975527c8644"). InnerVolumeSpecName "kube-api-access-mcsz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:08:34.193443 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.193274 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b3468eb-1b4e-4727-910f-d975527c8644-istio-token" pod "0b3468eb-1b4e-4727-910f-d975527c8644" (UID: "0b3468eb-1b4e-4727-910f-d975527c8644"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:08:34.193526 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.193460 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b3468eb-1b4e-4727-910f-d975527c8644-istio-podinfo" pod "0b3468eb-1b4e-4727-910f-d975527c8644" (UID: "0b3468eb-1b4e-4727-910f-d975527c8644"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:08:34.193526 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.193476 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b3468eb-1b4e-4727-910f-d975527c8644-istio-envoy" pod "0b3468eb-1b4e-4727-910f-d975527c8644" (UID: "0b3468eb-1b4e-4727-910f-d975527c8644"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:34.282463 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.282381 5605 generic.go:356] "Generic (PLEG): container finished" podID="0b3468eb-1b4e-4727-910f-d975527c8644" containerID="8681bb515ff81651c2615f5062da8424e8fcd1ea05484a48046bf79e4fa44060" exitCode=0 Nov 03 03:08:34.282463 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.282426 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-644zf" Nov 03 03:08:34.282463 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.282440 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-644zf" event={"ID":"0b3468eb-1b4e-4727-910f-d975527c8644","Type":"ContainerDied","Data":"8681bb515ff81651c2615f5062da8424e8fcd1ea05484a48046bf79e4fa44060"} Nov 03 03:08:34.282705 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.282502 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-644zf" event={"ID":"0b3468eb-1b4e-4727-910f-d975527c8644","Type":"ContainerDied","Data":"e60ec527a6fa16fd68d585442528876833fccc4d3c32be54ed7c61ac7cc792ac"} Nov 03 03:08:34.282705 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.282533 5605 scope.go:122] "RemoveContainer" containerID="8681bb515ff81651c2615f5062da8424e8fcd1ea05484a48046bf79e4fa44060" Nov 03 03:08:34.284126 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.283995 5605 generic.go:356] "Generic (PLEG): container finished" podID="d76cc067-c62a-40e3-90d2-6435dc8cf655" containerID="cb3e796b406e3abaeef2192c6926a294dec1b9b8d73476089aa9b133a1dd7a24" exitCode=0 Nov 03 03:08:34.284126 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.284025 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-flj4q" event={"ID":"d76cc067-c62a-40e3-90d2-6435dc8cf655","Type":"ContainerDied","Data":"cb3e796b406e3abaeef2192c6926a294dec1b9b8d73476089aa9b133a1dd7a24"} Nov 03 03:08:34.284126 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.284049 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-flj4q" Nov 03 03:08:34.284126 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.284081 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-flj4q" event={"ID":"d76cc067-c62a-40e3-90d2-6435dc8cf655","Type":"ContainerDied","Data":"4dbfa3d5a846c421d43c3dcb846d65ad91be41e36ec19fa05c6194074199e6b7"} Nov 03 03:08:34.285753 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.285730 5605 generic.go:356] "Generic (PLEG): container finished" podID="80871945-cbe2-40b4-8715-472c062fc27f" containerID="ec8627709f986d040ae797ce67bac125af0f73f36320ee6e8309cfc60c2e5de9" exitCode=0 Nov 03 03:08:34.285844 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.285789 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jgcjh" Nov 03 03:08:34.285844 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.285815 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jgcjh" event={"ID":"80871945-cbe2-40b4-8715-472c062fc27f","Type":"ContainerDied","Data":"ec8627709f986d040ae797ce67bac125af0f73f36320ee6e8309cfc60c2e5de9"} Nov 03 03:08:34.285951 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.285868 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jgcjh" event={"ID":"80871945-cbe2-40b4-8715-472c062fc27f","Type":"ContainerDied","Data":"bbb0bb2540c66c44d1aa7feba3c94f0ddd7e96e816fe055713a08018b5031234"} Nov 03 03:08:34.288713 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.288688 5605 generic.go:356] "Generic (PLEG): container finished" podID="21200be4-59d6-4955-8d62-5394f4de28df" containerID="c7d79248d1b67b2fcf22e4ea304961bf1019daae99df51f335af48dd78b637ca" exitCode=0 Nov 03 03:08:34.288827 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.288761 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7dn2l" event={"ID":"21200be4-59d6-4955-8d62-5394f4de28df","Type":"ContainerDied","Data":"c7d79248d1b67b2fcf22e4ea304961bf1019daae99df51f335af48dd78b637ca"} Nov 03 03:08:34.288827 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.288798 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7dn2l" event={"ID":"21200be4-59d6-4955-8d62-5394f4de28df","Type":"ContainerDied","Data":"eb9f4a497f38a8fa8dfcbae222653d3426b236c9ae3fbba48ec1cc2e0f6e8598"} Nov 03 03:08:34.288973 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.288860 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7dn2l" Nov 03 03:08:34.293035 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.293014 5605 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b3468eb-1b4e-4727-910f-d975527c8644-istio-token\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:34.293151 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.293041 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcsz6\" (UniqueName: \"kubernetes.io/projected/0b3468eb-1b4e-4727-910f-d975527c8644-kube-api-access-mcsz6\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:34.293151 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.293058 5605 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b3468eb-1b4e-4727-910f-d975527c8644-istio-podinfo\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:34.293151 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.293072 5605 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b3468eb-1b4e-4727-910f-d975527c8644-istio-envoy\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:34.296426 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.296405 5605 scope.go:122] "RemoveContainer" containerID="8681bb515ff81651c2615f5062da8424e8fcd1ea05484a48046bf79e4fa44060" Nov 03 03:08:34.296736 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:08:34.296719 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8681bb515ff81651c2615f5062da8424e8fcd1ea05484a48046bf79e4fa44060\": container with ID starting with 8681bb515ff81651c2615f5062da8424e8fcd1ea05484a48046bf79e4fa44060 not found: ID does not exist" containerID="8681bb515ff81651c2615f5062da8424e8fcd1ea05484a48046bf79e4fa44060" Nov 03 03:08:34.296803 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.296742 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8681bb515ff81651c2615f5062da8424e8fcd1ea05484a48046bf79e4fa44060"} err="failed to get container status \"8681bb515ff81651c2615f5062da8424e8fcd1ea05484a48046bf79e4fa44060\": rpc error: code = NotFound desc = could not find container \"8681bb515ff81651c2615f5062da8424e8fcd1ea05484a48046bf79e4fa44060\": container with ID starting with 8681bb515ff81651c2615f5062da8424e8fcd1ea05484a48046bf79e4fa44060 not found: ID does not exist" Nov 03 03:08:34.296803 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.296756 5605 scope.go:122] "RemoveContainer" containerID="cb3e796b406e3abaeef2192c6926a294dec1b9b8d73476089aa9b133a1dd7a24" Nov 03 03:08:34.305768 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.305751 5605 scope.go:122] "RemoveContainer" containerID="cb3e796b406e3abaeef2192c6926a294dec1b9b8d73476089aa9b133a1dd7a24" Nov 03 03:08:34.306019 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:08:34.305990 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb3e796b406e3abaeef2192c6926a294dec1b9b8d73476089aa9b133a1dd7a24\": container with ID starting with cb3e796b406e3abaeef2192c6926a294dec1b9b8d73476089aa9b133a1dd7a24 not found: ID does not exist" containerID="cb3e796b406e3abaeef2192c6926a294dec1b9b8d73476089aa9b133a1dd7a24" Nov 03 03:08:34.306062 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.306018 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb3e796b406e3abaeef2192c6926a294dec1b9b8d73476089aa9b133a1dd7a24"} err="failed to get container status \"cb3e796b406e3abaeef2192c6926a294dec1b9b8d73476089aa9b133a1dd7a24\": rpc error: code = NotFound desc = could not find container \"cb3e796b406e3abaeef2192c6926a294dec1b9b8d73476089aa9b133a1dd7a24\": container with ID starting with cb3e796b406e3abaeef2192c6926a294dec1b9b8d73476089aa9b133a1dd7a24 not found: ID does not exist" Nov 03 03:08:34.306062 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.306034 5605 scope.go:122] "RemoveContainer" containerID="ec8627709f986d040ae797ce67bac125af0f73f36320ee6e8309cfc60c2e5de9" Nov 03 03:08:34.314686 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.314666 5605 scope.go:122] "RemoveContainer" containerID="ec8627709f986d040ae797ce67bac125af0f73f36320ee6e8309cfc60c2e5de9" Nov 03 03:08:34.314971 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:08:34.314949 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec8627709f986d040ae797ce67bac125af0f73f36320ee6e8309cfc60c2e5de9\": container with ID starting with ec8627709f986d040ae797ce67bac125af0f73f36320ee6e8309cfc60c2e5de9 not found: ID does not exist" containerID="ec8627709f986d040ae797ce67bac125af0f73f36320ee6e8309cfc60c2e5de9" Nov 03 03:08:34.315066 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.314978 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec8627709f986d040ae797ce67bac125af0f73f36320ee6e8309cfc60c2e5de9"} err="failed to get container status \"ec8627709f986d040ae797ce67bac125af0f73f36320ee6e8309cfc60c2e5de9\": rpc error: code = NotFound desc = could not find container \"ec8627709f986d040ae797ce67bac125af0f73f36320ee6e8309cfc60c2e5de9\": container with ID starting with ec8627709f986d040ae797ce67bac125af0f73f36320ee6e8309cfc60c2e5de9 not found: ID does not exist" Nov 03 03:08:34.315066 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.315000 5605 scope.go:122] "RemoveContainer" containerID="c7d79248d1b67b2fcf22e4ea304961bf1019daae99df51f335af48dd78b637ca" Nov 03 03:08:34.323418 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.323397 5605 scope.go:122] "RemoveContainer" containerID="c7d79248d1b67b2fcf22e4ea304961bf1019daae99df51f335af48dd78b637ca" Nov 03 03:08:34.323662 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:08:34.323641 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7d79248d1b67b2fcf22e4ea304961bf1019daae99df51f335af48dd78b637ca\": container with ID starting with c7d79248d1b67b2fcf22e4ea304961bf1019daae99df51f335af48dd78b637ca not found: ID does not exist" containerID="c7d79248d1b67b2fcf22e4ea304961bf1019daae99df51f335af48dd78b637ca" Nov 03 03:08:34.323761 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.323668 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7d79248d1b67b2fcf22e4ea304961bf1019daae99df51f335af48dd78b637ca"} err="failed to get container status \"c7d79248d1b67b2fcf22e4ea304961bf1019daae99df51f335af48dd78b637ca\": rpc error: code = NotFound desc = could not find container \"c7d79248d1b67b2fcf22e4ea304961bf1019daae99df51f335af48dd78b637ca\": container with ID starting with c7d79248d1b67b2fcf22e4ea304961bf1019daae99df51f335af48dd78b637ca not found: ID does not exist" Nov 03 03:08:34.572962 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.572921 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jgcjh"] Nov 03 03:08:34.575565 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.575541 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jgcjh"] Nov 03 03:08:34.589272 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.589241 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-c7d758-d578b44bc-cccmm" podUID="e2eff9e7-e011-46ce-a9e1-306ca1bf5893" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:08:34.680159 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.680126 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-flj4q"] Nov 03 03:08:34.685546 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.685517 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-flj4q"] Nov 03 03:08:34.787715 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.787680 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7dn2l"] Nov 03 03:08:34.791854 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.791826 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7dn2l"] Nov 03 03:08:34.849805 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.849725 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21200be4-59d6-4955-8d62-5394f4de28df" path="/var/lib/kubelet/pods/21200be4-59d6-4955-8d62-5394f4de28df/volumes" Nov 03 03:08:34.850208 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.850195 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80871945-cbe2-40b4-8715-472c062fc27f" path="/var/lib/kubelet/pods/80871945-cbe2-40b4-8715-472c062fc27f/volumes" Nov 03 03:08:34.850621 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.850608 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d76cc067-c62a-40e3-90d2-6435dc8cf655" path="/var/lib/kubelet/pods/d76cc067-c62a-40e3-90d2-6435dc8cf655/volumes" Nov 03 03:08:34.893619 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.893583 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-644zf"] Nov 03 03:08:34.897474 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:34.897448 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-644zf"] Nov 03 03:08:36.849283 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:36.849249 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b3468eb-1b4e-4727-910f-d975527c8644" path="/var/lib/kubelet/pods/0b3468eb-1b4e-4727-910f-d975527c8644/volumes" Nov 03 03:08:40.522176 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:40.521582 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-c7d758-d578b44bc-cccmm" Nov 03 03:08:40.661172 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:40.661129 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2eff9e7-e011-46ce-a9e1-306ca1bf5893-kube-api-access-46qld\" (UniqueName: \"kubernetes.io/projected/e2eff9e7-e011-46ce-a9e1-306ca1bf5893-kube-api-access-46qld\") pod \"e2eff9e7-e011-46ce-a9e1-306ca1bf5893\" (UID: \"e2eff9e7-e011-46ce-a9e1-306ca1bf5893\") " Nov 03 03:08:40.661423 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:40.661234 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2eff9e7-e011-46ce-a9e1-306ca1bf5893-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2eff9e7-e011-46ce-a9e1-306ca1bf5893-plugins-config-volume\") pod \"e2eff9e7-e011-46ce-a9e1-306ca1bf5893\" (UID: \"e2eff9e7-e011-46ce-a9e1-306ca1bf5893\") " Nov 03 03:08:40.661695 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:40.661663 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2eff9e7-e011-46ce-a9e1-306ca1bf5893-plugins-config-volume" pod "e2eff9e7-e011-46ce-a9e1-306ca1bf5893" (UID: "e2eff9e7-e011-46ce-a9e1-306ca1bf5893"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:08:40.668418 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:40.663930 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2eff9e7-e011-46ce-a9e1-306ca1bf5893-kube-api-access-46qld" pod "e2eff9e7-e011-46ce-a9e1-306ca1bf5893" (UID: "e2eff9e7-e011-46ce-a9e1-306ca1bf5893"). InnerVolumeSpecName "kube-api-access-46qld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:08:40.695586 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:40.695549 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-957b33/sub-test-model-957b33-kserve-5888c7bc5-tb6t5"] Nov 03 03:08:40.695960 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:40.695946 5605 cpu_manager.go:414] "containerMap: removing container" podUID="a9b9af44-ab6d-4caf-89af-c986ea25b4c9" containerName="main" Nov 03 03:08:40.695960 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:40.695959 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9b9af44-ab6d-4caf-89af-c986ea25b4c9" containerName="main" Nov 03 03:08:40.696105 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:40.695987 5605 cpu_manager.go:414] "containerMap: removing container" podUID="21200be4-59d6-4955-8d62-5394f4de28df" containerName="istio-proxy" Nov 03 03:08:40.696105 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:40.695992 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21200be4-59d6-4955-8d62-5394f4de28df" containerName="istio-proxy" Nov 03 03:08:40.696105 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:40.696003 5605 cpu_manager.go:414] "containerMap: removing container" podUID="e2eff9e7-e011-46ce-a9e1-306ca1bf5893" containerName="payload-processing" Nov 03 03:08:40.696105 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:40.696008 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2eff9e7-e011-46ce-a9e1-306ca1bf5893" containerName="payload-processing" Nov 03 03:08:40.696105 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:40.696016 5605 cpu_manager.go:414] "containerMap: removing container" podUID="1901f57b-279b-4cbf-bf0c-054abc5786d4" containerName="main" Nov 03 03:08:40.696105 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:40.696020 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1901f57b-279b-4cbf-bf0c-054abc5786d4" containerName="main" Nov 03 03:08:40.696105 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:40.696029 5605 cpu_manager.go:414] "containerMap: removing container" podUID="d76cc067-c62a-40e3-90d2-6435dc8cf655" containerName="istio-proxy" Nov 03 03:08:40.696105 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:40.696036 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d76cc067-c62a-40e3-90d2-6435dc8cf655" containerName="istio-proxy" Nov 03 03:08:40.696105 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:40.696051 5605 cpu_manager.go:414] "containerMap: removing container" podUID="70bd98e0-d8ac-47c2-901e-83f301d59a57" containerName="main" Nov 03 03:08:40.696105 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:40.696058 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70bd98e0-d8ac-47c2-901e-83f301d59a57" containerName="main" Nov 03 03:08:40.696105 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:40.696065 5605 cpu_manager.go:414] "containerMap: removing container" podUID="6b9772d3-3595-4ac3-b747-d7cbaae23c08" containerName="revoke-keys" Nov 03 03:08:40.696105 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:40.696070 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b9772d3-3595-4ac3-b747-d7cbaae23c08" containerName="revoke-keys" Nov 03 03:08:40.696105 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:40.696076 5605 cpu_manager.go:414] "containerMap: removing container" podUID="0b3468eb-1b4e-4727-910f-d975527c8644" containerName="istio-proxy" Nov 03 03:08:40.696105 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:40.696080 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b3468eb-1b4e-4727-910f-d975527c8644" containerName="istio-proxy" Nov 03 03:08:40.696105 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:40.696089 5605 cpu_manager.go:414] "containerMap: removing container" podUID="308bca3b-9bd5-4724-9b95-acdbc978aa10" containerName="main" Nov 03 03:08:40.696105 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:40.696094 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="308bca3b-9bd5-4724-9b95-acdbc978aa10" containerName="main" Nov 03 03:08:40.696105 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:40.696105 5605 cpu_manager.go:414] "containerMap: removing container" podUID="80871945-cbe2-40b4-8715-472c062fc27f" containerName="istio-proxy" Nov 03 03:08:40.696105 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:40.696109 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80871945-cbe2-40b4-8715-472c062fc27f" containerName="istio-proxy" Nov 03 03:08:40.697076 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:40.696166 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9b9af44-ab6d-4caf-89af-c986ea25b4c9" containerName="main" Nov 03 03:08:40.697076 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:40.696178 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="21200be4-59d6-4955-8d62-5394f4de28df" containerName="istio-proxy" Nov 03 03:08:40.697076 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:40.696185 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2eff9e7-e011-46ce-a9e1-306ca1bf5893" containerName="payload-processing" Nov 03 03:08:40.697076 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:40.696191 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="1901f57b-279b-4cbf-bf0c-054abc5786d4" containerName="main" Nov 03 03:08:40.697076 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:40.696196 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="d76cc067-c62a-40e3-90d2-6435dc8cf655" containerName="istio-proxy" Nov 03 03:08:40.697076 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:40.696201 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="70bd98e0-d8ac-47c2-901e-83f301d59a57" containerName="main" Nov 03 03:08:40.697076 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:40.696207 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b9772d3-3595-4ac3-b747-d7cbaae23c08" containerName="revoke-keys" Nov 03 03:08:40.697076 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:40.696211 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b3468eb-1b4e-4727-910f-d975527c8644" containerName="istio-proxy" Nov 03 03:08:40.697076 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:40.696217 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="308bca3b-9bd5-4724-9b95-acdbc978aa10" containerName="main" Nov 03 03:08:40.697076 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:40.696222 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="80871945-cbe2-40b4-8715-472c062fc27f" containerName="istio-proxy" Nov 03 03:08:40.699706 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:40.699684 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-957b33/sub-test-model-957b33-kserve-5888c7bc5-tb6t5" Nov 03 03:08:40.753604 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:40.753574 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-957b33\"/\"openshift-service-ca.crt\"" Nov 03 03:08:40.753783 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:40.753637 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-957b33\"/\"kube-root-ca.crt\"" Nov 03 03:08:40.762224 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:40.762196 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/392beb71-3497-4141-b8cb-5485f7b9c238-home\") pod \"sub-test-model-957b33-kserve-5888c7bc5-tb6t5\" (UID: \"392beb71-3497-4141-b8cb-5485f7b9c238\") " pod="ai-tenant-e2e-shared-b-w2-957b33/sub-test-model-957b33-kserve-5888c7bc5-tb6t5" Nov 03 03:08:40.762322 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:40.762232 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/392beb71-3497-4141-b8cb-5485f7b9c238-tls-certs\") pod \"sub-test-model-957b33-kserve-5888c7bc5-tb6t5\" (UID: \"392beb71-3497-4141-b8cb-5485f7b9c238\") " pod="ai-tenant-e2e-shared-b-w2-957b33/sub-test-model-957b33-kserve-5888c7bc5-tb6t5" Nov 03 03:08:40.762448 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:40.762423 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/392beb71-3497-4141-b8cb-5485f7b9c238-model-cache\") pod \"sub-test-model-957b33-kserve-5888c7bc5-tb6t5\" (UID: \"392beb71-3497-4141-b8cb-5485f7b9c238\") " pod="ai-tenant-e2e-shared-b-w2-957b33/sub-test-model-957b33-kserve-5888c7bc5-tb6t5" Nov 03 03:08:40.762543 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:40.762529 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/392beb71-3497-4141-b8cb-5485f7b9c238-tmp-dir\") pod \"sub-test-model-957b33-kserve-5888c7bc5-tb6t5\" (UID: \"392beb71-3497-4141-b8cb-5485f7b9c238\") " pod="ai-tenant-e2e-shared-b-w2-957b33/sub-test-model-957b33-kserve-5888c7bc5-tb6t5" Nov 03 03:08:40.762585 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:40.762567 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/392beb71-3497-4141-b8cb-5485f7b9c238-dshm\") pod \"sub-test-model-957b33-kserve-5888c7bc5-tb6t5\" (UID: \"392beb71-3497-4141-b8cb-5485f7b9c238\") " pod="ai-tenant-e2e-shared-b-w2-957b33/sub-test-model-957b33-kserve-5888c7bc5-tb6t5" Nov 03 03:08:40.762642 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:40.762625 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk9qv\" (UniqueName: \"kubernetes.io/projected/392beb71-3497-4141-b8cb-5485f7b9c238-kube-api-access-lk9qv\") pod \"sub-test-model-957b33-kserve-5888c7bc5-tb6t5\" (UID: \"392beb71-3497-4141-b8cb-5485f7b9c238\") " pod="ai-tenant-e2e-shared-b-w2-957b33/sub-test-model-957b33-kserve-5888c7bc5-tb6t5" Nov 03 03:08:40.762811 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:40.762788 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46qld\" (UniqueName: \"kubernetes.io/projected/e2eff9e7-e011-46ce-a9e1-306ca1bf5893-kube-api-access-46qld\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:40.762847 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:40.762817 5605 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2eff9e7-e011-46ce-a9e1-306ca1bf5893-plugins-config-volume\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:40.766679 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:40.766649 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-957b33/sub-test-model-957b33-kserve-5888c7bc5-tb6t5"] Nov 03 03:08:40.800641 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:40.800568 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-957b33\"/\"default-dockercfg-gpnj4\"" Nov 03 03:08:40.800641 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:40.800585 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-957b33\"/\"sub-test-model-957b33-kserve-self-signed-certs\"" Nov 03 03:08:40.863494 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:40.863466 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/392beb71-3497-4141-b8cb-5485f7b9c238-tmp-dir\") pod \"sub-test-model-957b33-kserve-5888c7bc5-tb6t5\" (UID: \"392beb71-3497-4141-b8cb-5485f7b9c238\") " pod="ai-tenant-e2e-shared-b-w2-957b33/sub-test-model-957b33-kserve-5888c7bc5-tb6t5" Nov 03 03:08:40.863639 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:40.863515 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/392beb71-3497-4141-b8cb-5485f7b9c238-dshm\") pod \"sub-test-model-957b33-kserve-5888c7bc5-tb6t5\" (UID: \"392beb71-3497-4141-b8cb-5485f7b9c238\") " pod="ai-tenant-e2e-shared-b-w2-957b33/sub-test-model-957b33-kserve-5888c7bc5-tb6t5" Nov 03 03:08:40.863639 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:40.863586 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk9qv\" (UniqueName: \"kubernetes.io/projected/392beb71-3497-4141-b8cb-5485f7b9c238-kube-api-access-lk9qv\") pod \"sub-test-model-957b33-kserve-5888c7bc5-tb6t5\" (UID: \"392beb71-3497-4141-b8cb-5485f7b9c238\") " pod="ai-tenant-e2e-shared-b-w2-957b33/sub-test-model-957b33-kserve-5888c7bc5-tb6t5" Nov 03 03:08:40.863639 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:40.863633 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/392beb71-3497-4141-b8cb-5485f7b9c238-home\") pod \"sub-test-model-957b33-kserve-5888c7bc5-tb6t5\" (UID: \"392beb71-3497-4141-b8cb-5485f7b9c238\") " pod="ai-tenant-e2e-shared-b-w2-957b33/sub-test-model-957b33-kserve-5888c7bc5-tb6t5" Nov 03 03:08:40.863830 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:40.863662 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/392beb71-3497-4141-b8cb-5485f7b9c238-tls-certs\") pod \"sub-test-model-957b33-kserve-5888c7bc5-tb6t5\" (UID: \"392beb71-3497-4141-b8cb-5485f7b9c238\") " pod="ai-tenant-e2e-shared-b-w2-957b33/sub-test-model-957b33-kserve-5888c7bc5-tb6t5" Nov 03 03:08:40.863830 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:40.863749 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/392beb71-3497-4141-b8cb-5485f7b9c238-model-cache\") pod \"sub-test-model-957b33-kserve-5888c7bc5-tb6t5\" (UID: \"392beb71-3497-4141-b8cb-5485f7b9c238\") " pod="ai-tenant-e2e-shared-b-w2-957b33/sub-test-model-957b33-kserve-5888c7bc5-tb6t5" Nov 03 03:08:40.863932 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:40.863887 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/392beb71-3497-4141-b8cb-5485f7b9c238-tmp-dir\") pod \"sub-test-model-957b33-kserve-5888c7bc5-tb6t5\" (UID: \"392beb71-3497-4141-b8cb-5485f7b9c238\") " pod="ai-tenant-e2e-shared-b-w2-957b33/sub-test-model-957b33-kserve-5888c7bc5-tb6t5" Nov 03 03:08:40.864050 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:40.864027 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/392beb71-3497-4141-b8cb-5485f7b9c238-home\") pod \"sub-test-model-957b33-kserve-5888c7bc5-tb6t5\" (UID: \"392beb71-3497-4141-b8cb-5485f7b9c238\") " pod="ai-tenant-e2e-shared-b-w2-957b33/sub-test-model-957b33-kserve-5888c7bc5-tb6t5" Nov 03 03:08:40.864138 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:40.864117 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/392beb71-3497-4141-b8cb-5485f7b9c238-model-cache\") pod \"sub-test-model-957b33-kserve-5888c7bc5-tb6t5\" (UID: \"392beb71-3497-4141-b8cb-5485f7b9c238\") " pod="ai-tenant-e2e-shared-b-w2-957b33/sub-test-model-957b33-kserve-5888c7bc5-tb6t5" Nov 03 03:08:40.865744 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:40.865725 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/392beb71-3497-4141-b8cb-5485f7b9c238-dshm\") pod \"sub-test-model-957b33-kserve-5888c7bc5-tb6t5\" (UID: \"392beb71-3497-4141-b8cb-5485f7b9c238\") " pod="ai-tenant-e2e-shared-b-w2-957b33/sub-test-model-957b33-kserve-5888c7bc5-tb6t5" Nov 03 03:08:40.866292 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:40.866274 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/392beb71-3497-4141-b8cb-5485f7b9c238-tls-certs\") pod \"sub-test-model-957b33-kserve-5888c7bc5-tb6t5\" (UID: \"392beb71-3497-4141-b8cb-5485f7b9c238\") " pod="ai-tenant-e2e-shared-b-w2-957b33/sub-test-model-957b33-kserve-5888c7bc5-tb6t5" Nov 03 03:08:40.919336 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:40.919266 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk9qv\" (UniqueName: \"kubernetes.io/projected/392beb71-3497-4141-b8cb-5485f7b9c238-kube-api-access-lk9qv\") pod \"sub-test-model-957b33-kserve-5888c7bc5-tb6t5\" (UID: \"392beb71-3497-4141-b8cb-5485f7b9c238\") " pod="ai-tenant-e2e-shared-b-w2-957b33/sub-test-model-957b33-kserve-5888c7bc5-tb6t5" Nov 03 03:08:41.013306 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:41.013268 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-957b33/sub-test-model-957b33-kserve-5888c7bc5-tb6t5" Nov 03 03:08:41.336144 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:41.336104 5605 generic.go:356] "Generic (PLEG): container finished" podID="e2eff9e7-e011-46ce-a9e1-306ca1bf5893" containerID="1ac18cacaa2401eeb837d21b4ebd191b0603a700887195bcb3343877b4b6bd03" exitCode=0 Nov 03 03:08:41.336350 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:41.336168 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-c7d758-d578b44bc-cccmm" Nov 03 03:08:41.336350 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:41.336182 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-c7d758-d578b44bc-cccmm" event={"ID":"e2eff9e7-e011-46ce-a9e1-306ca1bf5893","Type":"ContainerDied","Data":"1ac18cacaa2401eeb837d21b4ebd191b0603a700887195bcb3343877b4b6bd03"} Nov 03 03:08:41.336350 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:41.336263 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-c7d758-d578b44bc-cccmm" event={"ID":"e2eff9e7-e011-46ce-a9e1-306ca1bf5893","Type":"ContainerDied","Data":"fe96814f1d1723daa5984a7ea4e4095f4240cec8f84e865414595d1a651c0643"} Nov 03 03:08:41.336350 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:41.336286 5605 scope.go:122] "RemoveContainer" containerID="1ac18cacaa2401eeb837d21b4ebd191b0603a700887195bcb3343877b4b6bd03" Nov 03 03:08:41.347057 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:41.347032 5605 scope.go:122] "RemoveContainer" containerID="1ac18cacaa2401eeb837d21b4ebd191b0603a700887195bcb3343877b4b6bd03" Nov 03 03:08:41.347396 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:08:41.347372 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ac18cacaa2401eeb837d21b4ebd191b0603a700887195bcb3343877b4b6bd03\": container with ID starting with 1ac18cacaa2401eeb837d21b4ebd191b0603a700887195bcb3343877b4b6bd03 not found: ID does not exist" containerID="1ac18cacaa2401eeb837d21b4ebd191b0603a700887195bcb3343877b4b6bd03" Nov 03 03:08:41.347448 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:41.347407 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ac18cacaa2401eeb837d21b4ebd191b0603a700887195bcb3343877b4b6bd03"} err="failed to get container status \"1ac18cacaa2401eeb837d21b4ebd191b0603a700887195bcb3343877b4b6bd03\": rpc error: code = NotFound desc = could not find container \"1ac18cacaa2401eeb837d21b4ebd191b0603a700887195bcb3343877b4b6bd03\": container with ID starting with 1ac18cacaa2401eeb837d21b4ebd191b0603a700887195bcb3343877b4b6bd03 not found: ID does not exist" Nov 03 03:08:41.429481 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:41.429450 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-957b33/sub-test-model-957b33-kserve-5888c7bc5-tb6t5"] Nov 03 03:08:41.433818 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:08:41.433775 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod392beb71_3497_4141_b8cb_5485f7b9c238.slice/crio-3e5a4450e37aa2a247de72efa90f2cee2f435ac3db01b64ffd0b20be9e1d86bd WatchSource:0}: Error finding container 3e5a4450e37aa2a247de72efa90f2cee2f435ac3db01b64ffd0b20be9e1d86bd: Status 404 returned error can't find the container with id 3e5a4450e37aa2a247de72efa90f2cee2f435ac3db01b64ffd0b20be9e1d86bd Nov 03 03:08:41.465139 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:41.465102 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-a0e72e-64b874b665-4k4v9" podUID="5c524f2c-3641-4719-b906-2434a016e677" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:08:41.542728 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:41.542696 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c7d758-d578b44bc-cccmm"] Nov 03 03:08:41.544584 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:41.544560 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c7d758-d578b44bc-cccmm"] Nov 03 03:08:42.344160 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:42.344127 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-957b33/sub-test-model-957b33-kserve-5888c7bc5-tb6t5" event={"ID":"392beb71-3497-4141-b8cb-5485f7b9c238","Type":"ContainerStarted","Data":"f1a6c614119a4cdfb200ca40945310bd36d390881fa844f82d59ae6cbe52f7c7"} Nov 03 03:08:42.344329 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:42.344171 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-957b33/sub-test-model-957b33-kserve-5888c7bc5-tb6t5" event={"ID":"392beb71-3497-4141-b8cb-5485f7b9c238","Type":"ContainerStarted","Data":"3e5a4450e37aa2a247de72efa90f2cee2f435ac3db01b64ffd0b20be9e1d86bd"} Nov 03 03:08:42.453128 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:42.453066 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-957b33/sub-test-model-957b33-kserve-5888c7bc5-tb6t5" podStartSLOduration=2.45304971 podStartE2EDuration="2.45304971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:08:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:08:42.453002913 +0000 UTC m=+4599.712201066" watchObservedRunningTime="2026-11-03 03:08:42.45304971 +0000 UTC m=+4599.712247864" Nov 03 03:08:42.850405 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:42.850343 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2eff9e7-e011-46ce-a9e1-306ca1bf5893" path="/var/lib/kubelet/pods/e2eff9e7-e011-46ce-a9e1-306ca1bf5893/volumes" Nov 03 03:08:43.013716 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:43.013682 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-957b33/sub-test-model-957b33-kserve-5888c7bc5-tb6t5" Nov 03 03:08:50.812914 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:50.812883 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-a0e72e-64b874b665-4k4v9" Nov 03 03:08:50.972555 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:50.972524 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c524f2c-3641-4719-b906-2434a016e677-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c524f2c-3641-4719-b906-2434a016e677-plugins-config-volume\") pod \"5c524f2c-3641-4719-b906-2434a016e677\" (UID: \"5c524f2c-3641-4719-b906-2434a016e677\") " Nov 03 03:08:50.972734 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:50.972572 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c524f2c-3641-4719-b906-2434a016e677-kube-api-access-wl448\" (UniqueName: \"kubernetes.io/projected/5c524f2c-3641-4719-b906-2434a016e677-kube-api-access-wl448\") pod \"5c524f2c-3641-4719-b906-2434a016e677\" (UID: \"5c524f2c-3641-4719-b906-2434a016e677\") " Nov 03 03:08:50.972921 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:50.972895 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c524f2c-3641-4719-b906-2434a016e677-plugins-config-volume" pod "5c524f2c-3641-4719-b906-2434a016e677" (UID: "5c524f2c-3641-4719-b906-2434a016e677"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:08:50.974686 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:50.974657 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c524f2c-3641-4719-b906-2434a016e677-kube-api-access-wl448" pod "5c524f2c-3641-4719-b906-2434a016e677" (UID: "5c524f2c-3641-4719-b906-2434a016e677"). InnerVolumeSpecName "kube-api-access-wl448". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:08:51.013536 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:51.013502 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-957b33/sub-test-model-957b33-kserve-5888c7bc5-tb6t5" Nov 03 03:08:51.025883 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:51.025854 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-957b33/sub-test-model-957b33-kserve-5888c7bc5-tb6t5" Nov 03 03:08:51.074320 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:51.074284 5605 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c524f2c-3641-4719-b906-2434a016e677-plugins-config-volume\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:51.074320 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:51.074316 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wl448\" (UniqueName: \"kubernetes.io/projected/5c524f2c-3641-4719-b906-2434a016e677-kube-api-access-wl448\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:51.397153 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:51.397068 5605 generic.go:356] "Generic (PLEG): container finished" podID="5c524f2c-3641-4719-b906-2434a016e677" containerID="fb3e3dd8a330b2f38c9c8bd5d2ac7a81845b0246797e3ea2f62dcc5d68681d56" exitCode=0 Nov 03 03:08:51.397153 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:51.397119 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-a0e72e-64b874b665-4k4v9" Nov 03 03:08:51.397350 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:51.397150 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-a0e72e-64b874b665-4k4v9" event={"ID":"5c524f2c-3641-4719-b906-2434a016e677","Type":"ContainerDied","Data":"fb3e3dd8a330b2f38c9c8bd5d2ac7a81845b0246797e3ea2f62dcc5d68681d56"} Nov 03 03:08:51.397350 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:51.397212 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-a0e72e-64b874b665-4k4v9" event={"ID":"5c524f2c-3641-4719-b906-2434a016e677","Type":"ContainerDied","Data":"6be3eaf7c1c4a29cce68576d2e2c350f56799f317e0a122e45c6683a2484eb1c"} Nov 03 03:08:51.397350 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:51.397242 5605 scope.go:122] "RemoveContainer" containerID="fb3e3dd8a330b2f38c9c8bd5d2ac7a81845b0246797e3ea2f62dcc5d68681d56" Nov 03 03:08:51.406844 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:51.406825 5605 scope.go:122] "RemoveContainer" containerID="fb3e3dd8a330b2f38c9c8bd5d2ac7a81845b0246797e3ea2f62dcc5d68681d56" Nov 03 03:08:51.407143 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:08:51.407106 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb3e3dd8a330b2f38c9c8bd5d2ac7a81845b0246797e3ea2f62dcc5d68681d56\": container with ID starting with fb3e3dd8a330b2f38c9c8bd5d2ac7a81845b0246797e3ea2f62dcc5d68681d56 not found: ID does not exist" containerID="fb3e3dd8a330b2f38c9c8bd5d2ac7a81845b0246797e3ea2f62dcc5d68681d56" Nov 03 03:08:51.407242 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:51.407150 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb3e3dd8a330b2f38c9c8bd5d2ac7a81845b0246797e3ea2f62dcc5d68681d56"} err="failed to get container status \"fb3e3dd8a330b2f38c9c8bd5d2ac7a81845b0246797e3ea2f62dcc5d68681d56\": rpc error: code = NotFound desc = could not find container \"fb3e3dd8a330b2f38c9c8bd5d2ac7a81845b0246797e3ea2f62dcc5d68681d56\": container with ID starting with fb3e3dd8a330b2f38c9c8bd5d2ac7a81845b0246797e3ea2f62dcc5d68681d56 not found: ID does not exist" Nov 03 03:08:51.409393 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:51.409349 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-957b33/sub-test-model-957b33-kserve-5888c7bc5-tb6t5" Nov 03 03:08:51.563499 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:51.563464 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-a0e72e-64b874b665-4k4v9"] Nov 03 03:08:51.611297 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:51.611262 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-a0e72e-64b874b665-4k4v9"] Nov 03 03:08:51.855004 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:51.854970 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:08:52.849910 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:08:52.849881 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c524f2c-3641-4719-b906-2434a016e677" path="/var/lib/kubelet/pods/5c524f2c-3641-4719-b906-2434a016e677/volumes" Nov 03 03:08:53.912442 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:08:53.912409 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:09:05.203541 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:09:05.203509 5605 scope.go:122] "RemoveContainer" containerID="9ff9a7045e7e3c8d4413aa8c6c61f569712c22461d6a335418441655a0276a9c" Nov 03 03:09:23.266786 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:09:23.266740 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g"] Nov 03 03:09:23.267288 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:09:23.266986 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g" podUID="f05a0c36-8c90-4813-b84c-0825be1947a6" containerName="istio-proxy" containerID="cri-o://5524a870ee429e5a3c4d6dce6fe687c7d70e14239f347e681228972ad5e05016" gracePeriod=30 Nov 03 03:09:23.911792 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:09:23.911755 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:09:28.531863 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:09:28.531837 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g" Nov 03 03:09:28.607440 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:09:28.607403 5605 generic.go:356] "Generic (PLEG): container finished" podID="f05a0c36-8c90-4813-b84c-0825be1947a6" containerID="5524a870ee429e5a3c4d6dce6fe687c7d70e14239f347e681228972ad5e05016" exitCode=0 Nov 03 03:09:28.607638 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:09:28.607468 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g" Nov 03 03:09:28.607638 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:09:28.607493 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g" event={"ID":"f05a0c36-8c90-4813-b84c-0825be1947a6","Type":"ContainerDied","Data":"5524a870ee429e5a3c4d6dce6fe687c7d70e14239f347e681228972ad5e05016"} Nov 03 03:09:28.607638 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:09:28.607550 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g" event={"ID":"f05a0c36-8c90-4813-b84c-0825be1947a6","Type":"ContainerDied","Data":"4a661a2ff4b673c7ec09c6a439a71f8a1fd269aa47aa2f58e4b056a7eab00938"} Nov 03 03:09:28.607638 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:09:28.607574 5605 scope.go:122] "RemoveContainer" containerID="5524a870ee429e5a3c4d6dce6fe687c7d70e14239f347e681228972ad5e05016" Nov 03 03:09:28.616681 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:09:28.616662 5605 scope.go:122] "RemoveContainer" containerID="5524a870ee429e5a3c4d6dce6fe687c7d70e14239f347e681228972ad5e05016" Nov 03 03:09:28.616997 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:09:28.616972 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5524a870ee429e5a3c4d6dce6fe687c7d70e14239f347e681228972ad5e05016\": container with ID starting with 5524a870ee429e5a3c4d6dce6fe687c7d70e14239f347e681228972ad5e05016 not found: ID does not exist" containerID="5524a870ee429e5a3c4d6dce6fe687c7d70e14239f347e681228972ad5e05016" Nov 03 03:09:28.617062 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:09:28.617007 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5524a870ee429e5a3c4d6dce6fe687c7d70e14239f347e681228972ad5e05016"} err="failed to get container status \"5524a870ee429e5a3c4d6dce6fe687c7d70e14239f347e681228972ad5e05016\": rpc error: code = NotFound desc = could not find container \"5524a870ee429e5a3c4d6dce6fe687c7d70e14239f347e681228972ad5e05016\": container with ID starting with 5524a870ee429e5a3c4d6dce6fe687c7d70e14239f347e681228972ad5e05016 not found: ID does not exist" Nov 03 03:09:28.675666 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:09:28.675627 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f05a0c36-8c90-4813-b84c-0825be1947a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f05a0c36-8c90-4813-b84c-0825be1947a6-istio-data\") pod \"f05a0c36-8c90-4813-b84c-0825be1947a6\" (UID: \"f05a0c36-8c90-4813-b84c-0825be1947a6\") " Nov 03 03:09:28.675872 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:09:28.675723 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f05a0c36-8c90-4813-b84c-0825be1947a6-istio-token\" (UniqueName: \"kubernetes.io/projected/f05a0c36-8c90-4813-b84c-0825be1947a6-istio-token\") pod \"f05a0c36-8c90-4813-b84c-0825be1947a6\" (UID: \"f05a0c36-8c90-4813-b84c-0825be1947a6\") " Nov 03 03:09:28.675872 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:09:28.675789 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f05a0c36-8c90-4813-b84c-0825be1947a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f05a0c36-8c90-4813-b84c-0825be1947a6-istio-podinfo\") pod \"f05a0c36-8c90-4813-b84c-0825be1947a6\" (UID: \"f05a0c36-8c90-4813-b84c-0825be1947a6\") " Nov 03 03:09:28.675872 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:09:28.675842 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f05a0c36-8c90-4813-b84c-0825be1947a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f05a0c36-8c90-4813-b84c-0825be1947a6-credential-socket\") pod \"f05a0c36-8c90-4813-b84c-0825be1947a6\" (UID: \"f05a0c36-8c90-4813-b84c-0825be1947a6\") " Nov 03 03:09:28.675872 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:09:28.675867 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f05a0c36-8c90-4813-b84c-0825be1947a6-kube-api-access-frlq4\" (UniqueName: \"kubernetes.io/projected/f05a0c36-8c90-4813-b84c-0825be1947a6-kube-api-access-frlq4\") pod \"f05a0c36-8c90-4813-b84c-0825be1947a6\" (UID: \"f05a0c36-8c90-4813-b84c-0825be1947a6\") " Nov 03 03:09:28.676107 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:09:28.675893 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f05a0c36-8c90-4813-b84c-0825be1947a6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f05a0c36-8c90-4813-b84c-0825be1947a6-workload-socket\") pod \"f05a0c36-8c90-4813-b84c-0825be1947a6\" (UID: \"f05a0c36-8c90-4813-b84c-0825be1947a6\") " Nov 03 03:09:28.676107 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:09:28.675933 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f05a0c36-8c90-4813-b84c-0825be1947a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f05a0c36-8c90-4813-b84c-0825be1947a6-istiod-ca-cert\") pod \"f05a0c36-8c90-4813-b84c-0825be1947a6\" (UID: \"f05a0c36-8c90-4813-b84c-0825be1947a6\") " Nov 03 03:09:28.676107 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:09:28.675953 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f05a0c36-8c90-4813-b84c-0825be1947a6-istio-data" pod "f05a0c36-8c90-4813-b84c-0825be1947a6" (UID: "f05a0c36-8c90-4813-b84c-0825be1947a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:09:28.676107 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:09:28.676005 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f05a0c36-8c90-4813-b84c-0825be1947a6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f05a0c36-8c90-4813-b84c-0825be1947a6-istio-envoy\") pod \"f05a0c36-8c90-4813-b84c-0825be1947a6\" (UID: \"f05a0c36-8c90-4813-b84c-0825be1947a6\") " Nov 03 03:09:28.676107 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:09:28.676039 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f05a0c36-8c90-4813-b84c-0825be1947a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f05a0c36-8c90-4813-b84c-0825be1947a6-workload-certs\") pod \"f05a0c36-8c90-4813-b84c-0825be1947a6\" (UID: \"f05a0c36-8c90-4813-b84c-0825be1947a6\") " Nov 03 03:09:28.676416 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:09:28.676106 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f05a0c36-8c90-4813-b84c-0825be1947a6-credential-socket" pod "f05a0c36-8c90-4813-b84c-0825be1947a6" (UID: "f05a0c36-8c90-4813-b84c-0825be1947a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:09:28.676416 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:09:28.676159 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f05a0c36-8c90-4813-b84c-0825be1947a6-workload-socket" pod "f05a0c36-8c90-4813-b84c-0825be1947a6" (UID: "f05a0c36-8c90-4813-b84c-0825be1947a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:09:28.676416 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:09:28.676282 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f05a0c36-8c90-4813-b84c-0825be1947a6-istiod-ca-cert" pod "f05a0c36-8c90-4813-b84c-0825be1947a6" (UID: "f05a0c36-8c90-4813-b84c-0825be1947a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:09:28.676578 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:09:28.676445 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f05a0c36-8c90-4813-b84c-0825be1947a6-workload-certs" pod "f05a0c36-8c90-4813-b84c-0825be1947a6" (UID: "f05a0c36-8c90-4813-b84c-0825be1947a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:09:28.676637 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:09:28.676612 5605 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f05a0c36-8c90-4813-b84c-0825be1947a6-workload-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:09:28.676687 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:09:28.676642 5605 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f05a0c36-8c90-4813-b84c-0825be1947a6-istio-data\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:09:28.676687 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:09:28.676658 5605 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f05a0c36-8c90-4813-b84c-0825be1947a6-credential-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:09:28.676687 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:09:28.676672 5605 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f05a0c36-8c90-4813-b84c-0825be1947a6-workload-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:09:28.676809 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:09:28.676686 5605 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f05a0c36-8c90-4813-b84c-0825be1947a6-istiod-ca-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:09:28.678135 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:09:28.678104 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f05a0c36-8c90-4813-b84c-0825be1947a6-istio-podinfo" pod "f05a0c36-8c90-4813-b84c-0825be1947a6" (UID: "f05a0c36-8c90-4813-b84c-0825be1947a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:09:28.678262 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:09:28.678198 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f05a0c36-8c90-4813-b84c-0825be1947a6-istio-envoy" pod "f05a0c36-8c90-4813-b84c-0825be1947a6" (UID: "f05a0c36-8c90-4813-b84c-0825be1947a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:09:28.678452 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:09:28.678430 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f05a0c36-8c90-4813-b84c-0825be1947a6-istio-token" pod "f05a0c36-8c90-4813-b84c-0825be1947a6" (UID: "f05a0c36-8c90-4813-b84c-0825be1947a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:09:28.678527 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:09:28.678507 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f05a0c36-8c90-4813-b84c-0825be1947a6-kube-api-access-frlq4" pod "f05a0c36-8c90-4813-b84c-0825be1947a6" (UID: "f05a0c36-8c90-4813-b84c-0825be1947a6"). InnerVolumeSpecName "kube-api-access-frlq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:09:28.777732 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:09:28.777697 5605 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f05a0c36-8c90-4813-b84c-0825be1947a6-istio-podinfo\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:09:28.777732 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:09:28.777727 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frlq4\" (UniqueName: \"kubernetes.io/projected/f05a0c36-8c90-4813-b84c-0825be1947a6-kube-api-access-frlq4\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:09:28.777732 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:09:28.777740 5605 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f05a0c36-8c90-4813-b84c-0825be1947a6-istio-envoy\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:09:28.777994 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:09:28.777749 5605 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f05a0c36-8c90-4813-b84c-0825be1947a6-istio-token\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:09:29.015450 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:09:29.015348 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g"] Nov 03 03:09:29.022144 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:09:29.022117 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-lr46g"] Nov 03 03:09:30.850068 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:09:30.850030 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f05a0c36-8c90-4813-b84c-0825be1947a6" path="/var/lib/kubelet/pods/f05a0c36-8c90-4813-b84c-0825be1947a6/volumes" Nov 03 03:09:53.912928 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:09:53.912884 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:09:58.279123 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:09:58.279079 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:10:08.063123 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:08.063082 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-970fa6cc-jsdnt"] Nov 03 03:10:08.063571 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:08.063509 5605 cpu_manager.go:414] "containerMap: removing container" podUID="f05a0c36-8c90-4813-b84c-0825be1947a6" containerName="istio-proxy" Nov 03 03:10:08.063571 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:08.063524 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f05a0c36-8c90-4813-b84c-0825be1947a6" containerName="istio-proxy" Nov 03 03:10:08.063571 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:08.063545 5605 cpu_manager.go:414] "containerMap: removing container" podUID="5c524f2c-3641-4719-b906-2434a016e677" containerName="payload-processing" Nov 03 03:10:08.063571 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:08.063551 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c524f2c-3641-4719-b906-2434a016e677" containerName="payload-processing" Nov 03 03:10:08.063734 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:08.063615 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="f05a0c36-8c90-4813-b84c-0825be1947a6" containerName="istio-proxy" Nov 03 03:10:08.063734 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:08.063626 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c524f2c-3641-4719-b906-2434a016e677" containerName="payload-processing" Nov 03 03:10:08.068180 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:08.068159 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-970fa6cc-jsdnt" Nov 03 03:10:08.074529 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:08.074501 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffabd9df-96b6-4e1c-913c-cb29cbfbd83a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-970fa6cc-jsdnt\" (UID: \"ffabd9df-96b6-4e1c-913c-cb29cbfbd83a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-970fa6cc-jsdnt" Nov 03 03:10:08.075028 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:08.074716 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffabd9df-96b6-4e1c-913c-cb29cbfbd83a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-970fa6cc-jsdnt\" (UID: \"ffabd9df-96b6-4e1c-913c-cb29cbfbd83a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-970fa6cc-jsdnt" Nov 03 03:10:08.075028 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:08.074844 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjtxc\" (UniqueName: \"kubernetes.io/projected/ffabd9df-96b6-4e1c-913c-cb29cbfbd83a-kube-api-access-tjtxc\") pod \"maas-api-revoke-sub-970fa6cc-jsdnt\" (UID: \"ffabd9df-96b6-4e1c-913c-cb29cbfbd83a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-970fa6cc-jsdnt" Nov 03 03:10:08.119719 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:08.119692 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-970fa6cc-jsdnt"] Nov 03 03:10:08.164168 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:08.164123 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:10:08.164168 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:08.164160 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:10:08.164454 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:08.164191 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:10:08.164454 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:08.164252 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 03:10:08.175859 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:08.175821 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjtxc\" (UniqueName: \"kubernetes.io/projected/ffabd9df-96b6-4e1c-913c-cb29cbfbd83a-kube-api-access-tjtxc\") pod \"maas-api-revoke-sub-970fa6cc-jsdnt\" (UID: \"ffabd9df-96b6-4e1c-913c-cb29cbfbd83a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-970fa6cc-jsdnt" Nov 03 03:10:08.176031 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:08.175902 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffabd9df-96b6-4e1c-913c-cb29cbfbd83a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-970fa6cc-jsdnt\" (UID: \"ffabd9df-96b6-4e1c-913c-cb29cbfbd83a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-970fa6cc-jsdnt" Nov 03 03:10:08.176031 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:08.176017 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffabd9df-96b6-4e1c-913c-cb29cbfbd83a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-970fa6cc-jsdnt\" (UID: \"ffabd9df-96b6-4e1c-913c-cb29cbfbd83a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-970fa6cc-jsdnt" Nov 03 03:10:08.176750 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:08.176729 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffabd9df-96b6-4e1c-913c-cb29cbfbd83a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-970fa6cc-jsdnt\" (UID: \"ffabd9df-96b6-4e1c-913c-cb29cbfbd83a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-970fa6cc-jsdnt" Nov 03 03:10:08.178413 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:08.178396 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffabd9df-96b6-4e1c-913c-cb29cbfbd83a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-970fa6cc-jsdnt\" (UID: \"ffabd9df-96b6-4e1c-913c-cb29cbfbd83a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-970fa6cc-jsdnt" Nov 03 03:10:08.229224 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:08.229199 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjtxc\" (UniqueName: \"kubernetes.io/projected/ffabd9df-96b6-4e1c-913c-cb29cbfbd83a-kube-api-access-tjtxc\") pod \"maas-api-revoke-sub-970fa6cc-jsdnt\" (UID: \"ffabd9df-96b6-4e1c-913c-cb29cbfbd83a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-970fa6cc-jsdnt" Nov 03 03:10:08.381284 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:08.381201 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-970fa6cc-jsdnt" Nov 03 03:10:08.752415 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:08.752310 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-970fa6cc-jsdnt"] Nov 03 03:10:08.758558 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:10:08.758529 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffabd9df_96b6_4e1c_913c_cb29cbfbd83a.slice/crio-663420b6d8e5264edefd731f5520403a9ca78b863effb64fc055f031d6a0ee49 WatchSource:0}: Error finding container 663420b6d8e5264edefd731f5520403a9ca78b863effb64fc055f031d6a0ee49: Status 404 returned error can't find the container with id 663420b6d8e5264edefd731f5520403a9ca78b863effb64fc055f031d6a0ee49 Nov 03 03:10:08.819390 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:08.819346 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-970fa6cc-jsdnt" event={"ID":"ffabd9df-96b6-4e1c-913c-cb29cbfbd83a","Type":"ContainerStarted","Data":"663420b6d8e5264edefd731f5520403a9ca78b863effb64fc055f031d6a0ee49"} Nov 03 03:10:09.827847 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:09.827814 5605 generic.go:356] "Generic (PLEG): container finished" podID="ffabd9df-96b6-4e1c-913c-cb29cbfbd83a" containerID="b8b24ea820d90f60906b5e3a647800dee2ff953d260440f90fc233847c0b04f4" exitCode=0 Nov 03 03:10:09.828264 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:09.827907 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-970fa6cc-jsdnt" event={"ID":"ffabd9df-96b6-4e1c-913c-cb29cbfbd83a","Type":"ContainerDied","Data":"b8b24ea820d90f60906b5e3a647800dee2ff953d260440f90fc233847c0b04f4"} Nov 03 03:10:10.969881 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:10.969853 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-970fa6cc-jsdnt" Nov 03 03:10:11.005660 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:11.005628 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffabd9df-96b6-4e1c-913c-cb29cbfbd83a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffabd9df-96b6-4e1c-913c-cb29cbfbd83a-maas-api-openshift-ca\") pod \"ffabd9df-96b6-4e1c-913c-cb29cbfbd83a\" (UID: \"ffabd9df-96b6-4e1c-913c-cb29cbfbd83a\") " Nov 03 03:10:11.005829 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:11.005685 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffabd9df-96b6-4e1c-913c-cb29cbfbd83a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffabd9df-96b6-4e1c-913c-cb29cbfbd83a-maas-api-xks-ca\") pod \"ffabd9df-96b6-4e1c-913c-cb29cbfbd83a\" (UID: \"ffabd9df-96b6-4e1c-913c-cb29cbfbd83a\") " Nov 03 03:10:11.005829 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:11.005756 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffabd9df-96b6-4e1c-913c-cb29cbfbd83a-kube-api-access-tjtxc\" (UniqueName: \"kubernetes.io/projected/ffabd9df-96b6-4e1c-913c-cb29cbfbd83a-kube-api-access-tjtxc\") pod \"ffabd9df-96b6-4e1c-913c-cb29cbfbd83a\" (UID: \"ffabd9df-96b6-4e1c-913c-cb29cbfbd83a\") " Nov 03 03:10:11.006008 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:11.005985 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffabd9df-96b6-4e1c-913c-cb29cbfbd83a-maas-api-openshift-ca" pod "ffabd9df-96b6-4e1c-913c-cb29cbfbd83a" (UID: "ffabd9df-96b6-4e1c-913c-cb29cbfbd83a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:10:11.006225 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:11.006204 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffabd9df-96b6-4e1c-913c-cb29cbfbd83a-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:10:11.007745 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:11.007721 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffabd9df-96b6-4e1c-913c-cb29cbfbd83a-maas-api-xks-ca" pod "ffabd9df-96b6-4e1c-913c-cb29cbfbd83a" (UID: "ffabd9df-96b6-4e1c-913c-cb29cbfbd83a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:10:11.007866 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:11.007849 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffabd9df-96b6-4e1c-913c-cb29cbfbd83a-kube-api-access-tjtxc" pod "ffabd9df-96b6-4e1c-913c-cb29cbfbd83a" (UID: "ffabd9df-96b6-4e1c-913c-cb29cbfbd83a"). InnerVolumeSpecName "kube-api-access-tjtxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:10:11.107567 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:11.107475 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffabd9df-96b6-4e1c-913c-cb29cbfbd83a-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:10:11.107567 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:11.107503 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjtxc\" (UniqueName: \"kubernetes.io/projected/ffabd9df-96b6-4e1c-913c-cb29cbfbd83a-kube-api-access-tjtxc\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:10:11.839351 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:11.839313 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-970fa6cc-jsdnt" event={"ID":"ffabd9df-96b6-4e1c-913c-cb29cbfbd83a","Type":"ContainerDied","Data":"663420b6d8e5264edefd731f5520403a9ca78b863effb64fc055f031d6a0ee49"} Nov 03 03:10:11.839540 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:11.839381 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="663420b6d8e5264edefd731f5520403a9ca78b863effb64fc055f031d6a0ee49" Nov 03 03:10:11.839540 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:11.839330 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-970fa6cc-jsdnt" Nov 03 03:10:18.657726 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:18.657685 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08ba1427-r84b9"] Nov 03 03:10:18.658160 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:18.658108 5605 cpu_manager.go:414] "containerMap: removing container" podUID="ffabd9df-96b6-4e1c-913c-cb29cbfbd83a" containerName="revoke-keys" Nov 03 03:10:18.658160 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:18.658119 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffabd9df-96b6-4e1c-913c-cb29cbfbd83a" containerName="revoke-keys" Nov 03 03:10:18.658247 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:18.658194 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffabd9df-96b6-4e1c-913c-cb29cbfbd83a" containerName="revoke-keys" Nov 03 03:10:18.661508 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:18.661485 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ba1427-r84b9" Nov 03 03:10:18.711903 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:18.711873 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:10:18.758681 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:18.758641 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:10:18.758681 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:18.758663 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:10:18.758681 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:18.758687 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 03:10:18.760777 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:18.760756 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08ba1427-r84b9"] Nov 03 03:10:18.783752 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:18.783728 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1c56256-530d-408e-b532-d54ea4ecfac6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08ba1427-r84b9\" (UID: \"d1c56256-530d-408e-b532-d54ea4ecfac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ba1427-r84b9" Nov 03 03:10:18.783903 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:18.783780 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcrh6\" (UniqueName: \"kubernetes.io/projected/d1c56256-530d-408e-b532-d54ea4ecfac6-kube-api-access-qcrh6\") pod \"maas-api-revoke-sub-08ba1427-r84b9\" (UID: \"d1c56256-530d-408e-b532-d54ea4ecfac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ba1427-r84b9" Nov 03 03:10:18.783985 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:18.783963 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1c56256-530d-408e-b532-d54ea4ecfac6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08ba1427-r84b9\" (UID: \"d1c56256-530d-408e-b532-d54ea4ecfac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ba1427-r84b9" Nov 03 03:10:18.885543 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:18.885511 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1c56256-530d-408e-b532-d54ea4ecfac6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08ba1427-r84b9\" (UID: \"d1c56256-530d-408e-b532-d54ea4ecfac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ba1427-r84b9" Nov 03 03:10:18.885747 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:18.885571 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcrh6\" (UniqueName: \"kubernetes.io/projected/d1c56256-530d-408e-b532-d54ea4ecfac6-kube-api-access-qcrh6\") pod \"maas-api-revoke-sub-08ba1427-r84b9\" (UID: \"d1c56256-530d-408e-b532-d54ea4ecfac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ba1427-r84b9" Nov 03 03:10:18.885747 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:18.885650 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1c56256-530d-408e-b532-d54ea4ecfac6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08ba1427-r84b9\" (UID: \"d1c56256-530d-408e-b532-d54ea4ecfac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ba1427-r84b9" Nov 03 03:10:18.886201 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:18.886170 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1c56256-530d-408e-b532-d54ea4ecfac6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08ba1427-r84b9\" (UID: \"d1c56256-530d-408e-b532-d54ea4ecfac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ba1427-r84b9" Nov 03 03:10:18.888006 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:18.887989 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1c56256-530d-408e-b532-d54ea4ecfac6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08ba1427-r84b9\" (UID: \"d1c56256-530d-408e-b532-d54ea4ecfac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ba1427-r84b9" Nov 03 03:10:18.938870 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:18.938791 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcrh6\" (UniqueName: \"kubernetes.io/projected/d1c56256-530d-408e-b532-d54ea4ecfac6-kube-api-access-qcrh6\") pod \"maas-api-revoke-sub-08ba1427-r84b9\" (UID: \"d1c56256-530d-408e-b532-d54ea4ecfac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ba1427-r84b9" Nov 03 03:10:18.974737 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:18.974695 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ba1427-r84b9" Nov 03 03:10:19.346862 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:19.346823 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08ba1427-r84b9"] Nov 03 03:10:19.353398 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:10:19.353347 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1c56256_530d_408e_b532_d54ea4ecfac6.slice/crio-21c806bb2318f57e28fd5ea90730d04ff5d13b101d9df60fba5b5703bc3711c7 WatchSource:0}: Error finding container 21c806bb2318f57e28fd5ea90730d04ff5d13b101d9df60fba5b5703bc3711c7: Status 404 returned error can't find the container with id 21c806bb2318f57e28fd5ea90730d04ff5d13b101d9df60fba5b5703bc3711c7 Nov 03 03:10:19.880734 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:19.880697 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ba1427-r84b9" event={"ID":"d1c56256-530d-408e-b532-d54ea4ecfac6","Type":"ContainerStarted","Data":"0cdeff5a2245ae57533588de7c0872c979e68e282c413e77147eeb5bde3ac5a8"} Nov 03 03:10:19.881118 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:19.880759 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ba1427-r84b9" event={"ID":"d1c56256-530d-408e-b532-d54ea4ecfac6","Type":"ContainerStarted","Data":"21c806bb2318f57e28fd5ea90730d04ff5d13b101d9df60fba5b5703bc3711c7"} Nov 03 03:10:19.987969 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:19.987867 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ba1427-r84b9" podStartSLOduration=1.987851904 podStartE2EDuration="1.987851904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:10:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:10:19.986864802 +0000 UTC m=+4697.246062981" watchObservedRunningTime="2026-11-03 03:10:19.987851904 +0000 UTC m=+4697.247050057" Nov 03 03:10:20.887068 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:20.887030 5605 generic.go:356] "Generic (PLEG): container finished" podID="d1c56256-530d-408e-b532-d54ea4ecfac6" containerID="0cdeff5a2245ae57533588de7c0872c979e68e282c413e77147eeb5bde3ac5a8" exitCode=0 Nov 03 03:10:20.887483 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:20.887120 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ba1427-r84b9" event={"ID":"d1c56256-530d-408e-b532-d54ea4ecfac6","Type":"ContainerDied","Data":"0cdeff5a2245ae57533588de7c0872c979e68e282c413e77147eeb5bde3ac5a8"} Nov 03 03:10:22.032088 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:22.032060 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ba1427-r84b9" Nov 03 03:10:22.119616 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:22.119589 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1c56256-530d-408e-b532-d54ea4ecfac6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1c56256-530d-408e-b532-d54ea4ecfac6-maas-api-openshift-ca\") pod \"d1c56256-530d-408e-b532-d54ea4ecfac6\" (UID: \"d1c56256-530d-408e-b532-d54ea4ecfac6\") " Nov 03 03:10:22.119779 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:22.119642 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1c56256-530d-408e-b532-d54ea4ecfac6-kube-api-access-qcrh6\" (UniqueName: \"kubernetes.io/projected/d1c56256-530d-408e-b532-d54ea4ecfac6-kube-api-access-qcrh6\") pod \"d1c56256-530d-408e-b532-d54ea4ecfac6\" (UID: \"d1c56256-530d-408e-b532-d54ea4ecfac6\") " Nov 03 03:10:22.119779 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:22.119754 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1c56256-530d-408e-b532-d54ea4ecfac6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1c56256-530d-408e-b532-d54ea4ecfac6-maas-api-xks-ca\") pod \"d1c56256-530d-408e-b532-d54ea4ecfac6\" (UID: \"d1c56256-530d-408e-b532-d54ea4ecfac6\") " Nov 03 03:10:22.119985 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:22.119963 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1c56256-530d-408e-b532-d54ea4ecfac6-maas-api-openshift-ca" pod "d1c56256-530d-408e-b532-d54ea4ecfac6" (UID: "d1c56256-530d-408e-b532-d54ea4ecfac6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:10:22.120114 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:22.120100 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1c56256-530d-408e-b532-d54ea4ecfac6-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:10:22.121723 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:22.121699 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1c56256-530d-408e-b532-d54ea4ecfac6-kube-api-access-qcrh6" pod "d1c56256-530d-408e-b532-d54ea4ecfac6" (UID: "d1c56256-530d-408e-b532-d54ea4ecfac6"). InnerVolumeSpecName "kube-api-access-qcrh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:10:22.121723 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:22.121708 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1c56256-530d-408e-b532-d54ea4ecfac6-maas-api-xks-ca" pod "d1c56256-530d-408e-b532-d54ea4ecfac6" (UID: "d1c56256-530d-408e-b532-d54ea4ecfac6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:10:22.221527 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:22.221432 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcrh6\" (UniqueName: \"kubernetes.io/projected/d1c56256-530d-408e-b532-d54ea4ecfac6-kube-api-access-qcrh6\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:10:22.221527 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:22.221469 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1c56256-530d-408e-b532-d54ea4ecfac6-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:10:22.897431 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:22.897397 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ba1427-r84b9" event={"ID":"d1c56256-530d-408e-b532-d54ea4ecfac6","Type":"ContainerDied","Data":"21c806bb2318f57e28fd5ea90730d04ff5d13b101d9df60fba5b5703bc3711c7"} Nov 03 03:10:22.897431 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:22.897416 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ba1427-r84b9" Nov 03 03:10:22.897624 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:22.897446 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21c806bb2318f57e28fd5ea90730d04ff5d13b101d9df60fba5b5703bc3711c7" Nov 03 03:10:23.912089 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:10:23.912049 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:10:31.813996 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:31.813959 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-9aebe8/rate-test-model-9aebe8-kserve-76d8d8c6c7-bnqlv"] Nov 03 03:10:33.209116 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:33.209085 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-ffba43/rate-test-model-ffba43-kserve-78c4969977-mrssx"] Nov 03 03:10:40.034565 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:40.034522 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:10:46.563467 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:46.563430 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7232-3e2ac9"] Nov 03 03:10:46.563947 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:46.563929 5605 cpu_manager.go:414] "containerMap: removing container" podUID="d1c56256-530d-408e-b532-d54ea4ecfac6" containerName="revoke-keys" Nov 03 03:10:46.564015 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:46.563948 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1c56256-530d-408e-b532-d54ea4ecfac6" containerName="revoke-keys" Nov 03 03:10:46.564071 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:46.564016 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1c56256-530d-408e-b532-d54ea4ecfac6" containerName="revoke-keys" Nov 03 03:10:46.567186 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:46.567156 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7232-3e2ac9" Nov 03 03:10:46.618788 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:46.618753 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7232-3e2ac9"] Nov 03 03:10:46.662734 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:46.662699 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-579jc\"" Nov 03 03:10:46.668496 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:46.668468 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md9vk\" (UniqueName: \"kubernetes.io/projected/4fde13d1-2c1e-45fa-ade9-a41a605ccb34-kube-api-access-md9vk\") pod \"test-curl-7232-3e2ac9\" (UID: \"4fde13d1-2c1e-45fa-ade9-a41a605ccb34\") " pod="opendatahub/test-curl-7232-3e2ac9" Nov 03 03:10:46.769822 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:46.769780 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md9vk\" (UniqueName: \"kubernetes.io/projected/4fde13d1-2c1e-45fa-ade9-a41a605ccb34-kube-api-access-md9vk\") pod \"test-curl-7232-3e2ac9\" (UID: \"4fde13d1-2c1e-45fa-ade9-a41a605ccb34\") " pod="opendatahub/test-curl-7232-3e2ac9" Nov 03 03:10:46.823578 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:46.823497 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-md9vk\" (UniqueName: \"kubernetes.io/projected/4fde13d1-2c1e-45fa-ade9-a41a605ccb34-kube-api-access-md9vk\") pod \"test-curl-7232-3e2ac9\" (UID: \"4fde13d1-2c1e-45fa-ade9-a41a605ccb34\") " pod="opendatahub/test-curl-7232-3e2ac9" Nov 03 03:10:46.833117 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:46.833077 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-9aebe8/rate-test-model-9aebe8-kserve-76d8d8c6c7-bnqlv" podUID="e54a46d9-0329-4076-9d46-c63d5827f8ea" containerName="main" containerID="cri-o://b45d67b3a9a08bc6d2f01461f608595e3bab006247418972ffffe0dd9b1f389f" gracePeriod=45 Nov 03 03:10:46.887802 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:46.887765 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7232-3e2ac9" Nov 03 03:10:47.032891 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:47.032858 5605 generic.go:356] "Generic (PLEG): container finished" podID="e54a46d9-0329-4076-9d46-c63d5827f8ea" containerID="b45d67b3a9a08bc6d2f01461f608595e3bab006247418972ffffe0dd9b1f389f" exitCode=0 Nov 03 03:10:47.033070 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:47.032936 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-9aebe8/rate-test-model-9aebe8-kserve-76d8d8c6c7-bnqlv" event={"ID":"e54a46d9-0329-4076-9d46-c63d5827f8ea","Type":"ContainerDied","Data":"b45d67b3a9a08bc6d2f01461f608595e3bab006247418972ffffe0dd9b1f389f"} Nov 03 03:10:47.096651 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:47.096624 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-9aebe8/rate-test-model-9aebe8-kserve-76d8d8c6c7-bnqlv" Nov 03 03:10:47.173245 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:47.173204 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e54a46d9-0329-4076-9d46-c63d5827f8ea-home\" (UniqueName: \"kubernetes.io/empty-dir/e54a46d9-0329-4076-9d46-c63d5827f8ea-home\") pod \"e54a46d9-0329-4076-9d46-c63d5827f8ea\" (UID: \"e54a46d9-0329-4076-9d46-c63d5827f8ea\") " Nov 03 03:10:47.173461 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:47.173326 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e54a46d9-0329-4076-9d46-c63d5827f8ea-tls-certs\" (UniqueName: \"kubernetes.io/secret/e54a46d9-0329-4076-9d46-c63d5827f8ea-tls-certs\") pod \"e54a46d9-0329-4076-9d46-c63d5827f8ea\" (UID: \"e54a46d9-0329-4076-9d46-c63d5827f8ea\") " Nov 03 03:10:47.173461 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:47.173424 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e54a46d9-0329-4076-9d46-c63d5827f8ea-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e54a46d9-0329-4076-9d46-c63d5827f8ea-dshm\") pod \"e54a46d9-0329-4076-9d46-c63d5827f8ea\" (UID: \"e54a46d9-0329-4076-9d46-c63d5827f8ea\") " Nov 03 03:10:47.173587 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:47.173475 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e54a46d9-0329-4076-9d46-c63d5827f8ea-kube-api-access-2gwvc\" (UniqueName: \"kubernetes.io/projected/e54a46d9-0329-4076-9d46-c63d5827f8ea-kube-api-access-2gwvc\") pod \"e54a46d9-0329-4076-9d46-c63d5827f8ea\" (UID: \"e54a46d9-0329-4076-9d46-c63d5827f8ea\") " Nov 03 03:10:47.173587 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:47.173527 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e54a46d9-0329-4076-9d46-c63d5827f8ea-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e54a46d9-0329-4076-9d46-c63d5827f8ea-tmp-dir\") pod \"e54a46d9-0329-4076-9d46-c63d5827f8ea\" (UID: \"e54a46d9-0329-4076-9d46-c63d5827f8ea\") " Nov 03 03:10:47.173587 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:47.173535 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e54a46d9-0329-4076-9d46-c63d5827f8ea-home" pod "e54a46d9-0329-4076-9d46-c63d5827f8ea" (UID: "e54a46d9-0329-4076-9d46-c63d5827f8ea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:10:47.173764 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:47.173621 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e54a46d9-0329-4076-9d46-c63d5827f8ea-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e54a46d9-0329-4076-9d46-c63d5827f8ea-model-cache\") pod \"e54a46d9-0329-4076-9d46-c63d5827f8ea\" (UID: \"e54a46d9-0329-4076-9d46-c63d5827f8ea\") " Nov 03 03:10:47.173855 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:47.173807 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e54a46d9-0329-4076-9d46-c63d5827f8ea-tmp-dir" pod "e54a46d9-0329-4076-9d46-c63d5827f8ea" (UID: "e54a46d9-0329-4076-9d46-c63d5827f8ea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:10:47.174028 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:47.174001 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e54a46d9-0329-4076-9d46-c63d5827f8ea-model-cache" pod "e54a46d9-0329-4076-9d46-c63d5827f8ea" (UID: "e54a46d9-0329-4076-9d46-c63d5827f8ea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:10:47.174028 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:47.174019 5605 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e54a46d9-0329-4076-9d46-c63d5827f8ea-tmp-dir\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:10:47.174171 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:47.174040 5605 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e54a46d9-0329-4076-9d46-c63d5827f8ea-home\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:10:47.175705 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:47.175680 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e54a46d9-0329-4076-9d46-c63d5827f8ea-dshm" pod "e54a46d9-0329-4076-9d46-c63d5827f8ea" (UID: "e54a46d9-0329-4076-9d46-c63d5827f8ea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:10:47.175855 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:47.175776 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e54a46d9-0329-4076-9d46-c63d5827f8ea-kube-api-access-2gwvc" pod "e54a46d9-0329-4076-9d46-c63d5827f8ea" (UID: "e54a46d9-0329-4076-9d46-c63d5827f8ea"). InnerVolumeSpecName "kube-api-access-2gwvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:10:47.176068 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:47.176053 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e54a46d9-0329-4076-9d46-c63d5827f8ea-tls-certs" pod "e54a46d9-0329-4076-9d46-c63d5827f8ea" (UID: "e54a46d9-0329-4076-9d46-c63d5827f8ea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:10:47.257100 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:47.257060 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7232-3e2ac9"] Nov 03 03:10:47.263108 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:10:47.263076 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4fde13d1_2c1e_45fa_ade9_a41a605ccb34.slice/crio-e83c671af794f7f2fea8fea516ff1ed6310fb223427e3eec54acbab339391f01 WatchSource:0}: Error finding container e83c671af794f7f2fea8fea516ff1ed6310fb223427e3eec54acbab339391f01: Status 404 returned error can't find the container with id e83c671af794f7f2fea8fea516ff1ed6310fb223427e3eec54acbab339391f01 Nov 03 03:10:47.274659 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:47.274630 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gwvc\" (UniqueName: \"kubernetes.io/projected/e54a46d9-0329-4076-9d46-c63d5827f8ea-kube-api-access-2gwvc\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:10:47.274782 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:47.274663 5605 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e54a46d9-0329-4076-9d46-c63d5827f8ea-model-cache\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:10:47.274782 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:47.274679 5605 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e54a46d9-0329-4076-9d46-c63d5827f8ea-tls-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:10:47.274782 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:47.274691 5605 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e54a46d9-0329-4076-9d46-c63d5827f8ea-dshm\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:10:48.039762 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:48.039733 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-9aebe8/rate-test-model-9aebe8-kserve-76d8d8c6c7-bnqlv" Nov 03 03:10:48.040259 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:48.039781 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-9aebe8/rate-test-model-9aebe8-kserve-76d8d8c6c7-bnqlv" event={"ID":"e54a46d9-0329-4076-9d46-c63d5827f8ea","Type":"ContainerDied","Data":"ff32253792a9336a85cdeb250105b370fa23671958cf0614c3d93a9549422389"} Nov 03 03:10:48.040259 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:48.039830 5605 scope.go:122] "RemoveContainer" containerID="b45d67b3a9a08bc6d2f01461f608595e3bab006247418972ffffe0dd9b1f389f" Nov 03 03:10:48.041549 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:48.041301 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7232-3e2ac9" event={"ID":"4fde13d1-2c1e-45fa-ade9-a41a605ccb34","Type":"ContainerStarted","Data":"5c4561e75fb2c856af7c0deb45e0c38e7c4127388ed7d7f32af27d8ba13e4b0d"} Nov 03 03:10:48.041549 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:48.041330 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7232-3e2ac9" event={"ID":"4fde13d1-2c1e-45fa-ade9-a41a605ccb34","Type":"ContainerStarted","Data":"e83c671af794f7f2fea8fea516ff1ed6310fb223427e3eec54acbab339391f01"} Nov 03 03:10:48.192591 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:48.192522 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-9aebe8/rate-test-model-9aebe8-kserve-76d8d8c6c7-bnqlv"] Nov 03 03:10:48.228665 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:48.228625 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-ffba43/rate-test-model-ffba43-kserve-78c4969977-mrssx" podUID="7c7d95c2-f8bc-4b44-b696-4db10b9a9207" containerName="main" containerID="cri-o://7cb8599912c9d137947e7ec18a20a3508c31d2a68a0d0414efa2406cf45e48b0" gracePeriod=45 Nov 03 03:10:48.239260 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:48.239236 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-9aebe8/rate-test-model-9aebe8-kserve-76d8d8c6c7-bnqlv"] Nov 03 03:10:48.253082 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:48.253050 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-ffba43/rate-test-model-ffba43-kserve-78c4969977-mrssx" podUID="7c7d95c2-f8bc-4b44-b696-4db10b9a9207" containerName="main" probeResult="failure" output="Get \"https://10.134.1.3:8000/ready\": dial tcp 10.134.1.3:8000: connect: connection refused" Nov 03 03:10:48.342467 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:48.342414 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7232-3e2ac9" podStartSLOduration=2.342401027 podStartE2EDuration="2.342401027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:10:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:10:48.340541652 +0000 UTC m=+4725.599739808" watchObservedRunningTime="2026-11-03 03:10:48.342401027 +0000 UTC m=+4725.601599180" Nov 03 03:10:48.482459 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:48.482432 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-ffba43/rate-test-model-ffba43-kserve-78c4969977-mrssx" Nov 03 03:10:48.587930 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:48.587899 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c7d95c2-f8bc-4b44-b696-4db10b9a9207-kube-api-access-x9hgk\" (UniqueName: \"kubernetes.io/projected/7c7d95c2-f8bc-4b44-b696-4db10b9a9207-kube-api-access-x9hgk\") pod \"7c7d95c2-f8bc-4b44-b696-4db10b9a9207\" (UID: \"7c7d95c2-f8bc-4b44-b696-4db10b9a9207\") " Nov 03 03:10:48.588120 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:48.587957 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c7d95c2-f8bc-4b44-b696-4db10b9a9207-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c7d95c2-f8bc-4b44-b696-4db10b9a9207-tmp-dir\") pod \"7c7d95c2-f8bc-4b44-b696-4db10b9a9207\" (UID: \"7c7d95c2-f8bc-4b44-b696-4db10b9a9207\") " Nov 03 03:10:48.588120 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:48.587991 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c7d95c2-f8bc-4b44-b696-4db10b9a9207-tls-certs\" (UniqueName: \"kubernetes.io/secret/7c7d95c2-f8bc-4b44-b696-4db10b9a9207-tls-certs\") pod \"7c7d95c2-f8bc-4b44-b696-4db10b9a9207\" (UID: \"7c7d95c2-f8bc-4b44-b696-4db10b9a9207\") " Nov 03 03:10:48.588120 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:48.588050 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c7d95c2-f8bc-4b44-b696-4db10b9a9207-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c7d95c2-f8bc-4b44-b696-4db10b9a9207-dshm\") pod \"7c7d95c2-f8bc-4b44-b696-4db10b9a9207\" (UID: \"7c7d95c2-f8bc-4b44-b696-4db10b9a9207\") " Nov 03 03:10:48.588120 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:48.588079 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c7d95c2-f8bc-4b44-b696-4db10b9a9207-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c7d95c2-f8bc-4b44-b696-4db10b9a9207-model-cache\") pod \"7c7d95c2-f8bc-4b44-b696-4db10b9a9207\" (UID: \"7c7d95c2-f8bc-4b44-b696-4db10b9a9207\") " Nov 03 03:10:48.588342 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:48.588131 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c7d95c2-f8bc-4b44-b696-4db10b9a9207-home\" (UniqueName: \"kubernetes.io/empty-dir/7c7d95c2-f8bc-4b44-b696-4db10b9a9207-home\") pod \"7c7d95c2-f8bc-4b44-b696-4db10b9a9207\" (UID: \"7c7d95c2-f8bc-4b44-b696-4db10b9a9207\") " Nov 03 03:10:48.588342 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:48.588228 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c7d95c2-f8bc-4b44-b696-4db10b9a9207-tmp-dir" pod "7c7d95c2-f8bc-4b44-b696-4db10b9a9207" (UID: "7c7d95c2-f8bc-4b44-b696-4db10b9a9207"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:10:48.588948 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:48.588449 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c7d95c2-f8bc-4b44-b696-4db10b9a9207-model-cache" pod "7c7d95c2-f8bc-4b44-b696-4db10b9a9207" (UID: "7c7d95c2-f8bc-4b44-b696-4db10b9a9207"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:10:48.588948 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:48.588465 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c7d95c2-f8bc-4b44-b696-4db10b9a9207-home" pod "7c7d95c2-f8bc-4b44-b696-4db10b9a9207" (UID: "7c7d95c2-f8bc-4b44-b696-4db10b9a9207"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:10:48.588948 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:48.588490 5605 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c7d95c2-f8bc-4b44-b696-4db10b9a9207-tmp-dir\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:10:48.590774 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:48.590693 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c7d95c2-f8bc-4b44-b696-4db10b9a9207-tls-certs" pod "7c7d95c2-f8bc-4b44-b696-4db10b9a9207" (UID: "7c7d95c2-f8bc-4b44-b696-4db10b9a9207"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:10:48.590907 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:48.590873 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c7d95c2-f8bc-4b44-b696-4db10b9a9207-kube-api-access-x9hgk" pod "7c7d95c2-f8bc-4b44-b696-4db10b9a9207" (UID: "7c7d95c2-f8bc-4b44-b696-4db10b9a9207"). InnerVolumeSpecName "kube-api-access-x9hgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:10:48.591490 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:48.591463 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c7d95c2-f8bc-4b44-b696-4db10b9a9207-dshm" pod "7c7d95c2-f8bc-4b44-b696-4db10b9a9207" (UID: "7c7d95c2-f8bc-4b44-b696-4db10b9a9207"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:10:48.689876 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:48.689849 5605 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c7d95c2-f8bc-4b44-b696-4db10b9a9207-home\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:10:48.689876 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:48.689872 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9hgk\" (UniqueName: \"kubernetes.io/projected/7c7d95c2-f8bc-4b44-b696-4db10b9a9207-kube-api-access-x9hgk\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:10:48.690053 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:48.689885 5605 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c7d95c2-f8bc-4b44-b696-4db10b9a9207-tls-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:10:48.690053 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:48.689895 5605 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c7d95c2-f8bc-4b44-b696-4db10b9a9207-dshm\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:10:48.690053 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:48.689903 5605 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c7d95c2-f8bc-4b44-b696-4db10b9a9207-model-cache\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:10:48.849301 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:48.849276 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e54a46d9-0329-4076-9d46-c63d5827f8ea" path="/var/lib/kubelet/pods/e54a46d9-0329-4076-9d46-c63d5827f8ea/volumes" Nov 03 03:10:49.047789 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:49.047757 5605 generic.go:356] "Generic (PLEG): container finished" podID="7c7d95c2-f8bc-4b44-b696-4db10b9a9207" containerID="7cb8599912c9d137947e7ec18a20a3508c31d2a68a0d0414efa2406cf45e48b0" exitCode=0 Nov 03 03:10:49.048172 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:49.047799 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-ffba43/rate-test-model-ffba43-kserve-78c4969977-mrssx" Nov 03 03:10:49.048172 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:49.047839 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-ffba43/rate-test-model-ffba43-kserve-78c4969977-mrssx" event={"ID":"7c7d95c2-f8bc-4b44-b696-4db10b9a9207","Type":"ContainerDied","Data":"7cb8599912c9d137947e7ec18a20a3508c31d2a68a0d0414efa2406cf45e48b0"} Nov 03 03:10:49.048172 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:49.047879 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-ffba43/rate-test-model-ffba43-kserve-78c4969977-mrssx" event={"ID":"7c7d95c2-f8bc-4b44-b696-4db10b9a9207","Type":"ContainerDied","Data":"b11ea3e7a1cc6dadc8482564398ad0c73e8475504a752ad406396c5eef1ce183"} Nov 03 03:10:49.048172 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:49.047897 5605 scope.go:122] "RemoveContainer" containerID="7cb8599912c9d137947e7ec18a20a3508c31d2a68a0d0414efa2406cf45e48b0" Nov 03 03:10:49.056453 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:49.056430 5605 scope.go:122] "RemoveContainer" containerID="7cb8599912c9d137947e7ec18a20a3508c31d2a68a0d0414efa2406cf45e48b0" Nov 03 03:10:49.056710 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:10:49.056695 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cb8599912c9d137947e7ec18a20a3508c31d2a68a0d0414efa2406cf45e48b0\": container with ID starting with 7cb8599912c9d137947e7ec18a20a3508c31d2a68a0d0414efa2406cf45e48b0 not found: ID does not exist" containerID="7cb8599912c9d137947e7ec18a20a3508c31d2a68a0d0414efa2406cf45e48b0" Nov 03 03:10:49.056759 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:49.056717 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cb8599912c9d137947e7ec18a20a3508c31d2a68a0d0414efa2406cf45e48b0"} err="failed to get container status \"7cb8599912c9d137947e7ec18a20a3508c31d2a68a0d0414efa2406cf45e48b0\": rpc error: code = NotFound desc = could not find container \"7cb8599912c9d137947e7ec18a20a3508c31d2a68a0d0414efa2406cf45e48b0\": container with ID starting with 7cb8599912c9d137947e7ec18a20a3508c31d2a68a0d0414efa2406cf45e48b0 not found: ID does not exist" Nov 03 03:10:49.200820 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:49.200762 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-ffba43/rate-test-model-ffba43-kserve-78c4969977-mrssx"] Nov 03 03:10:49.204750 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:49.204732 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-ffba43/rate-test-model-ffba43-kserve-78c4969977-mrssx"] Nov 03 03:10:49.754657 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:49.754622 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7232-3e2ac9"] Nov 03 03:10:49.759123 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:49.759096 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7232-3e2ac9"] Nov 03 03:10:49.908678 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:49.908643 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7232-2c2c15"] Nov 03 03:10:49.909073 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:49.909055 5605 cpu_manager.go:414] "containerMap: removing container" podUID="7c7d95c2-f8bc-4b44-b696-4db10b9a9207" containerName="main" Nov 03 03:10:49.909170 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:49.909075 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c7d95c2-f8bc-4b44-b696-4db10b9a9207" containerName="main" Nov 03 03:10:49.909170 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:49.909091 5605 cpu_manager.go:414] "containerMap: removing container" podUID="4fde13d1-2c1e-45fa-ade9-a41a605ccb34" containerName="test-curl-7232-3e2ac9" Nov 03 03:10:49.909170 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:49.909098 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fde13d1-2c1e-45fa-ade9-a41a605ccb34" containerName="test-curl-7232-3e2ac9" Nov 03 03:10:49.909170 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:49.909118 5605 cpu_manager.go:414] "containerMap: removing container" podUID="e54a46d9-0329-4076-9d46-c63d5827f8ea" containerName="main" Nov 03 03:10:49.909170 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:49.909126 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e54a46d9-0329-4076-9d46-c63d5827f8ea" containerName="main" Nov 03 03:10:49.909509 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:49.909284 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c7d95c2-f8bc-4b44-b696-4db10b9a9207" containerName="main" Nov 03 03:10:49.909509 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:49.909301 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fde13d1-2c1e-45fa-ade9-a41a605ccb34" containerName="test-curl-7232-3e2ac9" Nov 03 03:10:49.909509 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:49.909316 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="e54a46d9-0329-4076-9d46-c63d5827f8ea" containerName="main" Nov 03 03:10:49.913967 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:49.913948 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7232-2c2c15" Nov 03 03:10:49.962396 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:49.962374 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7232-2c2c15"] Nov 03 03:10:50.001004 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:50.000975 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp469\" (UniqueName: \"kubernetes.io/projected/fbb90dbd-c248-4132-b71e-a9703cbf1efe-kube-api-access-fp469\") pod \"test-curl-7232-2c2c15\" (UID: \"fbb90dbd-c248-4132-b71e-a9703cbf1efe\") " pod="opendatahub/test-curl-7232-2c2c15" Nov 03 03:10:50.054160 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:50.054133 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7232-3e2ac9" podUID="4fde13d1-2c1e-45fa-ade9-a41a605ccb34" containerName="test-curl-7232-3e2ac9" containerID="cri-o://5c4561e75fb2c856af7c0deb45e0c38e7c4127388ed7d7f32af27d8ba13e4b0d" gracePeriod=2 Nov 03 03:10:50.101942 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:50.101917 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp469\" (UniqueName: \"kubernetes.io/projected/fbb90dbd-c248-4132-b71e-a9703cbf1efe-kube-api-access-fp469\") pod \"test-curl-7232-2c2c15\" (UID: \"fbb90dbd-c248-4132-b71e-a9703cbf1efe\") " pod="opendatahub/test-curl-7232-2c2c15" Nov 03 03:10:50.102068 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:10:50.102038 5605 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7232-3e2ac9\" is forbidden: User \"system:node:ip-10-0-140-140.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-140.us-west-2.compute.internal' and this object" podUID="4fde13d1-2c1e-45fa-ade9-a41a605ccb34" pod="opendatahub/test-curl-7232-3e2ac9" Nov 03 03:10:50.154978 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:50.154953 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp469\" (UniqueName: \"kubernetes.io/projected/fbb90dbd-c248-4132-b71e-a9703cbf1efe-kube-api-access-fp469\") pod \"test-curl-7232-2c2c15\" (UID: \"fbb90dbd-c248-4132-b71e-a9703cbf1efe\") " pod="opendatahub/test-curl-7232-2c2c15" Nov 03 03:10:50.225097 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:50.225069 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7232-2c2c15" Nov 03 03:10:50.590990 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:50.590960 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7232-2c2c15"] Nov 03 03:10:50.596533 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:10:50.596506 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfbb90dbd_c248_4132_b71e_a9703cbf1efe.slice/crio-9d438825c08f9fd99d90537d551da8d011d8714abbe225ea0e011377b1fed4ea WatchSource:0}: Error finding container 9d438825c08f9fd99d90537d551da8d011d8714abbe225ea0e011377b1fed4ea: Status 404 returned error can't find the container with id 9d438825c08f9fd99d90537d551da8d011d8714abbe225ea0e011377b1fed4ea Nov 03 03:10:50.849617 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:50.849530 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c7d95c2-f8bc-4b44-b696-4db10b9a9207" path="/var/lib/kubelet/pods/7c7d95c2-f8bc-4b44-b696-4db10b9a9207/volumes" Nov 03 03:10:51.061606 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:51.061565 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7232-2c2c15" event={"ID":"fbb90dbd-c248-4132-b71e-a9703cbf1efe","Type":"ContainerStarted","Data":"db24f13217aef493d43b8c1f1ec86d4240915321ab6890ab404daf01f782d6aa"} Nov 03 03:10:51.061983 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:51.061619 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7232-2c2c15" event={"ID":"fbb90dbd-c248-4132-b71e-a9703cbf1efe","Type":"ContainerStarted","Data":"9d438825c08f9fd99d90537d551da8d011d8714abbe225ea0e011377b1fed4ea"} Nov 03 03:10:51.166093 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:51.166004 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7232-2c2c15" podStartSLOduration=2.165991804 podStartE2EDuration="2.165991804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:10:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:10:51.165247318 +0000 UTC m=+4728.424445473" watchObservedRunningTime="2026-11-03 03:10:51.165991804 +0000 UTC m=+4728.425189957" Nov 03 03:10:52.287970 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:52.287946 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7232-3e2ac9" Nov 03 03:10:52.323821 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:52.323734 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fde13d1-2c1e-45fa-ade9-a41a605ccb34-kube-api-access-md9vk\" (UniqueName: \"kubernetes.io/projected/4fde13d1-2c1e-45fa-ade9-a41a605ccb34-kube-api-access-md9vk\") pod \"4fde13d1-2c1e-45fa-ade9-a41a605ccb34\" (UID: \"4fde13d1-2c1e-45fa-ade9-a41a605ccb34\") " Nov 03 03:10:52.326247 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:52.326218 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fde13d1-2c1e-45fa-ade9-a41a605ccb34-kube-api-access-md9vk" pod "4fde13d1-2c1e-45fa-ade9-a41a605ccb34" (UID: "4fde13d1-2c1e-45fa-ade9-a41a605ccb34"). InnerVolumeSpecName "kube-api-access-md9vk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:10:52.424918 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:52.424876 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-md9vk\" (UniqueName: \"kubernetes.io/projected/4fde13d1-2c1e-45fa-ade9-a41a605ccb34-kube-api-access-md9vk\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:10:52.524097 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:52.524060 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7232-2c2c15"] Nov 03 03:10:52.527018 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:52.526993 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7232-2c2c15"] Nov 03 03:10:52.850301 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:52.850268 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fde13d1-2c1e-45fa-ade9-a41a605ccb34" path="/var/lib/kubelet/pods/4fde13d1-2c1e-45fa-ade9-a41a605ccb34/volumes" Nov 03 03:10:53.072482 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:53.072444 5605 generic.go:356] "Generic (PLEG): container finished" podID="4fde13d1-2c1e-45fa-ade9-a41a605ccb34" containerID="5c4561e75fb2c856af7c0deb45e0c38e7c4127388ed7d7f32af27d8ba13e4b0d" exitCode=137 Nov 03 03:10:53.072690 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:53.072491 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7232-3e2ac9" Nov 03 03:10:53.072690 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:53.072544 5605 scope.go:122] "RemoveContainer" containerID="5c4561e75fb2c856af7c0deb45e0c38e7c4127388ed7d7f32af27d8ba13e4b0d" Nov 03 03:10:53.072690 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:53.072633 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7232-2c2c15" podUID="fbb90dbd-c248-4132-b71e-a9703cbf1efe" containerName="test-curl-7232-2c2c15" containerID="cri-o://db24f13217aef493d43b8c1f1ec86d4240915321ab6890ab404daf01f782d6aa" gracePeriod=2 Nov 03 03:10:53.081197 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:53.081179 5605 scope.go:122] "RemoveContainer" containerID="5c4561e75fb2c856af7c0deb45e0c38e7c4127388ed7d7f32af27d8ba13e4b0d" Nov 03 03:10:53.081503 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:10:53.081483 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c4561e75fb2c856af7c0deb45e0c38e7c4127388ed7d7f32af27d8ba13e4b0d\": container with ID starting with 5c4561e75fb2c856af7c0deb45e0c38e7c4127388ed7d7f32af27d8ba13e4b0d not found: ID does not exist" containerID="5c4561e75fb2c856af7c0deb45e0c38e7c4127388ed7d7f32af27d8ba13e4b0d" Nov 03 03:10:53.081556 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:53.081512 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c4561e75fb2c856af7c0deb45e0c38e7c4127388ed7d7f32af27d8ba13e4b0d"} err="failed to get container status \"5c4561e75fb2c856af7c0deb45e0c38e7c4127388ed7d7f32af27d8ba13e4b0d\": rpc error: code = NotFound desc = could not find container \"5c4561e75fb2c856af7c0deb45e0c38e7c4127388ed7d7f32af27d8ba13e4b0d\": container with ID starting with 5c4561e75fb2c856af7c0deb45e0c38e7c4127388ed7d7f32af27d8ba13e4b0d not found: ID does not exist" Nov 03 03:10:53.589210 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:53.589170 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b83899d3-v4qdb"] Nov 03 03:10:53.589651 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:53.589589 5605 cpu_manager.go:414] "containerMap: removing container" podUID="fbb90dbd-c248-4132-b71e-a9703cbf1efe" containerName="test-curl-7232-2c2c15" Nov 03 03:10:53.589651 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:53.589602 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbb90dbd-c248-4132-b71e-a9703cbf1efe" containerName="test-curl-7232-2c2c15" Nov 03 03:10:53.589756 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:53.589682 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbb90dbd-c248-4132-b71e-a9703cbf1efe" containerName="test-curl-7232-2c2c15" Nov 03 03:10:53.594226 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:53.594204 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b83899d3-v4qdb" Nov 03 03:10:53.637285 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:53.637245 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk6x6\" (UniqueName: \"kubernetes.io/projected/2a8641cb-d366-4df0-8ea4-c62b50b77428-kube-api-access-fk6x6\") pod \"maas-api-revoke-sub-b83899d3-v4qdb\" (UID: \"2a8641cb-d366-4df0-8ea4-c62b50b77428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b83899d3-v4qdb" Nov 03 03:10:53.637285 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:53.637283 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a8641cb-d366-4df0-8ea4-c62b50b77428-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b83899d3-v4qdb\" (UID: \"2a8641cb-d366-4df0-8ea4-c62b50b77428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b83899d3-v4qdb" Nov 03 03:10:53.637515 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:53.637450 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a8641cb-d366-4df0-8ea4-c62b50b77428-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b83899d3-v4qdb\" (UID: \"2a8641cb-d366-4df0-8ea4-c62b50b77428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b83899d3-v4qdb" Nov 03 03:10:53.646433 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:53.646393 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b83899d3-v4qdb"] Nov 03 03:10:53.690647 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:53.690604 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:10:53.690647 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:53.690611 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:10:53.690647 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:53.690649 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 03:10:53.690904 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:53.690613 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:10:53.738669 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:53.738629 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a8641cb-d366-4df0-8ea4-c62b50b77428-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b83899d3-v4qdb\" (UID: \"2a8641cb-d366-4df0-8ea4-c62b50b77428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b83899d3-v4qdb" Nov 03 03:10:53.738866 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:53.738715 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk6x6\" (UniqueName: \"kubernetes.io/projected/2a8641cb-d366-4df0-8ea4-c62b50b77428-kube-api-access-fk6x6\") pod \"maas-api-revoke-sub-b83899d3-v4qdb\" (UID: \"2a8641cb-d366-4df0-8ea4-c62b50b77428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b83899d3-v4qdb" Nov 03 03:10:53.738866 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:53.738734 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a8641cb-d366-4df0-8ea4-c62b50b77428-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b83899d3-v4qdb\" (UID: \"2a8641cb-d366-4df0-8ea4-c62b50b77428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b83899d3-v4qdb" Nov 03 03:10:53.739430 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:53.739411 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a8641cb-d366-4df0-8ea4-c62b50b77428-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b83899d3-v4qdb\" (UID: \"2a8641cb-d366-4df0-8ea4-c62b50b77428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b83899d3-v4qdb" Nov 03 03:10:53.741101 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:53.741079 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a8641cb-d366-4df0-8ea4-c62b50b77428-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b83899d3-v4qdb\" (UID: \"2a8641cb-d366-4df0-8ea4-c62b50b77428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b83899d3-v4qdb" Nov 03 03:10:53.792127 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:53.792090 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk6x6\" (UniqueName: \"kubernetes.io/projected/2a8641cb-d366-4df0-8ea4-c62b50b77428-kube-api-access-fk6x6\") pod \"maas-api-revoke-sub-b83899d3-v4qdb\" (UID: \"2a8641cb-d366-4df0-8ea4-c62b50b77428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b83899d3-v4qdb" Nov 03 03:10:53.907183 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:53.907089 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b83899d3-v4qdb" Nov 03 03:10:53.912974 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:10:53.912937 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:10:54.275427 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:54.275321 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b83899d3-v4qdb"] Nov 03 03:10:54.281242 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:10:54.281213 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a8641cb_d366_4df0_8ea4_c62b50b77428.slice/crio-727a10b97a1f30cc91fdde7ecb1e53a0778a9c03f27834778493a30010e6e185 WatchSource:0}: Error finding container 727a10b97a1f30cc91fdde7ecb1e53a0778a9c03f27834778493a30010e6e185: Status 404 returned error can't find the container with id 727a10b97a1f30cc91fdde7ecb1e53a0778a9c03f27834778493a30010e6e185 Nov 03 03:10:55.042417 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:55.042313 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26741169-4jxgn"] Nov 03 03:10:55.086854 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:55.086821 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b83899d3-v4qdb" event={"ID":"2a8641cb-d366-4df0-8ea4-c62b50b77428","Type":"ContainerStarted","Data":"a9d39a253eaa219161f514ff022172bd46b281e77f617b59ce6e2525400fb115"} Nov 03 03:10:55.087013 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:55.086885 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b83899d3-v4qdb" event={"ID":"2a8641cb-d366-4df0-8ea4-c62b50b77428","Type":"ContainerStarted","Data":"727a10b97a1f30cc91fdde7ecb1e53a0778a9c03f27834778493a30010e6e185"} Nov 03 03:10:55.089111 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:55.089090 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26741169-4jxgn"] Nov 03 03:10:55.193936 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:55.193879 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b83899d3-v4qdb" podStartSLOduration=2.19386469 podStartE2EDuration="2.19386469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:10:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:10:55.192694024 +0000 UTC m=+4732.451892178" watchObservedRunningTime="2026-11-03 03:10:55.19386469 +0000 UTC m=+4732.453062841" Nov 03 03:10:55.298543 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:55.298519 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7232-2c2c15" Nov 03 03:10:55.353972 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:55.353935 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbb90dbd-c248-4132-b71e-a9703cbf1efe-kube-api-access-fp469\" (UniqueName: \"kubernetes.io/projected/fbb90dbd-c248-4132-b71e-a9703cbf1efe-kube-api-access-fp469\") pod \"fbb90dbd-c248-4132-b71e-a9703cbf1efe\" (UID: \"fbb90dbd-c248-4132-b71e-a9703cbf1efe\") " Nov 03 03:10:55.356019 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:55.355985 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbb90dbd-c248-4132-b71e-a9703cbf1efe-kube-api-access-fp469" pod "fbb90dbd-c248-4132-b71e-a9703cbf1efe" (UID: "fbb90dbd-c248-4132-b71e-a9703cbf1efe"). InnerVolumeSpecName "kube-api-access-fp469". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:10:55.455616 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:55.455574 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fp469\" (UniqueName: \"kubernetes.io/projected/fbb90dbd-c248-4132-b71e-a9703cbf1efe-kube-api-access-fp469\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:10:56.093933 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:56.093889 5605 generic.go:356] "Generic (PLEG): container finished" podID="2a8641cb-d366-4df0-8ea4-c62b50b77428" containerID="a9d39a253eaa219161f514ff022172bd46b281e77f617b59ce6e2525400fb115" exitCode=0 Nov 03 03:10:56.094430 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:56.093970 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b83899d3-v4qdb" event={"ID":"2a8641cb-d366-4df0-8ea4-c62b50b77428","Type":"ContainerDied","Data":"a9d39a253eaa219161f514ff022172bd46b281e77f617b59ce6e2525400fb115"} Nov 03 03:10:56.095322 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:56.095301 5605 generic.go:356] "Generic (PLEG): container finished" podID="fbb90dbd-c248-4132-b71e-a9703cbf1efe" containerID="db24f13217aef493d43b8c1f1ec86d4240915321ab6890ab404daf01f782d6aa" exitCode=137 Nov 03 03:10:56.095427 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:56.095347 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7232-2c2c15" Nov 03 03:10:56.095427 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:56.095400 5605 scope.go:122] "RemoveContainer" containerID="db24f13217aef493d43b8c1f1ec86d4240915321ab6890ab404daf01f782d6aa" Nov 03 03:10:56.105054 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:56.105027 5605 scope.go:122] "RemoveContainer" containerID="db24f13217aef493d43b8c1f1ec86d4240915321ab6890ab404daf01f782d6aa" Nov 03 03:10:56.105390 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:10:56.105352 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db24f13217aef493d43b8c1f1ec86d4240915321ab6890ab404daf01f782d6aa\": container with ID starting with db24f13217aef493d43b8c1f1ec86d4240915321ab6890ab404daf01f782d6aa not found: ID does not exist" containerID="db24f13217aef493d43b8c1f1ec86d4240915321ab6890ab404daf01f782d6aa" Nov 03 03:10:56.105490 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:56.105401 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db24f13217aef493d43b8c1f1ec86d4240915321ab6890ab404daf01f782d6aa"} err="failed to get container status \"db24f13217aef493d43b8c1f1ec86d4240915321ab6890ab404daf01f782d6aa\": rpc error: code = NotFound desc = could not find container \"db24f13217aef493d43b8c1f1ec86d4240915321ab6890ab404daf01f782d6aa\": container with ID starting with db24f13217aef493d43b8c1f1ec86d4240915321ab6890ab404daf01f782d6aa not found: ID does not exist" Nov 03 03:10:56.247212 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:10:56.247174 5605 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7232-2c2c15\" is forbidden: User \"system:node:ip-10-0-140-140.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-140.us-west-2.compute.internal' and this object" podUID="fbb90dbd-c248-4132-b71e-a9703cbf1efe" pod="opendatahub/test-curl-7232-2c2c15" Nov 03 03:10:56.849613 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:56.849576 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dfe0768-867a-4f0c-9d33-c9f0e2e947d6" path="/var/lib/kubelet/pods/4dfe0768-867a-4f0c-9d33-c9f0e2e947d6/volumes" Nov 03 03:10:56.849966 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:56.849953 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbb90dbd-c248-4132-b71e-a9703cbf1efe" path="/var/lib/kubelet/pods/fbb90dbd-c248-4132-b71e-a9703cbf1efe/volumes" Nov 03 03:10:57.242303 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:57.242274 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b83899d3-v4qdb" Nov 03 03:10:57.374985 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:57.374889 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a8641cb-d366-4df0-8ea4-c62b50b77428-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a8641cb-d366-4df0-8ea4-c62b50b77428-maas-api-openshift-ca\") pod \"2a8641cb-d366-4df0-8ea4-c62b50b77428\" (UID: \"2a8641cb-d366-4df0-8ea4-c62b50b77428\") " Nov 03 03:10:57.375180 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:57.375064 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a8641cb-d366-4df0-8ea4-c62b50b77428-kube-api-access-fk6x6\" (UniqueName: \"kubernetes.io/projected/2a8641cb-d366-4df0-8ea4-c62b50b77428-kube-api-access-fk6x6\") pod \"2a8641cb-d366-4df0-8ea4-c62b50b77428\" (UID: \"2a8641cb-d366-4df0-8ea4-c62b50b77428\") " Nov 03 03:10:57.375180 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:57.375096 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a8641cb-d366-4df0-8ea4-c62b50b77428-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a8641cb-d366-4df0-8ea4-c62b50b77428-maas-api-xks-ca\") pod \"2a8641cb-d366-4df0-8ea4-c62b50b77428\" (UID: \"2a8641cb-d366-4df0-8ea4-c62b50b77428\") " Nov 03 03:10:57.375371 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:57.375333 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a8641cb-d366-4df0-8ea4-c62b50b77428-maas-api-openshift-ca" pod "2a8641cb-d366-4df0-8ea4-c62b50b77428" (UID: "2a8641cb-d366-4df0-8ea4-c62b50b77428"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:10:57.375529 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:57.375511 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a8641cb-d366-4df0-8ea4-c62b50b77428-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:10:57.377213 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:57.377181 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a8641cb-d366-4df0-8ea4-c62b50b77428-maas-api-xks-ca" pod "2a8641cb-d366-4df0-8ea4-c62b50b77428" (UID: "2a8641cb-d366-4df0-8ea4-c62b50b77428"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:10:57.377336 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:57.377310 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a8641cb-d366-4df0-8ea4-c62b50b77428-kube-api-access-fk6x6" pod "2a8641cb-d366-4df0-8ea4-c62b50b77428" (UID: "2a8641cb-d366-4df0-8ea4-c62b50b77428"). InnerVolumeSpecName "kube-api-access-fk6x6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:10:57.476715 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:57.476673 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk6x6\" (UniqueName: \"kubernetes.io/projected/2a8641cb-d366-4df0-8ea4-c62b50b77428-kube-api-access-fk6x6\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:10:57.476715 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:57.476708 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a8641cb-d366-4df0-8ea4-c62b50b77428-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:10:58.108376 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:58.108324 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b83899d3-v4qdb" Nov 03 03:10:58.108575 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:58.108382 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b83899d3-v4qdb" event={"ID":"2a8641cb-d366-4df0-8ea4-c62b50b77428","Type":"ContainerDied","Data":"727a10b97a1f30cc91fdde7ecb1e53a0778a9c03f27834778493a30010e6e185"} Nov 03 03:10:58.108575 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:10:58.108429 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="727a10b97a1f30cc91fdde7ecb1e53a0778a9c03f27834778493a30010e6e185" Nov 03 03:11:04.315019 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:04.314975 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6c1354a-z88hg"] Nov 03 03:11:04.315627 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:04.315443 5605 cpu_manager.go:414] "containerMap: removing container" podUID="2a8641cb-d366-4df0-8ea4-c62b50b77428" containerName="revoke-keys" Nov 03 03:11:04.315627 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:04.315457 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a8641cb-d366-4df0-8ea4-c62b50b77428" containerName="revoke-keys" Nov 03 03:11:04.315627 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:04.315534 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a8641cb-d366-4df0-8ea4-c62b50b77428" containerName="revoke-keys" Nov 03 03:11:04.318793 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:04.318771 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c1354a-z88hg" Nov 03 03:11:04.374056 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:04.374005 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6c1354a-z88hg"] Nov 03 03:11:04.416835 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:04.416798 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:11:04.416835 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:04.416828 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:11:04.417064 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:04.416848 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:11:04.417064 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:04.416886 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 03:11:04.451237 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:04.451202 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69bjn\" (UniqueName: \"kubernetes.io/projected/2c62c4b3-d3df-4240-b233-5d38db1acee7-kube-api-access-69bjn\") pod \"maas-api-revoke-sub-b6c1354a-z88hg\" (UID: \"2c62c4b3-d3df-4240-b233-5d38db1acee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c1354a-z88hg" Nov 03 03:11:04.451475 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:04.451264 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c62c4b3-d3df-4240-b233-5d38db1acee7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6c1354a-z88hg\" (UID: \"2c62c4b3-d3df-4240-b233-5d38db1acee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c1354a-z88hg" Nov 03 03:11:04.451475 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:04.451351 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c62c4b3-d3df-4240-b233-5d38db1acee7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6c1354a-z88hg\" (UID: \"2c62c4b3-d3df-4240-b233-5d38db1acee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c1354a-z88hg" Nov 03 03:11:04.552452 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:04.552419 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69bjn\" (UniqueName: \"kubernetes.io/projected/2c62c4b3-d3df-4240-b233-5d38db1acee7-kube-api-access-69bjn\") pod \"maas-api-revoke-sub-b6c1354a-z88hg\" (UID: \"2c62c4b3-d3df-4240-b233-5d38db1acee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c1354a-z88hg" Nov 03 03:11:04.552648 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:04.552481 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c62c4b3-d3df-4240-b233-5d38db1acee7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6c1354a-z88hg\" (UID: \"2c62c4b3-d3df-4240-b233-5d38db1acee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c1354a-z88hg" Nov 03 03:11:04.552714 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:04.552652 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c62c4b3-d3df-4240-b233-5d38db1acee7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6c1354a-z88hg\" (UID: \"2c62c4b3-d3df-4240-b233-5d38db1acee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c1354a-z88hg" Nov 03 03:11:04.553283 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:04.553255 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c62c4b3-d3df-4240-b233-5d38db1acee7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6c1354a-z88hg\" (UID: \"2c62c4b3-d3df-4240-b233-5d38db1acee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c1354a-z88hg" Nov 03 03:11:04.554916 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:04.554892 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c62c4b3-d3df-4240-b233-5d38db1acee7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6c1354a-z88hg\" (UID: \"2c62c4b3-d3df-4240-b233-5d38db1acee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c1354a-z88hg" Nov 03 03:11:04.605971 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:04.605886 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69bjn\" (UniqueName: \"kubernetes.io/projected/2c62c4b3-d3df-4240-b233-5d38db1acee7-kube-api-access-69bjn\") pod \"maas-api-revoke-sub-b6c1354a-z88hg\" (UID: \"2c62c4b3-d3df-4240-b233-5d38db1acee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c1354a-z88hg" Nov 03 03:11:04.632588 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:04.632555 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c1354a-z88hg" Nov 03 03:11:05.002062 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:05.001973 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6c1354a-z88hg"] Nov 03 03:11:05.008299 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:11:05.008270 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c62c4b3_d3df_4240_b233_5d38db1acee7.slice/crio-0429870f6fbdc8f5c99f11af74e953ee4a216e50c5c8253119a1269a2f3b980f WatchSource:0}: Error finding container 0429870f6fbdc8f5c99f11af74e953ee4a216e50c5c8253119a1269a2f3b980f: Status 404 returned error can't find the container with id 0429870f6fbdc8f5c99f11af74e953ee4a216e50c5c8253119a1269a2f3b980f Nov 03 03:11:05.149148 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:05.149108 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c1354a-z88hg" event={"ID":"2c62c4b3-d3df-4240-b233-5d38db1acee7","Type":"ContainerStarted","Data":"0429870f6fbdc8f5c99f11af74e953ee4a216e50c5c8253119a1269a2f3b980f"} Nov 03 03:11:06.043377 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:06.043322 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c61dc5c-pll6l"] Nov 03 03:11:06.090444 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:06.090332 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c61dc5c-pll6l"] Nov 03 03:11:06.155512 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:06.155476 5605 generic.go:356] "Generic (PLEG): container finished" podID="2c62c4b3-d3df-4240-b233-5d38db1acee7" containerID="02f57d300f11b69c33fcd1598158c2287407cf9771a4c4289cdfb836488fe2c1" exitCode=0 Nov 03 03:11:06.155707 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:06.155559 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c1354a-z88hg" event={"ID":"2c62c4b3-d3df-4240-b233-5d38db1acee7","Type":"ContainerDied","Data":"02f57d300f11b69c33fcd1598158c2287407cf9771a4c4289cdfb836488fe2c1"} Nov 03 03:11:06.849701 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:06.849666 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99b91a2c-93fc-4fad-a448-bf5677d892f3" path="/var/lib/kubelet/pods/99b91a2c-93fc-4fad-a448-bf5677d892f3/volumes" Nov 03 03:11:07.312800 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:07.312776 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c1354a-z88hg" Nov 03 03:11:07.480974 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:07.480881 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c62c4b3-d3df-4240-b233-5d38db1acee7-kube-api-access-69bjn\" (UniqueName: \"kubernetes.io/projected/2c62c4b3-d3df-4240-b233-5d38db1acee7-kube-api-access-69bjn\") pod \"2c62c4b3-d3df-4240-b233-5d38db1acee7\" (UID: \"2c62c4b3-d3df-4240-b233-5d38db1acee7\") " Nov 03 03:11:07.481142 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:07.481040 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c62c4b3-d3df-4240-b233-5d38db1acee7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c62c4b3-d3df-4240-b233-5d38db1acee7-maas-api-xks-ca\") pod \"2c62c4b3-d3df-4240-b233-5d38db1acee7\" (UID: \"2c62c4b3-d3df-4240-b233-5d38db1acee7\") " Nov 03 03:11:07.481142 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:07.481080 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c62c4b3-d3df-4240-b233-5d38db1acee7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c62c4b3-d3df-4240-b233-5d38db1acee7-maas-api-openshift-ca\") pod \"2c62c4b3-d3df-4240-b233-5d38db1acee7\" (UID: \"2c62c4b3-d3df-4240-b233-5d38db1acee7\") " Nov 03 03:11:07.481542 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:07.481512 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c62c4b3-d3df-4240-b233-5d38db1acee7-maas-api-openshift-ca" pod "2c62c4b3-d3df-4240-b233-5d38db1acee7" (UID: "2c62c4b3-d3df-4240-b233-5d38db1acee7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:11:07.483011 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:07.482993 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c62c4b3-d3df-4240-b233-5d38db1acee7-kube-api-access-69bjn" pod "2c62c4b3-d3df-4240-b233-5d38db1acee7" (UID: "2c62c4b3-d3df-4240-b233-5d38db1acee7"). InnerVolumeSpecName "kube-api-access-69bjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:11:07.483110 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:07.483077 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c62c4b3-d3df-4240-b233-5d38db1acee7-maas-api-xks-ca" pod "2c62c4b3-d3df-4240-b233-5d38db1acee7" (UID: "2c62c4b3-d3df-4240-b233-5d38db1acee7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:11:07.582556 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:07.582512 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c62c4b3-d3df-4240-b233-5d38db1acee7-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:11:07.582556 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:07.582548 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c62c4b3-d3df-4240-b233-5d38db1acee7-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:11:07.582556 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:07.582562 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69bjn\" (UniqueName: \"kubernetes.io/projected/2c62c4b3-d3df-4240-b233-5d38db1acee7-kube-api-access-69bjn\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:11:08.167249 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:08.167218 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c1354a-z88hg" Nov 03 03:11:08.167465 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:08.167251 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c1354a-z88hg" event={"ID":"2c62c4b3-d3df-4240-b233-5d38db1acee7","Type":"ContainerDied","Data":"0429870f6fbdc8f5c99f11af74e953ee4a216e50c5c8253119a1269a2f3b980f"} Nov 03 03:11:08.167465 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:08.167296 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0429870f6fbdc8f5c99f11af74e953ee4a216e50c5c8253119a1269a2f3b980f" Nov 03 03:11:14.123943 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:14.123904 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:11:15.926972 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:15.926933 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3787f5e3-vd7cs"] Nov 03 03:11:15.927566 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:15.927545 5605 cpu_manager.go:414] "containerMap: removing container" podUID="2c62c4b3-d3df-4240-b233-5d38db1acee7" containerName="revoke-keys" Nov 03 03:11:15.927670 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:15.927568 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c62c4b3-d3df-4240-b233-5d38db1acee7" containerName="revoke-keys" Nov 03 03:11:15.927670 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:15.927659 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c62c4b3-d3df-4240-b233-5d38db1acee7" containerName="revoke-keys" Nov 03 03:11:15.930245 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:15.930223 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3787f5e3-vd7cs" Nov 03 03:11:15.974910 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:15.974876 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02c94438-cf20-439b-bab7-42c2bf20c4a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3787f5e3-vd7cs\" (UID: \"02c94438-cf20-439b-bab7-42c2bf20c4a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3787f5e3-vd7cs" Nov 03 03:11:15.975039 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:15.974919 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02c94438-cf20-439b-bab7-42c2bf20c4a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3787f5e3-vd7cs\" (UID: \"02c94438-cf20-439b-bab7-42c2bf20c4a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3787f5e3-vd7cs" Nov 03 03:11:15.975084 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:15.975041 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9bsv\" (UniqueName: \"kubernetes.io/projected/02c94438-cf20-439b-bab7-42c2bf20c4a4-kube-api-access-p9bsv\") pod \"maas-api-revoke-sub-3787f5e3-vd7cs\" (UID: \"02c94438-cf20-439b-bab7-42c2bf20c4a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3787f5e3-vd7cs" Nov 03 03:11:15.979256 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:15.979236 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:11:15.979350 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:15.979274 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:11:15.979350 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:15.979294 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:11:16.026079 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:16.026043 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 03:11:16.027898 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:16.027877 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3787f5e3-vd7cs"] Nov 03 03:11:16.075755 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:16.075726 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02c94438-cf20-439b-bab7-42c2bf20c4a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3787f5e3-vd7cs\" (UID: \"02c94438-cf20-439b-bab7-42c2bf20c4a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3787f5e3-vd7cs" Nov 03 03:11:16.075924 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:16.075765 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02c94438-cf20-439b-bab7-42c2bf20c4a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3787f5e3-vd7cs\" (UID: \"02c94438-cf20-439b-bab7-42c2bf20c4a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3787f5e3-vd7cs" Nov 03 03:11:16.075924 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:16.075796 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9bsv\" (UniqueName: \"kubernetes.io/projected/02c94438-cf20-439b-bab7-42c2bf20c4a4-kube-api-access-p9bsv\") pod \"maas-api-revoke-sub-3787f5e3-vd7cs\" (UID: \"02c94438-cf20-439b-bab7-42c2bf20c4a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3787f5e3-vd7cs" Nov 03 03:11:16.076603 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:16.076575 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02c94438-cf20-439b-bab7-42c2bf20c4a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3787f5e3-vd7cs\" (UID: \"02c94438-cf20-439b-bab7-42c2bf20c4a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3787f5e3-vd7cs" Nov 03 03:11:16.078170 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:16.078148 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02c94438-cf20-439b-bab7-42c2bf20c4a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3787f5e3-vd7cs\" (UID: \"02c94438-cf20-439b-bab7-42c2bf20c4a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3787f5e3-vd7cs" Nov 03 03:11:16.129452 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:16.129416 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9bsv\" (UniqueName: \"kubernetes.io/projected/02c94438-cf20-439b-bab7-42c2bf20c4a4-kube-api-access-p9bsv\") pod \"maas-api-revoke-sub-3787f5e3-vd7cs\" (UID: \"02c94438-cf20-439b-bab7-42c2bf20c4a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3787f5e3-vd7cs" Nov 03 03:11:16.243403 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:16.243296 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3787f5e3-vd7cs" Nov 03 03:11:16.358866 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:16.358833 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-9aebe8-9cb69fc69-plhsd"] Nov 03 03:11:16.464556 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:16.464517 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9aebe8-66bd979655-9t5kx"] Nov 03 03:11:16.612325 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:16.612288 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3787f5e3-vd7cs"] Nov 03 03:11:16.619072 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:11:16.619045 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02c94438_cf20_439b_bab7_42c2bf20c4a4.slice/crio-4adbc64a6a90ce93f0278dfd9546d0892a771b3cbf26a97c263351169b5ecca2 WatchSource:0}: Error finding container 4adbc64a6a90ce93f0278dfd9546d0892a771b3cbf26a97c263351169b5ecca2: Status 404 returned error can't find the container with id 4adbc64a6a90ce93f0278dfd9546d0892a771b3cbf26a97c263351169b5ecca2 Nov 03 03:11:17.215700 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:17.215612 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3787f5e3-vd7cs" event={"ID":"02c94438-cf20-439b-bab7-42c2bf20c4a4","Type":"ContainerStarted","Data":"3d4d906f825118ddbcf563b2edc336199c6eeebf3d5f82d22adf89e31d8ae4f7"} Nov 03 03:11:17.215700 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:17.215668 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3787f5e3-vd7cs" event={"ID":"02c94438-cf20-439b-bab7-42c2bf20c4a4","Type":"ContainerStarted","Data":"4adbc64a6a90ce93f0278dfd9546d0892a771b3cbf26a97c263351169b5ecca2"} Nov 03 03:11:17.325141 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:17.325084 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3787f5e3-vd7cs" podStartSLOduration=2.325068404 podStartE2EDuration="2.325068404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:11:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:11:17.323072374 +0000 UTC m=+4754.582270527" watchObservedRunningTime="2026-11-03 03:11:17.325068404 +0000 UTC m=+4754.584266557" Nov 03 03:11:18.222222 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:18.222189 5605 generic.go:356] "Generic (PLEG): container finished" podID="02c94438-cf20-439b-bab7-42c2bf20c4a4" containerID="3d4d906f825118ddbcf563b2edc336199c6eeebf3d5f82d22adf89e31d8ae4f7" exitCode=0 Nov 03 03:11:18.222660 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:18.222230 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3787f5e3-vd7cs" event={"ID":"02c94438-cf20-439b-bab7-42c2bf20c4a4","Type":"ContainerDied","Data":"3d4d906f825118ddbcf563b2edc336199c6eeebf3d5f82d22adf89e31d8ae4f7"} Nov 03 03:11:19.358287 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:19.358261 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3787f5e3-vd7cs" Nov 03 03:11:19.409476 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:19.409451 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02c94438-cf20-439b-bab7-42c2bf20c4a4-kube-api-access-p9bsv\" (UniqueName: \"kubernetes.io/projected/02c94438-cf20-439b-bab7-42c2bf20c4a4-kube-api-access-p9bsv\") pod \"02c94438-cf20-439b-bab7-42c2bf20c4a4\" (UID: \"02c94438-cf20-439b-bab7-42c2bf20c4a4\") " Nov 03 03:11:19.409589 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:19.409546 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02c94438-cf20-439b-bab7-42c2bf20c4a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02c94438-cf20-439b-bab7-42c2bf20c4a4-maas-api-openshift-ca\") pod \"02c94438-cf20-439b-bab7-42c2bf20c4a4\" (UID: \"02c94438-cf20-439b-bab7-42c2bf20c4a4\") " Nov 03 03:11:19.409589 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:19.409575 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02c94438-cf20-439b-bab7-42c2bf20c4a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02c94438-cf20-439b-bab7-42c2bf20c4a4-maas-api-xks-ca\") pod \"02c94438-cf20-439b-bab7-42c2bf20c4a4\" (UID: \"02c94438-cf20-439b-bab7-42c2bf20c4a4\") " Nov 03 03:11:19.409928 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:19.409889 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02c94438-cf20-439b-bab7-42c2bf20c4a4-maas-api-openshift-ca" pod "02c94438-cf20-439b-bab7-42c2bf20c4a4" (UID: "02c94438-cf20-439b-bab7-42c2bf20c4a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:11:19.411411 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:19.411390 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02c94438-cf20-439b-bab7-42c2bf20c4a4-kube-api-access-p9bsv" pod "02c94438-cf20-439b-bab7-42c2bf20c4a4" (UID: "02c94438-cf20-439b-bab7-42c2bf20c4a4"). InnerVolumeSpecName "kube-api-access-p9bsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:11:19.411529 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:19.411509 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02c94438-cf20-439b-bab7-42c2bf20c4a4-maas-api-xks-ca" pod "02c94438-cf20-439b-bab7-42c2bf20c4a4" (UID: "02c94438-cf20-439b-bab7-42c2bf20c4a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:11:19.510850 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:19.510780 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02c94438-cf20-439b-bab7-42c2bf20c4a4-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:11:19.510850 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:19.510809 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02c94438-cf20-439b-bab7-42c2bf20c4a4-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:11:19.510850 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:19.510822 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9bsv\" (UniqueName: \"kubernetes.io/projected/02c94438-cf20-439b-bab7-42c2bf20c4a4-kube-api-access-p9bsv\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:11:20.233802 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:20.233768 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3787f5e3-vd7cs" Nov 03 03:11:20.234000 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:20.233801 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3787f5e3-vd7cs" event={"ID":"02c94438-cf20-439b-bab7-42c2bf20c4a4","Type":"ContainerDied","Data":"4adbc64a6a90ce93f0278dfd9546d0892a771b3cbf26a97c263351169b5ecca2"} Nov 03 03:11:20.234000 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:20.233849 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4adbc64a6a90ce93f0278dfd9546d0892a771b3cbf26a97c263351169b5ecca2" Nov 03 03:11:21.359708 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:21.359647 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-9aebe8-9cb69fc69-plhsd" podUID="4f4d6e34-c9a7-491e-a4d7-6bf5d24b28fc" containerName="payload-processing" containerID="cri-o://228a5bc5ec03eba955e0bc916a4d0ddc44bebca1cf5c6d4917a5851f557a16c7" gracePeriod=25 Nov 03 03:11:21.465078 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:21.464993 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9aebe8-66bd979655-9t5kx" podUID="00fd0034-3867-4d9e-8f8c-4dd3c0d6fb07" containerName="payload-pre-processing" containerID="cri-o://1df7ee1a916b995d175031814aaa01872c1e3ca8cbc2dc9a37dcb19c586e4f58" gracePeriod=25 Nov 03 03:11:23.915085 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:11:23.915041 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:11:25.143624 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:25.143576 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9aebe8-66bd979655-9t5kx" podUID="00fd0034-3867-4d9e-8f8c-4dd3c0d6fb07" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:11:26.149993 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:26.149951 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-9aebe8-9cb69fc69-plhsd" podUID="4f4d6e34-c9a7-491e-a4d7-6bf5d24b28fc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:11:26.794871 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:26.794827 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb024034-n74dx"] Nov 03 03:11:26.795273 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:26.795258 5605 cpu_manager.go:414] "containerMap: removing container" podUID="02c94438-cf20-439b-bab7-42c2bf20c4a4" containerName="revoke-keys" Nov 03 03:11:26.795321 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:26.795275 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02c94438-cf20-439b-bab7-42c2bf20c4a4" containerName="revoke-keys" Nov 03 03:11:26.795372 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:26.795365 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="02c94438-cf20-439b-bab7-42c2bf20c4a4" containerName="revoke-keys" Nov 03 03:11:26.798569 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:26.798546 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb024034-n74dx" Nov 03 03:11:26.857625 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:26.857591 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:11:26.886847 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:26.886806 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/772d8930-c408-44dc-93ce-6f552334959c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb024034-n74dx\" (UID: \"772d8930-c408-44dc-93ce-6f552334959c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb024034-n74dx" Nov 03 03:11:26.886847 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:26.886852 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/772d8930-c408-44dc-93ce-6f552334959c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb024034-n74dx\" (UID: \"772d8930-c408-44dc-93ce-6f552334959c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb024034-n74dx" Nov 03 03:11:26.887119 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:26.887088 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb72q\" (UniqueName: \"kubernetes.io/projected/772d8930-c408-44dc-93ce-6f552334959c-kube-api-access-tb72q\") pod \"maas-api-revoke-sub-cb024034-n74dx\" (UID: \"772d8930-c408-44dc-93ce-6f552334959c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb024034-n74dx" Nov 03 03:11:26.903939 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:26.903902 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:11:26.904117 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:26.903954 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:11:26.904117 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:26.904014 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 03:11:26.906192 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:26.906166 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb024034-n74dx"] Nov 03 03:11:26.987727 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:26.987688 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/772d8930-c408-44dc-93ce-6f552334959c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb024034-n74dx\" (UID: \"772d8930-c408-44dc-93ce-6f552334959c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb024034-n74dx" Nov 03 03:11:26.987727 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:26.987729 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/772d8930-c408-44dc-93ce-6f552334959c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb024034-n74dx\" (UID: \"772d8930-c408-44dc-93ce-6f552334959c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb024034-n74dx" Nov 03 03:11:26.988094 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:26.987824 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb72q\" (UniqueName: \"kubernetes.io/projected/772d8930-c408-44dc-93ce-6f552334959c-kube-api-access-tb72q\") pod \"maas-api-revoke-sub-cb024034-n74dx\" (UID: \"772d8930-c408-44dc-93ce-6f552334959c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb024034-n74dx" Nov 03 03:11:26.988555 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:26.988528 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/772d8930-c408-44dc-93ce-6f552334959c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb024034-n74dx\" (UID: \"772d8930-c408-44dc-93ce-6f552334959c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb024034-n74dx" Nov 03 03:11:26.990266 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:26.990242 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/772d8930-c408-44dc-93ce-6f552334959c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb024034-n74dx\" (UID: \"772d8930-c408-44dc-93ce-6f552334959c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb024034-n74dx" Nov 03 03:11:27.045281 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:27.045205 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb72q\" (UniqueName: \"kubernetes.io/projected/772d8930-c408-44dc-93ce-6f552334959c-kube-api-access-tb72q\") pod \"maas-api-revoke-sub-cb024034-n74dx\" (UID: \"772d8930-c408-44dc-93ce-6f552334959c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb024034-n74dx" Nov 03 03:11:27.114043 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:27.114004 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb024034-n74dx" Nov 03 03:11:27.485306 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:27.485265 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb024034-n74dx"] Nov 03 03:11:27.490971 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:11:27.490925 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod772d8930_c408_44dc_93ce_6f552334959c.slice/crio-21b2d655135bc31e2fb1ec1edaa9d03291f4e59d32cefa826b9197a6a449bdbc WatchSource:0}: Error finding container 21b2d655135bc31e2fb1ec1edaa9d03291f4e59d32cefa826b9197a6a449bdbc: Status 404 returned error can't find the container with id 21b2d655135bc31e2fb1ec1edaa9d03291f4e59d32cefa826b9197a6a449bdbc Nov 03 03:11:28.283965 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:28.283921 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb024034-n74dx" event={"ID":"772d8930-c408-44dc-93ce-6f552334959c","Type":"ContainerStarted","Data":"02542ff862cba3a697859422870697aeb4b7a8af4efd62964bc776b1e1bf0f50"} Nov 03 03:11:28.284202 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:28.283999 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb024034-n74dx" event={"ID":"772d8930-c408-44dc-93ce-6f552334959c","Type":"ContainerStarted","Data":"21b2d655135bc31e2fb1ec1edaa9d03291f4e59d32cefa826b9197a6a449bdbc"} Nov 03 03:11:28.390683 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:28.390624 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb024034-n74dx" podStartSLOduration=2.390607415 podStartE2EDuration="2.390607415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:11:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:11:28.388865757 +0000 UTC m=+4765.648063922" watchObservedRunningTime="2026-11-03 03:11:28.390607415 +0000 UTC m=+4765.649805601" Nov 03 03:11:29.291019 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:29.290980 5605 generic.go:356] "Generic (PLEG): container finished" podID="772d8930-c408-44dc-93ce-6f552334959c" containerID="02542ff862cba3a697859422870697aeb4b7a8af4efd62964bc776b1e1bf0f50" exitCode=0 Nov 03 03:11:29.291483 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:29.291071 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb024034-n74dx" event={"ID":"772d8930-c408-44dc-93ce-6f552334959c","Type":"ContainerDied","Data":"02542ff862cba3a697859422870697aeb4b7a8af4efd62964bc776b1e1bf0f50"} Nov 03 03:11:30.427529 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:30.427498 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb024034-n74dx" Nov 03 03:11:30.528775 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:30.528739 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/772d8930-c408-44dc-93ce-6f552334959c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/772d8930-c408-44dc-93ce-6f552334959c-maas-api-openshift-ca\") pod \"772d8930-c408-44dc-93ce-6f552334959c\" (UID: \"772d8930-c408-44dc-93ce-6f552334959c\") " Nov 03 03:11:30.528991 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:30.528865 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/772d8930-c408-44dc-93ce-6f552334959c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/772d8930-c408-44dc-93ce-6f552334959c-maas-api-xks-ca\") pod \"772d8930-c408-44dc-93ce-6f552334959c\" (UID: \"772d8930-c408-44dc-93ce-6f552334959c\") " Nov 03 03:11:30.528991 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:30.528902 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/772d8930-c408-44dc-93ce-6f552334959c-kube-api-access-tb72q\" (UniqueName: \"kubernetes.io/projected/772d8930-c408-44dc-93ce-6f552334959c-kube-api-access-tb72q\") pod \"772d8930-c408-44dc-93ce-6f552334959c\" (UID: \"772d8930-c408-44dc-93ce-6f552334959c\") " Nov 03 03:11:30.529108 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:30.529090 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/772d8930-c408-44dc-93ce-6f552334959c-maas-api-openshift-ca" pod "772d8930-c408-44dc-93ce-6f552334959c" (UID: "772d8930-c408-44dc-93ce-6f552334959c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:11:30.529275 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:30.529259 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/772d8930-c408-44dc-93ce-6f552334959c-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:11:30.531019 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:30.530989 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/772d8930-c408-44dc-93ce-6f552334959c-maas-api-xks-ca" pod "772d8930-c408-44dc-93ce-6f552334959c" (UID: "772d8930-c408-44dc-93ce-6f552334959c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:11:30.531019 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:30.530994 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/772d8930-c408-44dc-93ce-6f552334959c-kube-api-access-tb72q" pod "772d8930-c408-44dc-93ce-6f552334959c" (UID: "772d8930-c408-44dc-93ce-6f552334959c"). InnerVolumeSpecName "kube-api-access-tb72q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:11:30.630182 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:30.630155 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/772d8930-c408-44dc-93ce-6f552334959c-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:11:30.630182 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:30.630180 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tb72q\" (UniqueName: \"kubernetes.io/projected/772d8930-c408-44dc-93ce-6f552334959c-kube-api-access-tb72q\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:11:31.303711 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:31.303680 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb024034-n74dx" Nov 03 03:11:31.303919 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:31.303680 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb024034-n74dx" event={"ID":"772d8930-c408-44dc-93ce-6f552334959c","Type":"ContainerDied","Data":"21b2d655135bc31e2fb1ec1edaa9d03291f4e59d32cefa826b9197a6a449bdbc"} Nov 03 03:11:31.303919 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:31.303807 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21b2d655135bc31e2fb1ec1edaa9d03291f4e59d32cefa826b9197a6a449bdbc" Nov 03 03:11:33.700508 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:33.700475 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-110ef79d-t8x2x"] Nov 03 03:11:33.700904 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:33.700891 5605 cpu_manager.go:414] "containerMap: removing container" podUID="772d8930-c408-44dc-93ce-6f552334959c" containerName="revoke-keys" Nov 03 03:11:33.700947 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:33.700905 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="772d8930-c408-44dc-93ce-6f552334959c" containerName="revoke-keys" Nov 03 03:11:33.700989 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:33.700971 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="772d8930-c408-44dc-93ce-6f552334959c" containerName="revoke-keys" Nov 03 03:11:33.704254 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:33.704233 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-110ef79d-t8x2x" Nov 03 03:11:33.761750 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:33.761719 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f0dd36b-6507-4c53-92f9-b0f8d6f0abee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-110ef79d-t8x2x\" (UID: \"8f0dd36b-6507-4c53-92f9-b0f8d6f0abee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-110ef79d-t8x2x" Nov 03 03:11:33.761917 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:33.761862 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f0dd36b-6507-4c53-92f9-b0f8d6f0abee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-110ef79d-t8x2x\" (UID: \"8f0dd36b-6507-4c53-92f9-b0f8d6f0abee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-110ef79d-t8x2x" Nov 03 03:11:33.761987 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:33.761916 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8czv7\" (UniqueName: \"kubernetes.io/projected/8f0dd36b-6507-4c53-92f9-b0f8d6f0abee-kube-api-access-8czv7\") pod \"maas-api-revoke-sub-110ef79d-t8x2x\" (UID: \"8f0dd36b-6507-4c53-92f9-b0f8d6f0abee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-110ef79d-t8x2x" Nov 03 03:11:33.803240 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:33.803205 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:11:33.803240 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:33.803228 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:11:33.803511 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:33.803256 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:11:33.803511 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:33.803265 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 03:11:33.805469 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:33.805442 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-110ef79d-t8x2x"] Nov 03 03:11:33.863429 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:33.863404 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f0dd36b-6507-4c53-92f9-b0f8d6f0abee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-110ef79d-t8x2x\" (UID: \"8f0dd36b-6507-4c53-92f9-b0f8d6f0abee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-110ef79d-t8x2x" Nov 03 03:11:33.863556 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:33.863445 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8czv7\" (UniqueName: \"kubernetes.io/projected/8f0dd36b-6507-4c53-92f9-b0f8d6f0abee-kube-api-access-8czv7\") pod \"maas-api-revoke-sub-110ef79d-t8x2x\" (UID: \"8f0dd36b-6507-4c53-92f9-b0f8d6f0abee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-110ef79d-t8x2x" Nov 03 03:11:33.863556 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:33.863513 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f0dd36b-6507-4c53-92f9-b0f8d6f0abee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-110ef79d-t8x2x\" (UID: \"8f0dd36b-6507-4c53-92f9-b0f8d6f0abee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-110ef79d-t8x2x" Nov 03 03:11:33.864077 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:33.864059 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f0dd36b-6507-4c53-92f9-b0f8d6f0abee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-110ef79d-t8x2x\" (UID: \"8f0dd36b-6507-4c53-92f9-b0f8d6f0abee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-110ef79d-t8x2x" Nov 03 03:11:33.865881 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:33.865857 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f0dd36b-6507-4c53-92f9-b0f8d6f0abee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-110ef79d-t8x2x\" (UID: \"8f0dd36b-6507-4c53-92f9-b0f8d6f0abee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-110ef79d-t8x2x" Nov 03 03:11:33.917798 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:33.917770 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8czv7\" (UniqueName: \"kubernetes.io/projected/8f0dd36b-6507-4c53-92f9-b0f8d6f0abee-kube-api-access-8czv7\") pod \"maas-api-revoke-sub-110ef79d-t8x2x\" (UID: \"8f0dd36b-6507-4c53-92f9-b0f8d6f0abee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-110ef79d-t8x2x" Nov 03 03:11:34.016912 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:34.016844 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-110ef79d-t8x2x" Nov 03 03:11:34.385289 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:34.385253 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-110ef79d-t8x2x"] Nov 03 03:11:34.390877 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:11:34.390848 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f0dd36b_6507_4c53_92f9_b0f8d6f0abee.slice/crio-5434884896d7622dfd841664f04a357ce0cb84168619298d72d5a4e2e8a91f94 WatchSource:0}: Error finding container 5434884896d7622dfd841664f04a357ce0cb84168619298d72d5a4e2e8a91f94: Status 404 returned error can't find the container with id 5434884896d7622dfd841664f04a357ce0cb84168619298d72d5a4e2e8a91f94 Nov 03 03:11:35.144040 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:35.144005 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9aebe8-66bd979655-9t5kx" podUID="00fd0034-3867-4d9e-8f8c-4dd3c0d6fb07" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:11:35.326788 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:35.326748 5605 generic.go:356] "Generic (PLEG): container finished" podID="8f0dd36b-6507-4c53-92f9-b0f8d6f0abee" containerID="99d966d108a92cf368e79254ba7358cce6a52cd5b7cec7ac5f9e534c887f7b8d" exitCode=0 Nov 03 03:11:35.326973 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:35.326827 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-110ef79d-t8x2x" event={"ID":"8f0dd36b-6507-4c53-92f9-b0f8d6f0abee","Type":"ContainerDied","Data":"99d966d108a92cf368e79254ba7358cce6a52cd5b7cec7ac5f9e534c887f7b8d"} Nov 03 03:11:35.326973 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:35.326880 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-110ef79d-t8x2x" event={"ID":"8f0dd36b-6507-4c53-92f9-b0f8d6f0abee","Type":"ContainerStarted","Data":"5434884896d7622dfd841664f04a357ce0cb84168619298d72d5a4e2e8a91f94"} Nov 03 03:11:36.150149 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:36.150113 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-9aebe8-9cb69fc69-plhsd" podUID="4f4d6e34-c9a7-491e-a4d7-6bf5d24b28fc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:11:36.469326 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:36.469296 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-110ef79d-t8x2x" Nov 03 03:11:36.595899 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:36.595867 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f0dd36b-6507-4c53-92f9-b0f8d6f0abee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f0dd36b-6507-4c53-92f9-b0f8d6f0abee-maas-api-xks-ca\") pod \"8f0dd36b-6507-4c53-92f9-b0f8d6f0abee\" (UID: \"8f0dd36b-6507-4c53-92f9-b0f8d6f0abee\") " Nov 03 03:11:36.596070 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:36.595982 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f0dd36b-6507-4c53-92f9-b0f8d6f0abee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f0dd36b-6507-4c53-92f9-b0f8d6f0abee-maas-api-openshift-ca\") pod \"8f0dd36b-6507-4c53-92f9-b0f8d6f0abee\" (UID: \"8f0dd36b-6507-4c53-92f9-b0f8d6f0abee\") " Nov 03 03:11:36.596070 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:36.596036 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f0dd36b-6507-4c53-92f9-b0f8d6f0abee-kube-api-access-8czv7\" (UniqueName: \"kubernetes.io/projected/8f0dd36b-6507-4c53-92f9-b0f8d6f0abee-kube-api-access-8czv7\") pod \"8f0dd36b-6507-4c53-92f9-b0f8d6f0abee\" (UID: \"8f0dd36b-6507-4c53-92f9-b0f8d6f0abee\") " Nov 03 03:11:36.596442 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:36.596410 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f0dd36b-6507-4c53-92f9-b0f8d6f0abee-maas-api-openshift-ca" pod "8f0dd36b-6507-4c53-92f9-b0f8d6f0abee" (UID: "8f0dd36b-6507-4c53-92f9-b0f8d6f0abee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:11:36.598014 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:36.597984 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f0dd36b-6507-4c53-92f9-b0f8d6f0abee-maas-api-xks-ca" pod "8f0dd36b-6507-4c53-92f9-b0f8d6f0abee" (UID: "8f0dd36b-6507-4c53-92f9-b0f8d6f0abee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:11:36.598155 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:36.598087 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f0dd36b-6507-4c53-92f9-b0f8d6f0abee-kube-api-access-8czv7" pod "8f0dd36b-6507-4c53-92f9-b0f8d6f0abee" (UID: "8f0dd36b-6507-4c53-92f9-b0f8d6f0abee"). InnerVolumeSpecName "kube-api-access-8czv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:11:36.697818 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:36.697751 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f0dd36b-6507-4c53-92f9-b0f8d6f0abee-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:11:36.697818 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:36.697778 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f0dd36b-6507-4c53-92f9-b0f8d6f0abee-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:11:36.697818 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:36.697788 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8czv7\" (UniqueName: \"kubernetes.io/projected/8f0dd36b-6507-4c53-92f9-b0f8d6f0abee-kube-api-access-8czv7\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:11:37.338503 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:37.338466 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-110ef79d-t8x2x" event={"ID":"8f0dd36b-6507-4c53-92f9-b0f8d6f0abee","Type":"ContainerDied","Data":"5434884896d7622dfd841664f04a357ce0cb84168619298d72d5a4e2e8a91f94"} Nov 03 03:11:37.338959 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:37.338529 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5434884896d7622dfd841664f04a357ce0cb84168619298d72d5a4e2e8a91f94" Nov 03 03:11:37.338959 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:37.338483 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-110ef79d-t8x2x" Nov 03 03:11:38.764471 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:38.764440 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-3a4683/sub-test-model-3a4683-kserve-656f748dc5-nzhlm"] Nov 03 03:11:41.616579 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:41.616518 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-9aebe8-9cb69fc69-plhsd" Nov 03 03:11:41.710269 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:41.710244 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9aebe8-66bd979655-9t5kx" Nov 03 03:11:41.750070 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:41.750039 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f4d6e34-c9a7-491e-a4d7-6bf5d24b28fc-kube-api-access-vdrdf\" (UniqueName: \"kubernetes.io/projected/4f4d6e34-c9a7-491e-a4d7-6bf5d24b28fc-kube-api-access-vdrdf\") pod \"4f4d6e34-c9a7-491e-a4d7-6bf5d24b28fc\" (UID: \"4f4d6e34-c9a7-491e-a4d7-6bf5d24b28fc\") " Nov 03 03:11:41.750256 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:41.750128 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f4d6e34-c9a7-491e-a4d7-6bf5d24b28fc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f4d6e34-c9a7-491e-a4d7-6bf5d24b28fc-plugins-config-volume\") pod \"4f4d6e34-c9a7-491e-a4d7-6bf5d24b28fc\" (UID: \"4f4d6e34-c9a7-491e-a4d7-6bf5d24b28fc\") " Nov 03 03:11:41.750575 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:41.750551 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f4d6e34-c9a7-491e-a4d7-6bf5d24b28fc-plugins-config-volume" pod "4f4d6e34-c9a7-491e-a4d7-6bf5d24b28fc" (UID: "4f4d6e34-c9a7-491e-a4d7-6bf5d24b28fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:11:41.752123 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:41.752100 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f4d6e34-c9a7-491e-a4d7-6bf5d24b28fc-kube-api-access-vdrdf" pod "4f4d6e34-c9a7-491e-a4d7-6bf5d24b28fc" (UID: "4f4d6e34-c9a7-491e-a4d7-6bf5d24b28fc"). InnerVolumeSpecName "kube-api-access-vdrdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:11:41.851265 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:41.851226 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00fd0034-3867-4d9e-8f8c-4dd3c0d6fb07-kube-api-access-s8sm5\" (UniqueName: \"kubernetes.io/projected/00fd0034-3867-4d9e-8f8c-4dd3c0d6fb07-kube-api-access-s8sm5\") pod \"00fd0034-3867-4d9e-8f8c-4dd3c0d6fb07\" (UID: \"00fd0034-3867-4d9e-8f8c-4dd3c0d6fb07\") " Nov 03 03:11:41.851472 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:41.851444 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00fd0034-3867-4d9e-8f8c-4dd3c0d6fb07-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00fd0034-3867-4d9e-8f8c-4dd3c0d6fb07-plugins-config-volume\") pod \"00fd0034-3867-4d9e-8f8c-4dd3c0d6fb07\" (UID: \"00fd0034-3867-4d9e-8f8c-4dd3c0d6fb07\") " Nov 03 03:11:41.851689 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:41.851673 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdrdf\" (UniqueName: \"kubernetes.io/projected/4f4d6e34-c9a7-491e-a4d7-6bf5d24b28fc-kube-api-access-vdrdf\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:11:41.851765 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:41.851689 5605 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f4d6e34-c9a7-491e-a4d7-6bf5d24b28fc-plugins-config-volume\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:11:41.851825 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:41.851763 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00fd0034-3867-4d9e-8f8c-4dd3c0d6fb07-plugins-config-volume" pod "00fd0034-3867-4d9e-8f8c-4dd3c0d6fb07" (UID: "00fd0034-3867-4d9e-8f8c-4dd3c0d6fb07"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:11:41.853293 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:41.853269 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00fd0034-3867-4d9e-8f8c-4dd3c0d6fb07-kube-api-access-s8sm5" pod "00fd0034-3867-4d9e-8f8c-4dd3c0d6fb07" (UID: "00fd0034-3867-4d9e-8f8c-4dd3c0d6fb07"). InnerVolumeSpecName "kube-api-access-s8sm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:11:41.952595 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:41.952508 5605 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00fd0034-3867-4d9e-8f8c-4dd3c0d6fb07-plugins-config-volume\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:11:41.952595 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:41.952547 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8sm5\" (UniqueName: \"kubernetes.io/projected/00fd0034-3867-4d9e-8f8c-4dd3c0d6fb07-kube-api-access-s8sm5\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:11:42.259472 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:42.259388 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:11:42.366166 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:42.366136 5605 generic.go:356] "Generic (PLEG): container finished" podID="00fd0034-3867-4d9e-8f8c-4dd3c0d6fb07" containerID="1df7ee1a916b995d175031814aaa01872c1e3ca8cbc2dc9a37dcb19c586e4f58" exitCode=0 Nov 03 03:11:42.366342 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:42.366180 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9aebe8-66bd979655-9t5kx" Nov 03 03:11:42.366342 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:42.366222 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9aebe8-66bd979655-9t5kx" event={"ID":"00fd0034-3867-4d9e-8f8c-4dd3c0d6fb07","Type":"ContainerDied","Data":"1df7ee1a916b995d175031814aaa01872c1e3ca8cbc2dc9a37dcb19c586e4f58"} Nov 03 03:11:42.366342 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:42.366281 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9aebe8-66bd979655-9t5kx" event={"ID":"00fd0034-3867-4d9e-8f8c-4dd3c0d6fb07","Type":"ContainerDied","Data":"35f1054372d1fc1a52762f492d495705fddc9575c9e89c2cdbc9fa34e0315645"} Nov 03 03:11:42.366342 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:42.366313 5605 scope.go:122] "RemoveContainer" containerID="1df7ee1a916b995d175031814aaa01872c1e3ca8cbc2dc9a37dcb19c586e4f58" Nov 03 03:11:42.367700 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:42.367674 5605 generic.go:356] "Generic (PLEG): container finished" podID="4f4d6e34-c9a7-491e-a4d7-6bf5d24b28fc" containerID="228a5bc5ec03eba955e0bc916a4d0ddc44bebca1cf5c6d4917a5851f557a16c7" exitCode=0 Nov 03 03:11:42.367817 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:42.367729 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-9aebe8-9cb69fc69-plhsd" Nov 03 03:11:42.367817 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:42.367757 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-9aebe8-9cb69fc69-plhsd" event={"ID":"4f4d6e34-c9a7-491e-a4d7-6bf5d24b28fc","Type":"ContainerDied","Data":"228a5bc5ec03eba955e0bc916a4d0ddc44bebca1cf5c6d4917a5851f557a16c7"} Nov 03 03:11:42.367817 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:42.367798 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-9aebe8-9cb69fc69-plhsd" event={"ID":"4f4d6e34-c9a7-491e-a4d7-6bf5d24b28fc","Type":"ContainerDied","Data":"313d3d7a27549277d5f951aefb3f63cc965767fac194dc32b03192da78488ea1"} Nov 03 03:11:42.384563 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:42.384546 5605 scope.go:122] "RemoveContainer" containerID="1df7ee1a916b995d175031814aaa01872c1e3ca8cbc2dc9a37dcb19c586e4f58" Nov 03 03:11:42.384837 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:11:42.384819 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1df7ee1a916b995d175031814aaa01872c1e3ca8cbc2dc9a37dcb19c586e4f58\": container with ID starting with 1df7ee1a916b995d175031814aaa01872c1e3ca8cbc2dc9a37dcb19c586e4f58 not found: ID does not exist" containerID="1df7ee1a916b995d175031814aaa01872c1e3ca8cbc2dc9a37dcb19c586e4f58" Nov 03 03:11:42.384885 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:42.384845 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1df7ee1a916b995d175031814aaa01872c1e3ca8cbc2dc9a37dcb19c586e4f58"} err="failed to get container status \"1df7ee1a916b995d175031814aaa01872c1e3ca8cbc2dc9a37dcb19c586e4f58\": rpc error: code = NotFound desc = could not find container \"1df7ee1a916b995d175031814aaa01872c1e3ca8cbc2dc9a37dcb19c586e4f58\": container with ID starting with 1df7ee1a916b995d175031814aaa01872c1e3ca8cbc2dc9a37dcb19c586e4f58 not found: ID does not exist" Nov 03 03:11:42.384885 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:42.384863 5605 scope.go:122] "RemoveContainer" containerID="228a5bc5ec03eba955e0bc916a4d0ddc44bebca1cf5c6d4917a5851f557a16c7" Nov 03 03:11:42.392690 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:42.392672 5605 scope.go:122] "RemoveContainer" containerID="228a5bc5ec03eba955e0bc916a4d0ddc44bebca1cf5c6d4917a5851f557a16c7" Nov 03 03:11:42.392949 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:11:42.392930 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"228a5bc5ec03eba955e0bc916a4d0ddc44bebca1cf5c6d4917a5851f557a16c7\": container with ID starting with 228a5bc5ec03eba955e0bc916a4d0ddc44bebca1cf5c6d4917a5851f557a16c7 not found: ID does not exist" containerID="228a5bc5ec03eba955e0bc916a4d0ddc44bebca1cf5c6d4917a5851f557a16c7" Nov 03 03:11:42.393013 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:42.392958 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"228a5bc5ec03eba955e0bc916a4d0ddc44bebca1cf5c6d4917a5851f557a16c7"} err="failed to get container status \"228a5bc5ec03eba955e0bc916a4d0ddc44bebca1cf5c6d4917a5851f557a16c7\": rpc error: code = NotFound desc = could not find container \"228a5bc5ec03eba955e0bc916a4d0ddc44bebca1cf5c6d4917a5851f557a16c7\": container with ID starting with 228a5bc5ec03eba955e0bc916a4d0ddc44bebca1cf5c6d4917a5851f557a16c7 not found: ID does not exist" Nov 03 03:11:42.529050 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:42.521619 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9aebe8-66bd979655-9t5kx"] Nov 03 03:11:42.530767 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:42.530741 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9aebe8-66bd979655-9t5kx"] Nov 03 03:11:42.633523 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:42.633466 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-9aebe8-9cb69fc69-plhsd"] Nov 03 03:11:42.635601 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:42.635575 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-9aebe8-9cb69fc69-plhsd"] Nov 03 03:11:42.849840 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:42.849807 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00fd0034-3867-4d9e-8f8c-4dd3c0d6fb07" path="/var/lib/kubelet/pods/00fd0034-3867-4d9e-8f8c-4dd3c0d6fb07/volumes" Nov 03 03:11:42.850300 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:42.850280 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f4d6e34-c9a7-491e-a4d7-6bf5d24b28fc" path="/var/lib/kubelet/pods/4f4d6e34-c9a7-491e-a4d7-6bf5d24b28fc/volumes" Nov 03 03:11:44.460585 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:44.460550 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a497f943-cwt9b"] Nov 03 03:11:44.461019 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:44.460913 5605 cpu_manager.go:414] "containerMap: removing container" podUID="8f0dd36b-6507-4c53-92f9-b0f8d6f0abee" containerName="revoke-keys" Nov 03 03:11:44.461019 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:44.460924 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f0dd36b-6507-4c53-92f9-b0f8d6f0abee" containerName="revoke-keys" Nov 03 03:11:44.461019 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:44.460956 5605 cpu_manager.go:414] "containerMap: removing container" podUID="4f4d6e34-c9a7-491e-a4d7-6bf5d24b28fc" containerName="payload-processing" Nov 03 03:11:44.461019 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:44.460961 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f4d6e34-c9a7-491e-a4d7-6bf5d24b28fc" containerName="payload-processing" Nov 03 03:11:44.461019 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:44.460972 5605 cpu_manager.go:414] "containerMap: removing container" podUID="00fd0034-3867-4d9e-8f8c-4dd3c0d6fb07" containerName="payload-pre-processing" Nov 03 03:11:44.461019 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:44.460978 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00fd0034-3867-4d9e-8f8c-4dd3c0d6fb07" containerName="payload-pre-processing" Nov 03 03:11:44.461251 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:44.461039 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f4d6e34-c9a7-491e-a4d7-6bf5d24b28fc" containerName="payload-processing" Nov 03 03:11:44.461251 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:44.461048 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="00fd0034-3867-4d9e-8f8c-4dd3c0d6fb07" containerName="payload-pre-processing" Nov 03 03:11:44.461251 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:44.461056 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f0dd36b-6507-4c53-92f9-b0f8d6f0abee" containerName="revoke-keys" Nov 03 03:11:44.465491 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:44.465470 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a497f943-cwt9b" Nov 03 03:11:44.515404 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:44.515352 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:11:44.561936 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:44.561905 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:11:44.561936 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:44.561929 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:11:44.562150 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:44.562014 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 03:11:44.563999 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:44.563976 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a497f943-cwt9b"] Nov 03 03:11:44.579479 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:44.579455 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1019142d-81ac-42e4-bd92-55e29ae2051a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a497f943-cwt9b\" (UID: \"1019142d-81ac-42e4-bd92-55e29ae2051a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a497f943-cwt9b" Nov 03 03:11:44.579581 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:44.579494 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1019142d-81ac-42e4-bd92-55e29ae2051a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a497f943-cwt9b\" (UID: \"1019142d-81ac-42e4-bd92-55e29ae2051a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a497f943-cwt9b" Nov 03 03:11:44.579664 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:44.579643 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxvk4\" (UniqueName: \"kubernetes.io/projected/1019142d-81ac-42e4-bd92-55e29ae2051a-kube-api-access-pxvk4\") pod \"maas-api-revoke-sub-a497f943-cwt9b\" (UID: \"1019142d-81ac-42e4-bd92-55e29ae2051a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a497f943-cwt9b" Nov 03 03:11:44.680751 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:44.680724 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1019142d-81ac-42e4-bd92-55e29ae2051a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a497f943-cwt9b\" (UID: \"1019142d-81ac-42e4-bd92-55e29ae2051a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a497f943-cwt9b" Nov 03 03:11:44.680889 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:44.680764 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1019142d-81ac-42e4-bd92-55e29ae2051a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a497f943-cwt9b\" (UID: \"1019142d-81ac-42e4-bd92-55e29ae2051a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a497f943-cwt9b" Nov 03 03:11:44.680889 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:44.680807 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxvk4\" (UniqueName: \"kubernetes.io/projected/1019142d-81ac-42e4-bd92-55e29ae2051a-kube-api-access-pxvk4\") pod \"maas-api-revoke-sub-a497f943-cwt9b\" (UID: \"1019142d-81ac-42e4-bd92-55e29ae2051a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a497f943-cwt9b" Nov 03 03:11:44.681374 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:44.681342 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1019142d-81ac-42e4-bd92-55e29ae2051a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a497f943-cwt9b\" (UID: \"1019142d-81ac-42e4-bd92-55e29ae2051a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a497f943-cwt9b" Nov 03 03:11:44.683151 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:44.683130 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1019142d-81ac-42e4-bd92-55e29ae2051a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a497f943-cwt9b\" (UID: \"1019142d-81ac-42e4-bd92-55e29ae2051a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a497f943-cwt9b" Nov 03 03:11:44.734185 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:44.734122 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxvk4\" (UniqueName: \"kubernetes.io/projected/1019142d-81ac-42e4-bd92-55e29ae2051a-kube-api-access-pxvk4\") pod \"maas-api-revoke-sub-a497f943-cwt9b\" (UID: \"1019142d-81ac-42e4-bd92-55e29ae2051a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a497f943-cwt9b" Nov 03 03:11:44.779700 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:44.779669 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a497f943-cwt9b" Nov 03 03:11:45.146527 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:45.146494 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a497f943-cwt9b"] Nov 03 03:11:45.152871 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:11:45.152840 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1019142d_81ac_42e4_bd92_55e29ae2051a.slice/crio-242e2d757a46ca427bf76074fd4d08d54dcf04e9768a1561ac1b77a6df6634ab WatchSource:0}: Error finding container 242e2d757a46ca427bf76074fd4d08d54dcf04e9768a1561ac1b77a6df6634ab: Status 404 returned error can't find the container with id 242e2d757a46ca427bf76074fd4d08d54dcf04e9768a1561ac1b77a6df6634ab Nov 03 03:11:45.386786 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:45.386750 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a497f943-cwt9b" event={"ID":"1019142d-81ac-42e4-bd92-55e29ae2051a","Type":"ContainerStarted","Data":"242e2d757a46ca427bf76074fd4d08d54dcf04e9768a1561ac1b77a6df6634ab"} Nov 03 03:11:46.393796 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:46.393765 5605 generic.go:356] "Generic (PLEG): container finished" podID="1019142d-81ac-42e4-bd92-55e29ae2051a" containerID="e5586df9e1e3691598fa8898dad032b83f5fd113d5cb79a8c385aef364f17eb4" exitCode=0 Nov 03 03:11:46.394172 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:46.393816 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a497f943-cwt9b" event={"ID":"1019142d-81ac-42e4-bd92-55e29ae2051a","Type":"ContainerDied","Data":"e5586df9e1e3691598fa8898dad032b83f5fd113d5cb79a8c385aef364f17eb4"} Nov 03 03:11:47.545212 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:47.545186 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a497f943-cwt9b" Nov 03 03:11:47.708914 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:47.708838 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1019142d-81ac-42e4-bd92-55e29ae2051a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1019142d-81ac-42e4-bd92-55e29ae2051a-maas-api-xks-ca\") pod \"1019142d-81ac-42e4-bd92-55e29ae2051a\" (UID: \"1019142d-81ac-42e4-bd92-55e29ae2051a\") " Nov 03 03:11:47.708914 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:47.708894 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1019142d-81ac-42e4-bd92-55e29ae2051a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1019142d-81ac-42e4-bd92-55e29ae2051a-maas-api-openshift-ca\") pod \"1019142d-81ac-42e4-bd92-55e29ae2051a\" (UID: \"1019142d-81ac-42e4-bd92-55e29ae2051a\") " Nov 03 03:11:47.709159 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:47.708919 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1019142d-81ac-42e4-bd92-55e29ae2051a-kube-api-access-pxvk4\" (UniqueName: \"kubernetes.io/projected/1019142d-81ac-42e4-bd92-55e29ae2051a-kube-api-access-pxvk4\") pod \"1019142d-81ac-42e4-bd92-55e29ae2051a\" (UID: \"1019142d-81ac-42e4-bd92-55e29ae2051a\") " Nov 03 03:11:47.709295 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:47.709263 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1019142d-81ac-42e4-bd92-55e29ae2051a-maas-api-openshift-ca" pod "1019142d-81ac-42e4-bd92-55e29ae2051a" (UID: "1019142d-81ac-42e4-bd92-55e29ae2051a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:11:47.710886 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:47.710865 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1019142d-81ac-42e4-bd92-55e29ae2051a-maas-api-xks-ca" pod "1019142d-81ac-42e4-bd92-55e29ae2051a" (UID: "1019142d-81ac-42e4-bd92-55e29ae2051a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:11:47.710965 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:47.710915 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1019142d-81ac-42e4-bd92-55e29ae2051a-kube-api-access-pxvk4" pod "1019142d-81ac-42e4-bd92-55e29ae2051a" (UID: "1019142d-81ac-42e4-bd92-55e29ae2051a"). InnerVolumeSpecName "kube-api-access-pxvk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:11:47.810261 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:47.810234 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxvk4\" (UniqueName: \"kubernetes.io/projected/1019142d-81ac-42e4-bd92-55e29ae2051a-kube-api-access-pxvk4\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:11:47.810261 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:47.810259 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1019142d-81ac-42e4-bd92-55e29ae2051a-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:11:47.810426 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:47.810269 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1019142d-81ac-42e4-bd92-55e29ae2051a-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:11:48.409020 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:48.408987 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a497f943-cwt9b" event={"ID":"1019142d-81ac-42e4-bd92-55e29ae2051a","Type":"ContainerDied","Data":"242e2d757a46ca427bf76074fd4d08d54dcf04e9768a1561ac1b77a6df6634ab"} Nov 03 03:11:48.409190 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:48.409044 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="242e2d757a46ca427bf76074fd4d08d54dcf04e9768a1561ac1b77a6df6634ab" Nov 03 03:11:48.409190 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:48.409001 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a497f943-cwt9b" Nov 03 03:11:53.784307 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:53.784263 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-3a4683/sub-test-model-3a4683-kserve-656f748dc5-nzhlm" podUID="d37c620a-1824-4c26-b8da-3e44aefcbe6b" containerName="main" containerID="cri-o://442939ee02dd9e2774683d5334b47bed5dbd85caa76189b21400a8c4fbf4c5d0" gracePeriod=45 Nov 03 03:11:53.915860 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:11:53.915818 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:11:54.043527 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:54.043460 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-3a4683/sub-test-model-3a4683-kserve-656f748dc5-nzhlm" Nov 03 03:11:54.074043 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:54.074014 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d37c620a-1824-4c26-b8da-3e44aefcbe6b-kube-api-access-xzfqm\" (UniqueName: \"kubernetes.io/projected/d37c620a-1824-4c26-b8da-3e44aefcbe6b-kube-api-access-xzfqm\") pod \"d37c620a-1824-4c26-b8da-3e44aefcbe6b\" (UID: \"d37c620a-1824-4c26-b8da-3e44aefcbe6b\") " Nov 03 03:11:54.074187 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:54.074067 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d37c620a-1824-4c26-b8da-3e44aefcbe6b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d37c620a-1824-4c26-b8da-3e44aefcbe6b-tmp-dir\") pod \"d37c620a-1824-4c26-b8da-3e44aefcbe6b\" (UID: \"d37c620a-1824-4c26-b8da-3e44aefcbe6b\") " Nov 03 03:11:54.074187 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:54.074115 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d37c620a-1824-4c26-b8da-3e44aefcbe6b-tls-certs\" (UniqueName: \"kubernetes.io/secret/d37c620a-1824-4c26-b8da-3e44aefcbe6b-tls-certs\") pod \"d37c620a-1824-4c26-b8da-3e44aefcbe6b\" (UID: \"d37c620a-1824-4c26-b8da-3e44aefcbe6b\") " Nov 03 03:11:54.074187 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:54.074131 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d37c620a-1824-4c26-b8da-3e44aefcbe6b-home\" (UniqueName: \"kubernetes.io/empty-dir/d37c620a-1824-4c26-b8da-3e44aefcbe6b-home\") pod \"d37c620a-1824-4c26-b8da-3e44aefcbe6b\" (UID: \"d37c620a-1824-4c26-b8da-3e44aefcbe6b\") " Nov 03 03:11:54.074187 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:54.074151 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d37c620a-1824-4c26-b8da-3e44aefcbe6b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d37c620a-1824-4c26-b8da-3e44aefcbe6b-model-cache\") pod \"d37c620a-1824-4c26-b8da-3e44aefcbe6b\" (UID: \"d37c620a-1824-4c26-b8da-3e44aefcbe6b\") " Nov 03 03:11:54.074391 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:54.074254 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d37c620a-1824-4c26-b8da-3e44aefcbe6b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d37c620a-1824-4c26-b8da-3e44aefcbe6b-dshm\") pod \"d37c620a-1824-4c26-b8da-3e44aefcbe6b\" (UID: \"d37c620a-1824-4c26-b8da-3e44aefcbe6b\") " Nov 03 03:11:54.074468 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:54.074342 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d37c620a-1824-4c26-b8da-3e44aefcbe6b-tmp-dir" pod "d37c620a-1824-4c26-b8da-3e44aefcbe6b" (UID: "d37c620a-1824-4c26-b8da-3e44aefcbe6b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:11:54.074530 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:54.074462 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d37c620a-1824-4c26-b8da-3e44aefcbe6b-home" pod "d37c620a-1824-4c26-b8da-3e44aefcbe6b" (UID: "d37c620a-1824-4c26-b8da-3e44aefcbe6b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:11:54.074591 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:54.074516 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d37c620a-1824-4c26-b8da-3e44aefcbe6b-model-cache" pod "d37c620a-1824-4c26-b8da-3e44aefcbe6b" (UID: "d37c620a-1824-4c26-b8da-3e44aefcbe6b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:11:54.074700 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:54.074677 5605 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d37c620a-1824-4c26-b8da-3e44aefcbe6b-home\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:11:54.074754 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:54.074703 5605 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d37c620a-1824-4c26-b8da-3e44aefcbe6b-model-cache\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:11:54.074754 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:54.074719 5605 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d37c620a-1824-4c26-b8da-3e44aefcbe6b-tmp-dir\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:11:54.076398 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:54.076350 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d37c620a-1824-4c26-b8da-3e44aefcbe6b-tls-certs" pod "d37c620a-1824-4c26-b8da-3e44aefcbe6b" (UID: "d37c620a-1824-4c26-b8da-3e44aefcbe6b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:11:54.076524 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:54.076415 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d37c620a-1824-4c26-b8da-3e44aefcbe6b-dshm" pod "d37c620a-1824-4c26-b8da-3e44aefcbe6b" (UID: "d37c620a-1824-4c26-b8da-3e44aefcbe6b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:11:54.076524 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:54.076516 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d37c620a-1824-4c26-b8da-3e44aefcbe6b-kube-api-access-xzfqm" pod "d37c620a-1824-4c26-b8da-3e44aefcbe6b" (UID: "d37c620a-1824-4c26-b8da-3e44aefcbe6b"). InnerVolumeSpecName "kube-api-access-xzfqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:11:54.175869 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:54.175829 5605 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d37c620a-1824-4c26-b8da-3e44aefcbe6b-dshm\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:11:54.175869 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:54.175856 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzfqm\" (UniqueName: \"kubernetes.io/projected/d37c620a-1824-4c26-b8da-3e44aefcbe6b-kube-api-access-xzfqm\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:11:54.175869 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:54.175869 5605 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d37c620a-1824-4c26-b8da-3e44aefcbe6b-tls-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:11:54.443969 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:54.443928 5605 generic.go:356] "Generic (PLEG): container finished" podID="d37c620a-1824-4c26-b8da-3e44aefcbe6b" containerID="442939ee02dd9e2774683d5334b47bed5dbd85caa76189b21400a8c4fbf4c5d0" exitCode=0 Nov 03 03:11:54.444172 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:54.443978 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-3a4683/sub-test-model-3a4683-kserve-656f748dc5-nzhlm" Nov 03 03:11:54.444172 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:54.444003 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-3a4683/sub-test-model-3a4683-kserve-656f748dc5-nzhlm" event={"ID":"d37c620a-1824-4c26-b8da-3e44aefcbe6b","Type":"ContainerDied","Data":"442939ee02dd9e2774683d5334b47bed5dbd85caa76189b21400a8c4fbf4c5d0"} Nov 03 03:11:54.444172 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:54.444049 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-3a4683/sub-test-model-3a4683-kserve-656f748dc5-nzhlm" event={"ID":"d37c620a-1824-4c26-b8da-3e44aefcbe6b","Type":"ContainerDied","Data":"a75ed5cb9d1324f2a27efeab99e926372a1f1a5b8e327340e40e9703d8c46a9a"} Nov 03 03:11:54.444172 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:54.444070 5605 scope.go:122] "RemoveContainer" containerID="442939ee02dd9e2774683d5334b47bed5dbd85caa76189b21400a8c4fbf4c5d0" Nov 03 03:11:54.456342 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:54.456311 5605 scope.go:122] "RemoveContainer" containerID="442939ee02dd9e2774683d5334b47bed5dbd85caa76189b21400a8c4fbf4c5d0" Nov 03 03:11:54.456695 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:11:54.456670 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"442939ee02dd9e2774683d5334b47bed5dbd85caa76189b21400a8c4fbf4c5d0\": container with ID starting with 442939ee02dd9e2774683d5334b47bed5dbd85caa76189b21400a8c4fbf4c5d0 not found: ID does not exist" containerID="442939ee02dd9e2774683d5334b47bed5dbd85caa76189b21400a8c4fbf4c5d0" Nov 03 03:11:54.456749 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:54.456708 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"442939ee02dd9e2774683d5334b47bed5dbd85caa76189b21400a8c4fbf4c5d0"} err="failed to get container status \"442939ee02dd9e2774683d5334b47bed5dbd85caa76189b21400a8c4fbf4c5d0\": rpc error: code = NotFound desc = could not find container \"442939ee02dd9e2774683d5334b47bed5dbd85caa76189b21400a8c4fbf4c5d0\": container with ID starting with 442939ee02dd9e2774683d5334b47bed5dbd85caa76189b21400a8c4fbf4c5d0 not found: ID does not exist" Nov 03 03:11:54.597170 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:54.597138 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-3a4683/sub-test-model-3a4683-kserve-656f748dc5-nzhlm"] Nov 03 03:11:54.598970 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:54.598949 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-3a4683/sub-test-model-3a4683-kserve-656f748dc5-nzhlm"] Nov 03 03:11:54.849500 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:54.849466 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d37c620a-1824-4c26-b8da-3e44aefcbe6b" path="/var/lib/kubelet/pods/d37c620a-1824-4c26-b8da-3e44aefcbe6b/volumes" Nov 03 03:11:56.382293 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:56.382259 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f36bf64b-nqdn4"] Nov 03 03:11:56.382711 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:56.382674 5605 cpu_manager.go:414] "containerMap: removing container" podUID="d37c620a-1824-4c26-b8da-3e44aefcbe6b" containerName="main" Nov 03 03:11:56.382711 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:56.382688 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d37c620a-1824-4c26-b8da-3e44aefcbe6b" containerName="main" Nov 03 03:11:56.382802 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:56.382725 5605 cpu_manager.go:414] "containerMap: removing container" podUID="1019142d-81ac-42e4-bd92-55e29ae2051a" containerName="revoke-keys" Nov 03 03:11:56.382802 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:56.382730 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1019142d-81ac-42e4-bd92-55e29ae2051a" containerName="revoke-keys" Nov 03 03:11:56.382802 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:56.382794 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="1019142d-81ac-42e4-bd92-55e29ae2051a" containerName="revoke-keys" Nov 03 03:11:56.382903 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:56.382807 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="d37c620a-1824-4c26-b8da-3e44aefcbe6b" containerName="main" Nov 03 03:11:56.387210 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:56.387190 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f36bf64b-nqdn4" Nov 03 03:11:56.484352 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:56.484319 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:11:56.484564 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:56.484387 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:11:56.484564 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:56.484395 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 03:11:56.484564 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:56.484439 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:11:56.486463 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:56.486444 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f36bf64b-nqdn4"] Nov 03 03:11:56.495265 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:56.495232 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b0e9dab-d915-423b-b2ff-20dc33b3a57f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f36bf64b-nqdn4\" (UID: \"9b0e9dab-d915-423b-b2ff-20dc33b3a57f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f36bf64b-nqdn4" Nov 03 03:11:56.495443 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:56.495279 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b0e9dab-d915-423b-b2ff-20dc33b3a57f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f36bf64b-nqdn4\" (UID: \"9b0e9dab-d915-423b-b2ff-20dc33b3a57f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f36bf64b-nqdn4" Nov 03 03:11:56.495443 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:56.495416 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn6q4\" (UniqueName: \"kubernetes.io/projected/9b0e9dab-d915-423b-b2ff-20dc33b3a57f-kube-api-access-rn6q4\") pod \"maas-api-revoke-sub-f36bf64b-nqdn4\" (UID: \"9b0e9dab-d915-423b-b2ff-20dc33b3a57f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f36bf64b-nqdn4" Nov 03 03:11:56.596859 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:56.596823 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn6q4\" (UniqueName: \"kubernetes.io/projected/9b0e9dab-d915-423b-b2ff-20dc33b3a57f-kube-api-access-rn6q4\") pod \"maas-api-revoke-sub-f36bf64b-nqdn4\" (UID: \"9b0e9dab-d915-423b-b2ff-20dc33b3a57f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f36bf64b-nqdn4" Nov 03 03:11:56.597049 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:56.596913 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b0e9dab-d915-423b-b2ff-20dc33b3a57f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f36bf64b-nqdn4\" (UID: \"9b0e9dab-d915-423b-b2ff-20dc33b3a57f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f36bf64b-nqdn4" Nov 03 03:11:56.597049 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:56.596953 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b0e9dab-d915-423b-b2ff-20dc33b3a57f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f36bf64b-nqdn4\" (UID: \"9b0e9dab-d915-423b-b2ff-20dc33b3a57f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f36bf64b-nqdn4" Nov 03 03:11:56.597631 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:56.597606 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b0e9dab-d915-423b-b2ff-20dc33b3a57f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f36bf64b-nqdn4\" (UID: \"9b0e9dab-d915-423b-b2ff-20dc33b3a57f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f36bf64b-nqdn4" Nov 03 03:11:56.599343 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:56.599304 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b0e9dab-d915-423b-b2ff-20dc33b3a57f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f36bf64b-nqdn4\" (UID: \"9b0e9dab-d915-423b-b2ff-20dc33b3a57f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f36bf64b-nqdn4" Nov 03 03:11:56.650529 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:56.650456 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn6q4\" (UniqueName: \"kubernetes.io/projected/9b0e9dab-d915-423b-b2ff-20dc33b3a57f-kube-api-access-rn6q4\") pod \"maas-api-revoke-sub-f36bf64b-nqdn4\" (UID: \"9b0e9dab-d915-423b-b2ff-20dc33b3a57f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f36bf64b-nqdn4" Nov 03 03:11:56.700508 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:56.700474 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f36bf64b-nqdn4" Nov 03 03:11:57.070551 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:57.070510 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f36bf64b-nqdn4"] Nov 03 03:11:57.471097 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:57.471059 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f36bf64b-nqdn4" event={"ID":"9b0e9dab-d915-423b-b2ff-20dc33b3a57f","Type":"ContainerStarted","Data":"d6a07f504b06c70dea2fe70aaf4bb7e1521f32d641fdf61aff0476a1c652e315"} Nov 03 03:11:57.471544 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:57.471138 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f36bf64b-nqdn4" event={"ID":"9b0e9dab-d915-423b-b2ff-20dc33b3a57f","Type":"ContainerStarted","Data":"fd43cabf2f535e013411c2b59784ba7123507a05e9a0e0848bc826037b89f963"} Nov 03 03:11:57.579221 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:57.579155 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f36bf64b-nqdn4" podStartSLOduration=1.579140529 podStartE2EDuration="1.579140529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:11:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:11:57.577883579 +0000 UTC m=+4794.837081743" watchObservedRunningTime="2026-11-03 03:11:57.579140529 +0000 UTC m=+4794.838338705" Nov 03 03:11:58.477682 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:58.477646 5605 generic.go:356] "Generic (PLEG): container finished" podID="9b0e9dab-d915-423b-b2ff-20dc33b3a57f" containerID="d6a07f504b06c70dea2fe70aaf4bb7e1521f32d641fdf61aff0476a1c652e315" exitCode=0 Nov 03 03:11:58.478102 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:58.477737 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f36bf64b-nqdn4" event={"ID":"9b0e9dab-d915-423b-b2ff-20dc33b3a57f","Type":"ContainerDied","Data":"d6a07f504b06c70dea2fe70aaf4bb7e1521f32d641fdf61aff0476a1c652e315"} Nov 03 03:11:59.611567 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:59.611539 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f36bf64b-nqdn4" Nov 03 03:11:59.625624 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:59.625594 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b0e9dab-d915-423b-b2ff-20dc33b3a57f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b0e9dab-d915-423b-b2ff-20dc33b3a57f-maas-api-xks-ca\") pod \"9b0e9dab-d915-423b-b2ff-20dc33b3a57f\" (UID: \"9b0e9dab-d915-423b-b2ff-20dc33b3a57f\") " Nov 03 03:11:59.625770 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:59.625654 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b0e9dab-d915-423b-b2ff-20dc33b3a57f-kube-api-access-rn6q4\" (UniqueName: \"kubernetes.io/projected/9b0e9dab-d915-423b-b2ff-20dc33b3a57f-kube-api-access-rn6q4\") pod \"9b0e9dab-d915-423b-b2ff-20dc33b3a57f\" (UID: \"9b0e9dab-d915-423b-b2ff-20dc33b3a57f\") " Nov 03 03:11:59.625770 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:59.625696 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b0e9dab-d915-423b-b2ff-20dc33b3a57f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b0e9dab-d915-423b-b2ff-20dc33b3a57f-maas-api-openshift-ca\") pod \"9b0e9dab-d915-423b-b2ff-20dc33b3a57f\" (UID: \"9b0e9dab-d915-423b-b2ff-20dc33b3a57f\") " Nov 03 03:11:59.626121 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:59.626083 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b0e9dab-d915-423b-b2ff-20dc33b3a57f-maas-api-openshift-ca" pod "9b0e9dab-d915-423b-b2ff-20dc33b3a57f" (UID: "9b0e9dab-d915-423b-b2ff-20dc33b3a57f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:11:59.627843 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:59.627817 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b0e9dab-d915-423b-b2ff-20dc33b3a57f-kube-api-access-rn6q4" pod "9b0e9dab-d915-423b-b2ff-20dc33b3a57f" (UID: "9b0e9dab-d915-423b-b2ff-20dc33b3a57f"). InnerVolumeSpecName "kube-api-access-rn6q4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:11:59.627942 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:59.627882 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b0e9dab-d915-423b-b2ff-20dc33b3a57f-maas-api-xks-ca" pod "9b0e9dab-d915-423b-b2ff-20dc33b3a57f" (UID: "9b0e9dab-d915-423b-b2ff-20dc33b3a57f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:11:59.727215 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:59.727183 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b0e9dab-d915-423b-b2ff-20dc33b3a57f-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:11:59.727215 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:59.727216 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rn6q4\" (UniqueName: \"kubernetes.io/projected/9b0e9dab-d915-423b-b2ff-20dc33b3a57f-kube-api-access-rn6q4\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:11:59.727472 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:11:59.727233 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b0e9dab-d915-423b-b2ff-20dc33b3a57f-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:00.491148 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:00.491121 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f36bf64b-nqdn4" Nov 03 03:12:00.491334 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:00.491159 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f36bf64b-nqdn4" event={"ID":"9b0e9dab-d915-423b-b2ff-20dc33b3a57f","Type":"ContainerDied","Data":"fd43cabf2f535e013411c2b59784ba7123507a05e9a0e0848bc826037b89f963"} Nov 03 03:12:00.491334 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:00.491209 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd43cabf2f535e013411c2b59784ba7123507a05e9a0e0848bc826037b89f963" Nov 03 03:12:05.301875 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:05.301839 5605 scope.go:122] "RemoveContainer" containerID="73c34d1382596fb22d42c25a27598a26b869a60417589e0d1d642c5c1f2b8d96" Nov 03 03:12:05.310730 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:05.310705 5605 scope.go:122] "RemoveContainer" containerID="f6ec3b850272a6b04e204c56f410b3c6667b307b5dfecd180ed8404032801fb6" Nov 03 03:12:07.397238 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:07.397205 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b81742eb-ggpd6"] Nov 03 03:12:07.397713 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:07.397582 5605 cpu_manager.go:414] "containerMap: removing container" podUID="9b0e9dab-d915-423b-b2ff-20dc33b3a57f" containerName="revoke-keys" Nov 03 03:12:07.397713 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:07.397593 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b0e9dab-d915-423b-b2ff-20dc33b3a57f" containerName="revoke-keys" Nov 03 03:12:07.397713 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:07.397672 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b0e9dab-d915-423b-b2ff-20dc33b3a57f" containerName="revoke-keys" Nov 03 03:12:07.402223 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:07.402200 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81742eb-ggpd6" Nov 03 03:12:07.454894 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:07.454867 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b81742eb-ggpd6"] Nov 03 03:12:07.482017 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:07.481996 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:12:07.498345 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:07.498324 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:12:07.498482 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:07.498383 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:12:07.498482 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:07.498426 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 03:12:07.503809 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:07.503784 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75fc2ee9-2682-4feb-b35b-b792d3aad995-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b81742eb-ggpd6\" (UID: \"75fc2ee9-2682-4feb-b35b-b792d3aad995\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81742eb-ggpd6" Nov 03 03:12:07.503888 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:07.503840 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75fc2ee9-2682-4feb-b35b-b792d3aad995-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b81742eb-ggpd6\" (UID: \"75fc2ee9-2682-4feb-b35b-b792d3aad995\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81742eb-ggpd6" Nov 03 03:12:07.503979 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:07.503959 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq7f2\" (UniqueName: \"kubernetes.io/projected/75fc2ee9-2682-4feb-b35b-b792d3aad995-kube-api-access-lq7f2\") pod \"maas-api-revoke-sub-b81742eb-ggpd6\" (UID: \"75fc2ee9-2682-4feb-b35b-b792d3aad995\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81742eb-ggpd6" Nov 03 03:12:07.605089 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:07.605059 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq7f2\" (UniqueName: \"kubernetes.io/projected/75fc2ee9-2682-4feb-b35b-b792d3aad995-kube-api-access-lq7f2\") pod \"maas-api-revoke-sub-b81742eb-ggpd6\" (UID: \"75fc2ee9-2682-4feb-b35b-b792d3aad995\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81742eb-ggpd6" Nov 03 03:12:07.605252 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:07.605124 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75fc2ee9-2682-4feb-b35b-b792d3aad995-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b81742eb-ggpd6\" (UID: \"75fc2ee9-2682-4feb-b35b-b792d3aad995\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81742eb-ggpd6" Nov 03 03:12:07.605252 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:07.605171 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75fc2ee9-2682-4feb-b35b-b792d3aad995-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b81742eb-ggpd6\" (UID: \"75fc2ee9-2682-4feb-b35b-b792d3aad995\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81742eb-ggpd6" Nov 03 03:12:07.605824 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:07.605804 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75fc2ee9-2682-4feb-b35b-b792d3aad995-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b81742eb-ggpd6\" (UID: \"75fc2ee9-2682-4feb-b35b-b792d3aad995\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81742eb-ggpd6" Nov 03 03:12:07.607496 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:07.607477 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75fc2ee9-2682-4feb-b35b-b792d3aad995-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b81742eb-ggpd6\" (UID: \"75fc2ee9-2682-4feb-b35b-b792d3aad995\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81742eb-ggpd6" Nov 03 03:12:07.658793 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:07.658736 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq7f2\" (UniqueName: \"kubernetes.io/projected/75fc2ee9-2682-4feb-b35b-b792d3aad995-kube-api-access-lq7f2\") pod \"maas-api-revoke-sub-b81742eb-ggpd6\" (UID: \"75fc2ee9-2682-4feb-b35b-b792d3aad995\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81742eb-ggpd6" Nov 03 03:12:07.715136 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:07.715115 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81742eb-ggpd6" Nov 03 03:12:08.080972 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:08.080943 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b81742eb-ggpd6"] Nov 03 03:12:08.086327 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:12:08.086295 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75fc2ee9_2682_4feb_b35b_b792d3aad995.slice/crio-753ed7a5f07cf2224abacfa1ca4880dd8fd21eada3f2c7ac18e86e7e08692119 WatchSource:0}: Error finding container 753ed7a5f07cf2224abacfa1ca4880dd8fd21eada3f2c7ac18e86e7e08692119: Status 404 returned error can't find the container with id 753ed7a5f07cf2224abacfa1ca4880dd8fd21eada3f2c7ac18e86e7e08692119 Nov 03 03:12:08.538586 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:08.538549 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81742eb-ggpd6" event={"ID":"75fc2ee9-2682-4feb-b35b-b792d3aad995","Type":"ContainerStarted","Data":"3f6d81044c13d278bdae1835818c1f8a2ec4479e9216af5e736222282aea37f4"} Nov 03 03:12:08.539096 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:08.538609 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81742eb-ggpd6" event={"ID":"75fc2ee9-2682-4feb-b35b-b792d3aad995","Type":"ContainerStarted","Data":"753ed7a5f07cf2224abacfa1ca4880dd8fd21eada3f2c7ac18e86e7e08692119"} Nov 03 03:12:08.646289 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:08.646182 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81742eb-ggpd6" podStartSLOduration=1.6461681320000001 podStartE2EDuration="1.646168132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:12:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:12:08.64463358 +0000 UTC m=+4805.903831735" watchObservedRunningTime="2026-11-03 03:12:08.646168132 +0000 UTC m=+4805.905366342" Nov 03 03:12:09.545101 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:09.545066 5605 generic.go:356] "Generic (PLEG): container finished" podID="75fc2ee9-2682-4feb-b35b-b792d3aad995" containerID="3f6d81044c13d278bdae1835818c1f8a2ec4479e9216af5e736222282aea37f4" exitCode=0 Nov 03 03:12:09.545583 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:09.545156 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81742eb-ggpd6" event={"ID":"75fc2ee9-2682-4feb-b35b-b792d3aad995","Type":"ContainerDied","Data":"3f6d81044c13d278bdae1835818c1f8a2ec4479e9216af5e736222282aea37f4"} Nov 03 03:12:10.680706 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:10.680679 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81742eb-ggpd6" Nov 03 03:12:10.740102 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:10.740069 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75fc2ee9-2682-4feb-b35b-b792d3aad995-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75fc2ee9-2682-4feb-b35b-b792d3aad995-maas-api-xks-ca\") pod \"75fc2ee9-2682-4feb-b35b-b792d3aad995\" (UID: \"75fc2ee9-2682-4feb-b35b-b792d3aad995\") " Nov 03 03:12:10.740102 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:10.740106 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75fc2ee9-2682-4feb-b35b-b792d3aad995-kube-api-access-lq7f2\" (UniqueName: \"kubernetes.io/projected/75fc2ee9-2682-4feb-b35b-b792d3aad995-kube-api-access-lq7f2\") pod \"75fc2ee9-2682-4feb-b35b-b792d3aad995\" (UID: \"75fc2ee9-2682-4feb-b35b-b792d3aad995\") " Nov 03 03:12:10.740347 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:10.740206 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75fc2ee9-2682-4feb-b35b-b792d3aad995-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75fc2ee9-2682-4feb-b35b-b792d3aad995-maas-api-openshift-ca\") pod \"75fc2ee9-2682-4feb-b35b-b792d3aad995\" (UID: \"75fc2ee9-2682-4feb-b35b-b792d3aad995\") " Nov 03 03:12:10.740649 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:10.740615 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75fc2ee9-2682-4feb-b35b-b792d3aad995-maas-api-openshift-ca" pod "75fc2ee9-2682-4feb-b35b-b792d3aad995" (UID: "75fc2ee9-2682-4feb-b35b-b792d3aad995"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:12:10.742308 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:10.742287 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75fc2ee9-2682-4feb-b35b-b792d3aad995-kube-api-access-lq7f2" pod "75fc2ee9-2682-4feb-b35b-b792d3aad995" (UID: "75fc2ee9-2682-4feb-b35b-b792d3aad995"). InnerVolumeSpecName "kube-api-access-lq7f2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:12:10.742437 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:10.742301 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75fc2ee9-2682-4feb-b35b-b792d3aad995-maas-api-xks-ca" pod "75fc2ee9-2682-4feb-b35b-b792d3aad995" (UID: "75fc2ee9-2682-4feb-b35b-b792d3aad995"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:12:10.841973 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:10.841888 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75fc2ee9-2682-4feb-b35b-b792d3aad995-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:10.841973 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:10.841922 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75fc2ee9-2682-4feb-b35b-b792d3aad995-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:10.841973 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:10.841933 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lq7f2\" (UniqueName: \"kubernetes.io/projected/75fc2ee9-2682-4feb-b35b-b792d3aad995-kube-api-access-lq7f2\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:11.557518 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:11.557488 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81742eb-ggpd6" Nov 03 03:12:11.557705 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:11.557487 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81742eb-ggpd6" event={"ID":"75fc2ee9-2682-4feb-b35b-b792d3aad995","Type":"ContainerDied","Data":"753ed7a5f07cf2224abacfa1ca4880dd8fd21eada3f2c7ac18e86e7e08692119"} Nov 03 03:12:11.557705 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:11.557609 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="753ed7a5f07cf2224abacfa1ca4880dd8fd21eada3f2c7ac18e86e7e08692119" Nov 03 03:12:17.154810 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:17.154777 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl"] Nov 03 03:12:17.155266 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:17.154970 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl" podUID="804673d5-10dd-463e-9c52-fc67bd1c307a" containerName="istio-proxy" containerID="cri-o://82c22ebb2f60b02952e61747fb938df5e0ac16162973a3a42a9ee177bfedc8bf" gracePeriod=30 Nov 03 03:12:22.413548 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:22.413520 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl" Nov 03 03:12:22.559738 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:22.559706 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/804673d5-10dd-463e-9c52-fc67bd1c307a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/804673d5-10dd-463e-9c52-fc67bd1c307a-istiod-ca-cert\") pod \"804673d5-10dd-463e-9c52-fc67bd1c307a\" (UID: \"804673d5-10dd-463e-9c52-fc67bd1c307a\") " Nov 03 03:12:22.559940 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:22.559753 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/804673d5-10dd-463e-9c52-fc67bd1c307a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/804673d5-10dd-463e-9c52-fc67bd1c307a-workload-socket\") pod \"804673d5-10dd-463e-9c52-fc67bd1c307a\" (UID: \"804673d5-10dd-463e-9c52-fc67bd1c307a\") " Nov 03 03:12:22.559940 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:22.559788 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/804673d5-10dd-463e-9c52-fc67bd1c307a-istio-token\" (UniqueName: \"kubernetes.io/projected/804673d5-10dd-463e-9c52-fc67bd1c307a-istio-token\") pod \"804673d5-10dd-463e-9c52-fc67bd1c307a\" (UID: \"804673d5-10dd-463e-9c52-fc67bd1c307a\") " Nov 03 03:12:22.559940 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:22.559851 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/804673d5-10dd-463e-9c52-fc67bd1c307a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/804673d5-10dd-463e-9c52-fc67bd1c307a-credential-socket\") pod \"804673d5-10dd-463e-9c52-fc67bd1c307a\" (UID: \"804673d5-10dd-463e-9c52-fc67bd1c307a\") " Nov 03 03:12:22.559940 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:22.559897 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/804673d5-10dd-463e-9c52-fc67bd1c307a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/804673d5-10dd-463e-9c52-fc67bd1c307a-istio-data\") pod \"804673d5-10dd-463e-9c52-fc67bd1c307a\" (UID: \"804673d5-10dd-463e-9c52-fc67bd1c307a\") " Nov 03 03:12:22.559940 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:22.559924 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/804673d5-10dd-463e-9c52-fc67bd1c307a-kube-api-access-8tfnn\" (UniqueName: \"kubernetes.io/projected/804673d5-10dd-463e-9c52-fc67bd1c307a-kube-api-access-8tfnn\") pod \"804673d5-10dd-463e-9c52-fc67bd1c307a\" (UID: \"804673d5-10dd-463e-9c52-fc67bd1c307a\") " Nov 03 03:12:22.560237 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:22.559951 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/804673d5-10dd-463e-9c52-fc67bd1c307a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/804673d5-10dd-463e-9c52-fc67bd1c307a-istio-envoy\") pod \"804673d5-10dd-463e-9c52-fc67bd1c307a\" (UID: \"804673d5-10dd-463e-9c52-fc67bd1c307a\") " Nov 03 03:12:22.560237 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:22.559978 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/804673d5-10dd-463e-9c52-fc67bd1c307a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/804673d5-10dd-463e-9c52-fc67bd1c307a-workload-certs\") pod \"804673d5-10dd-463e-9c52-fc67bd1c307a\" (UID: \"804673d5-10dd-463e-9c52-fc67bd1c307a\") " Nov 03 03:12:22.560237 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:22.560083 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/804673d5-10dd-463e-9c52-fc67bd1c307a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/804673d5-10dd-463e-9c52-fc67bd1c307a-istio-podinfo\") pod \"804673d5-10dd-463e-9c52-fc67bd1c307a\" (UID: \"804673d5-10dd-463e-9c52-fc67bd1c307a\") " Nov 03 03:12:22.560442 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:22.560062 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/804673d5-10dd-463e-9c52-fc67bd1c307a-workload-socket" pod "804673d5-10dd-463e-9c52-fc67bd1c307a" (UID: "804673d5-10dd-463e-9c52-fc67bd1c307a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:12:22.560442 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:22.560292 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/804673d5-10dd-463e-9c52-fc67bd1c307a-workload-certs" pod "804673d5-10dd-463e-9c52-fc67bd1c307a" (UID: "804673d5-10dd-463e-9c52-fc67bd1c307a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:12:22.560442 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:22.560310 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/804673d5-10dd-463e-9c52-fc67bd1c307a-istio-data" pod "804673d5-10dd-463e-9c52-fc67bd1c307a" (UID: "804673d5-10dd-463e-9c52-fc67bd1c307a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:12:22.560442 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:22.560321 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/804673d5-10dd-463e-9c52-fc67bd1c307a-credential-socket" pod "804673d5-10dd-463e-9c52-fc67bd1c307a" (UID: "804673d5-10dd-463e-9c52-fc67bd1c307a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:12:22.560662 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:22.560508 5605 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/804673d5-10dd-463e-9c52-fc67bd1c307a-workload-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:22.560662 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:22.560532 5605 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/804673d5-10dd-463e-9c52-fc67bd1c307a-credential-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:22.560662 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:22.560547 5605 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/804673d5-10dd-463e-9c52-fc67bd1c307a-istio-data\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:22.560662 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:22.560562 5605 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/804673d5-10dd-463e-9c52-fc67bd1c307a-workload-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:22.560867 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:22.560218 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/804673d5-10dd-463e-9c52-fc67bd1c307a-istiod-ca-cert" pod "804673d5-10dd-463e-9c52-fc67bd1c307a" (UID: "804673d5-10dd-463e-9c52-fc67bd1c307a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:12:22.562305 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:22.562281 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/804673d5-10dd-463e-9c52-fc67bd1c307a-istio-podinfo" pod "804673d5-10dd-463e-9c52-fc67bd1c307a" (UID: "804673d5-10dd-463e-9c52-fc67bd1c307a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:12:22.562440 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:22.562326 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/804673d5-10dd-463e-9c52-fc67bd1c307a-kube-api-access-8tfnn" pod "804673d5-10dd-463e-9c52-fc67bd1c307a" (UID: "804673d5-10dd-463e-9c52-fc67bd1c307a"). InnerVolumeSpecName "kube-api-access-8tfnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:12:22.562671 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:22.562646 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/804673d5-10dd-463e-9c52-fc67bd1c307a-istio-envoy" pod "804673d5-10dd-463e-9c52-fc67bd1c307a" (UID: "804673d5-10dd-463e-9c52-fc67bd1c307a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:12:22.562671 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:22.562661 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/804673d5-10dd-463e-9c52-fc67bd1c307a-istio-token" pod "804673d5-10dd-463e-9c52-fc67bd1c307a" (UID: "804673d5-10dd-463e-9c52-fc67bd1c307a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:12:22.622060 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:22.622027 5605 generic.go:356] "Generic (PLEG): container finished" podID="804673d5-10dd-463e-9c52-fc67bd1c307a" containerID="82c22ebb2f60b02952e61747fb938df5e0ac16162973a3a42a9ee177bfedc8bf" exitCode=0 Nov 03 03:12:22.622233 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:22.622092 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl" Nov 03 03:12:22.622233 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:22.622105 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl" event={"ID":"804673d5-10dd-463e-9c52-fc67bd1c307a","Type":"ContainerDied","Data":"82c22ebb2f60b02952e61747fb938df5e0ac16162973a3a42a9ee177bfedc8bf"} Nov 03 03:12:22.622233 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:22.622165 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl" event={"ID":"804673d5-10dd-463e-9c52-fc67bd1c307a","Type":"ContainerDied","Data":"61fe1c49ace8bbbe052f70a96b82b5e7d0f8eb041065941b72741b7d0c39388a"} Nov 03 03:12:22.622233 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:22.622179 5605 scope.go:122] "RemoveContainer" containerID="82c22ebb2f60b02952e61747fb938df5e0ac16162973a3a42a9ee177bfedc8bf" Nov 03 03:12:22.632228 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:22.632207 5605 scope.go:122] "RemoveContainer" containerID="ad496f17e0211a23c7841cc6aeee7430bf3e2835b24fdb916a66b3ee65b60328" Nov 03 03:12:22.640305 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:22.640288 5605 scope.go:122] "RemoveContainer" containerID="82c22ebb2f60b02952e61747fb938df5e0ac16162973a3a42a9ee177bfedc8bf" Nov 03 03:12:22.640607 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:12:22.640588 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82c22ebb2f60b02952e61747fb938df5e0ac16162973a3a42a9ee177bfedc8bf\": container with ID starting with 82c22ebb2f60b02952e61747fb938df5e0ac16162973a3a42a9ee177bfedc8bf not found: ID does not exist" containerID="82c22ebb2f60b02952e61747fb938df5e0ac16162973a3a42a9ee177bfedc8bf" Nov 03 03:12:22.640665 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:22.640616 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82c22ebb2f60b02952e61747fb938df5e0ac16162973a3a42a9ee177bfedc8bf"} err="failed to get container status \"82c22ebb2f60b02952e61747fb938df5e0ac16162973a3a42a9ee177bfedc8bf\": rpc error: code = NotFound desc = could not find container \"82c22ebb2f60b02952e61747fb938df5e0ac16162973a3a42a9ee177bfedc8bf\": container with ID starting with 82c22ebb2f60b02952e61747fb938df5e0ac16162973a3a42a9ee177bfedc8bf not found: ID does not exist" Nov 03 03:12:22.640665 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:22.640635 5605 scope.go:122] "RemoveContainer" containerID="ad496f17e0211a23c7841cc6aeee7430bf3e2835b24fdb916a66b3ee65b60328" Nov 03 03:12:22.640877 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:12:22.640858 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad496f17e0211a23c7841cc6aeee7430bf3e2835b24fdb916a66b3ee65b60328\": container with ID starting with ad496f17e0211a23c7841cc6aeee7430bf3e2835b24fdb916a66b3ee65b60328 not found: ID does not exist" containerID="ad496f17e0211a23c7841cc6aeee7430bf3e2835b24fdb916a66b3ee65b60328" Nov 03 03:12:22.640931 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:22.640886 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad496f17e0211a23c7841cc6aeee7430bf3e2835b24fdb916a66b3ee65b60328"} err="failed to get container status \"ad496f17e0211a23c7841cc6aeee7430bf3e2835b24fdb916a66b3ee65b60328\": rpc error: code = NotFound desc = could not find container \"ad496f17e0211a23c7841cc6aeee7430bf3e2835b24fdb916a66b3ee65b60328\": container with ID starting with ad496f17e0211a23c7841cc6aeee7430bf3e2835b24fdb916a66b3ee65b60328 not found: ID does not exist" Nov 03 03:12:22.661485 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:22.661455 5605 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/804673d5-10dd-463e-9c52-fc67bd1c307a-istio-token\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:22.661485 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:22.661484 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tfnn\" (UniqueName: \"kubernetes.io/projected/804673d5-10dd-463e-9c52-fc67bd1c307a-kube-api-access-8tfnn\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:22.661650 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:22.661496 5605 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/804673d5-10dd-463e-9c52-fc67bd1c307a-istio-envoy\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:22.661650 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:22.661505 5605 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/804673d5-10dd-463e-9c52-fc67bd1c307a-istio-podinfo\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:22.661650 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:22.661513 5605 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/804673d5-10dd-463e-9c52-fc67bd1c307a-istiod-ca-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:22.776753 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:22.776717 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl"] Nov 03 03:12:22.780486 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:22.780462 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-3a4683-openshift-default-bc75d8d55-j7vnl"] Nov 03 03:12:22.848908 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:22.848828 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="804673d5-10dd-463e-9c52-fc67bd1c307a" path="/var/lib/kubelet/pods/804673d5-10dd-463e-9c52-fc67bd1c307a/volumes" Nov 03 03:12:23.527901 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:23.527861 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-78b5c5-58dd96d55c-5jz2j"] Nov 03 03:12:23.912624 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:12:23.912581 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:12:25.155846 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:25.155804 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:12:28.528216 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:28.528167 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-78b5c5-58dd96d55c-5jz2j" podUID="754ba27d-cd77-4f65-a165-bb6f79b49664" containerName="payload-processing" containerID="cri-o://4d38a44a7076ec4c8b85134bab414ba905ad81e0f639a1bdd3ff20265224513e" gracePeriod=25 Nov 03 03:12:33.797255 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:33.797214 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-78b5c5-58dd96d55c-5jz2j" podUID="754ba27d-cd77-4f65-a165-bb6f79b49664" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:12:37.039495 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:37.039464 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-648bc910-dp469"] Nov 03 03:12:37.039867 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:37.039792 5605 cpu_manager.go:414] "containerMap: removing container" podUID="804673d5-10dd-463e-9c52-fc67bd1c307a" containerName="istio-proxy" Nov 03 03:12:37.039867 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:37.039803 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="804673d5-10dd-463e-9c52-fc67bd1c307a" containerName="istio-proxy" Nov 03 03:12:37.039867 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:37.039814 5605 cpu_manager.go:414] "containerMap: removing container" podUID="804673d5-10dd-463e-9c52-fc67bd1c307a" containerName="istio-proxy" Nov 03 03:12:37.039867 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:37.039819 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="804673d5-10dd-463e-9c52-fc67bd1c307a" containerName="istio-proxy" Nov 03 03:12:37.039867 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:37.039838 5605 cpu_manager.go:414] "containerMap: removing container" podUID="75fc2ee9-2682-4feb-b35b-b792d3aad995" containerName="revoke-keys" Nov 03 03:12:37.039867 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:37.039843 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75fc2ee9-2682-4feb-b35b-b792d3aad995" containerName="revoke-keys" Nov 03 03:12:37.040077 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:37.039894 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="804673d5-10dd-463e-9c52-fc67bd1c307a" containerName="istio-proxy" Nov 03 03:12:37.040077 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:37.039901 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="804673d5-10dd-463e-9c52-fc67bd1c307a" containerName="istio-proxy" Nov 03 03:12:37.040077 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:37.039911 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="75fc2ee9-2682-4feb-b35b-b792d3aad995" containerName="revoke-keys" Nov 03 03:12:37.044433 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:37.044412 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-648bc910-dp469" Nov 03 03:12:37.097577 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:37.097551 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-648bc910-dp469"] Nov 03 03:12:37.105643 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:37.105619 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6899acd9-62fa-46f2-8682-fc9fce7e5d06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-648bc910-dp469\" (UID: \"6899acd9-62fa-46f2-8682-fc9fce7e5d06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-648bc910-dp469" Nov 03 03:12:37.105732 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:37.105668 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6899acd9-62fa-46f2-8682-fc9fce7e5d06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-648bc910-dp469\" (UID: \"6899acd9-62fa-46f2-8682-fc9fce7e5d06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-648bc910-dp469" Nov 03 03:12:37.105844 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:37.105819 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdqxp\" (UniqueName: \"kubernetes.io/projected/6899acd9-62fa-46f2-8682-fc9fce7e5d06-kube-api-access-fdqxp\") pod \"maas-api-revoke-sub-648bc910-dp469\" (UID: \"6899acd9-62fa-46f2-8682-fc9fce7e5d06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-648bc910-dp469" Nov 03 03:12:37.142008 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:37.141988 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:12:37.142008 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:37.141996 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:12:37.142150 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:37.142007 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:12:37.142150 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:37.142067 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 03:12:37.207153 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:37.207130 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6899acd9-62fa-46f2-8682-fc9fce7e5d06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-648bc910-dp469\" (UID: \"6899acd9-62fa-46f2-8682-fc9fce7e5d06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-648bc910-dp469" Nov 03 03:12:37.207259 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:37.207191 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdqxp\" (UniqueName: \"kubernetes.io/projected/6899acd9-62fa-46f2-8682-fc9fce7e5d06-kube-api-access-fdqxp\") pod \"maas-api-revoke-sub-648bc910-dp469\" (UID: \"6899acd9-62fa-46f2-8682-fc9fce7e5d06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-648bc910-dp469" Nov 03 03:12:37.207259 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:37.207223 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6899acd9-62fa-46f2-8682-fc9fce7e5d06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-648bc910-dp469\" (UID: \"6899acd9-62fa-46f2-8682-fc9fce7e5d06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-648bc910-dp469" Nov 03 03:12:37.207850 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:37.207832 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6899acd9-62fa-46f2-8682-fc9fce7e5d06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-648bc910-dp469\" (UID: \"6899acd9-62fa-46f2-8682-fc9fce7e5d06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-648bc910-dp469" Nov 03 03:12:37.209478 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:37.209459 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6899acd9-62fa-46f2-8682-fc9fce7e5d06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-648bc910-dp469\" (UID: \"6899acd9-62fa-46f2-8682-fc9fce7e5d06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-648bc910-dp469" Nov 03 03:12:37.260330 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:37.260307 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdqxp\" (UniqueName: \"kubernetes.io/projected/6899acd9-62fa-46f2-8682-fc9fce7e5d06-kube-api-access-fdqxp\") pod \"maas-api-revoke-sub-648bc910-dp469\" (UID: \"6899acd9-62fa-46f2-8682-fc9fce7e5d06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-648bc910-dp469" Nov 03 03:12:37.356131 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:37.356113 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-648bc910-dp469" Nov 03 03:12:37.722775 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:37.722702 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-648bc910-dp469"] Nov 03 03:12:37.728656 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:12:37.728627 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6899acd9_62fa_46f2_8682_fc9fce7e5d06.slice/crio-a7cda7a15b600b6ef1a1aebd1a95b3d6976d7ab2dde6720a47ae01d371f8d579 WatchSource:0}: Error finding container a7cda7a15b600b6ef1a1aebd1a95b3d6976d7ab2dde6720a47ae01d371f8d579: Status 404 returned error can't find the container with id a7cda7a15b600b6ef1a1aebd1a95b3d6976d7ab2dde6720a47ae01d371f8d579 Nov 03 03:12:38.711055 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:38.711022 5605 generic.go:356] "Generic (PLEG): container finished" podID="6899acd9-62fa-46f2-8682-fc9fce7e5d06" containerID="6855d79955c48bb4ec56c1a705da9676ead08e637c085ad5075177b35aa4eb5d" exitCode=0 Nov 03 03:12:38.711415 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:38.711111 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-648bc910-dp469" event={"ID":"6899acd9-62fa-46f2-8682-fc9fce7e5d06","Type":"ContainerDied","Data":"6855d79955c48bb4ec56c1a705da9676ead08e637c085ad5075177b35aa4eb5d"} Nov 03 03:12:38.711415 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:38.711166 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-648bc910-dp469" event={"ID":"6899acd9-62fa-46f2-8682-fc9fce7e5d06","Type":"ContainerStarted","Data":"a7cda7a15b600b6ef1a1aebd1a95b3d6976d7ab2dde6720a47ae01d371f8d579"} Nov 03 03:12:39.854102 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:39.854073 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-648bc910-dp469" Nov 03 03:12:39.929869 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:39.929834 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6899acd9-62fa-46f2-8682-fc9fce7e5d06-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6899acd9-62fa-46f2-8682-fc9fce7e5d06-maas-api-xks-ca\") pod \"6899acd9-62fa-46f2-8682-fc9fce7e5d06\" (UID: \"6899acd9-62fa-46f2-8682-fc9fce7e5d06\") " Nov 03 03:12:39.930057 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:39.929916 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6899acd9-62fa-46f2-8682-fc9fce7e5d06-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6899acd9-62fa-46f2-8682-fc9fce7e5d06-maas-api-openshift-ca\") pod \"6899acd9-62fa-46f2-8682-fc9fce7e5d06\" (UID: \"6899acd9-62fa-46f2-8682-fc9fce7e5d06\") " Nov 03 03:12:39.930057 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:39.929957 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6899acd9-62fa-46f2-8682-fc9fce7e5d06-kube-api-access-fdqxp\" (UniqueName: \"kubernetes.io/projected/6899acd9-62fa-46f2-8682-fc9fce7e5d06-kube-api-access-fdqxp\") pod \"6899acd9-62fa-46f2-8682-fc9fce7e5d06\" (UID: \"6899acd9-62fa-46f2-8682-fc9fce7e5d06\") " Nov 03 03:12:39.930376 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:39.930329 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6899acd9-62fa-46f2-8682-fc9fce7e5d06-maas-api-openshift-ca" pod "6899acd9-62fa-46f2-8682-fc9fce7e5d06" (UID: "6899acd9-62fa-46f2-8682-fc9fce7e5d06"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:12:39.932114 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:39.932081 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6899acd9-62fa-46f2-8682-fc9fce7e5d06-maas-api-xks-ca" pod "6899acd9-62fa-46f2-8682-fc9fce7e5d06" (UID: "6899acd9-62fa-46f2-8682-fc9fce7e5d06"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:12:39.932208 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:39.932117 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6899acd9-62fa-46f2-8682-fc9fce7e5d06-kube-api-access-fdqxp" pod "6899acd9-62fa-46f2-8682-fc9fce7e5d06" (UID: "6899acd9-62fa-46f2-8682-fc9fce7e5d06"). InnerVolumeSpecName "kube-api-access-fdqxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:12:40.030698 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:40.030609 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6899acd9-62fa-46f2-8682-fc9fce7e5d06-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:40.030698 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:40.030638 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6899acd9-62fa-46f2-8682-fc9fce7e5d06-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:40.030698 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:40.030648 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdqxp\" (UniqueName: \"kubernetes.io/projected/6899acd9-62fa-46f2-8682-fc9fce7e5d06-kube-api-access-fdqxp\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:40.723635 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:40.723599 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-648bc910-dp469" Nov 03 03:12:40.723869 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:40.723626 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-648bc910-dp469" event={"ID":"6899acd9-62fa-46f2-8682-fc9fce7e5d06","Type":"ContainerDied","Data":"a7cda7a15b600b6ef1a1aebd1a95b3d6976d7ab2dde6720a47ae01d371f8d579"} Nov 03 03:12:40.723869 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:40.723706 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7cda7a15b600b6ef1a1aebd1a95b3d6976d7ab2dde6720a47ae01d371f8d579" Nov 03 03:12:41.396238 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:41.396203 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f7d9bb/distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr"] Nov 03 03:12:41.442787 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:41.442744 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f7d9bb/simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls"] Nov 03 03:12:43.383395 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:43.383349 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-957b33/sub-test-model-957b33-kserve-5888c7bc5-tb6t5"] Nov 03 03:12:43.797287 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:43.797190 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-78b5c5-58dd96d55c-5jz2j" podUID="754ba27d-cd77-4f65-a165-bb6f79b49664" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:12:48.770653 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:48.770624 5605 generic.go:356] "Generic (PLEG): container finished" podID="754ba27d-cd77-4f65-a165-bb6f79b49664" containerID="4d38a44a7076ec4c8b85134bab414ba905ad81e0f639a1bdd3ff20265224513e" exitCode=0 Nov 03 03:12:48.770978 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:48.770697 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-78b5c5-58dd96d55c-5jz2j" event={"ID":"754ba27d-cd77-4f65-a165-bb6f79b49664","Type":"ContainerDied","Data":"4d38a44a7076ec4c8b85134bab414ba905ad81e0f639a1bdd3ff20265224513e"} Nov 03 03:12:48.770978 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:48.770750 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-78b5c5-58dd96d55c-5jz2j" event={"ID":"754ba27d-cd77-4f65-a165-bb6f79b49664","Type":"ContainerDied","Data":"0d2f38bb3720741095e8b9bfb6406a71350eb0d9dbd4c70a95b8b12274895f1f"} Nov 03 03:12:48.770978 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:48.770768 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d2f38bb3720741095e8b9bfb6406a71350eb0d9dbd4c70a95b8b12274895f1f" Nov 03 03:12:48.781733 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:48.781713 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-78b5c5-58dd96d55c-5jz2j" Nov 03 03:12:48.819998 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:48.819977 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/754ba27d-cd77-4f65-a165-bb6f79b49664-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/754ba27d-cd77-4f65-a165-bb6f79b49664-plugins-config-volume\") pod \"754ba27d-cd77-4f65-a165-bb6f79b49664\" (UID: \"754ba27d-cd77-4f65-a165-bb6f79b49664\") " Nov 03 03:12:48.820113 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:48.820091 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/754ba27d-cd77-4f65-a165-bb6f79b49664-kube-api-access-rfhdf\" (UniqueName: \"kubernetes.io/projected/754ba27d-cd77-4f65-a165-bb6f79b49664-kube-api-access-rfhdf\") pod \"754ba27d-cd77-4f65-a165-bb6f79b49664\" (UID: \"754ba27d-cd77-4f65-a165-bb6f79b49664\") " Nov 03 03:12:48.820413 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:48.820392 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/754ba27d-cd77-4f65-a165-bb6f79b49664-plugins-config-volume" pod "754ba27d-cd77-4f65-a165-bb6f79b49664" (UID: "754ba27d-cd77-4f65-a165-bb6f79b49664"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:12:48.820511 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:48.820498 5605 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/754ba27d-cd77-4f65-a165-bb6f79b49664-plugins-config-volume\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:48.822095 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:48.822074 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/754ba27d-cd77-4f65-a165-bb6f79b49664-kube-api-access-rfhdf" pod "754ba27d-cd77-4f65-a165-bb6f79b49664" (UID: "754ba27d-cd77-4f65-a165-bb6f79b49664"). InnerVolumeSpecName "kube-api-access-rfhdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:12:48.921281 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:48.921247 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfhdf\" (UniqueName: \"kubernetes.io/projected/754ba27d-cd77-4f65-a165-bb6f79b49664-kube-api-access-rfhdf\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:49.776107 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:49.776075 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-78b5c5-58dd96d55c-5jz2j" Nov 03 03:12:49.929336 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:49.929300 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-78b5c5-58dd96d55c-5jz2j"] Nov 03 03:12:49.933083 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:49.933048 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-78b5c5-58dd96d55c-5jz2j"] Nov 03 03:12:50.849063 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:50.849025 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="754ba27d-cd77-4f65-a165-bb6f79b49664" path="/var/lib/kubelet/pods/754ba27d-cd77-4f65-a165-bb6f79b49664/volumes" Nov 03 03:12:51.518771 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:51.518735 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5517bac8-nhgwv"] Nov 03 03:12:51.519096 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:51.519082 5605 cpu_manager.go:414] "containerMap: removing container" podUID="754ba27d-cd77-4f65-a165-bb6f79b49664" containerName="payload-processing" Nov 03 03:12:51.519096 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:51.519095 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="754ba27d-cd77-4f65-a165-bb6f79b49664" containerName="payload-processing" Nov 03 03:12:51.519229 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:51.519127 5605 cpu_manager.go:414] "containerMap: removing container" podUID="6899acd9-62fa-46f2-8682-fc9fce7e5d06" containerName="revoke-keys" Nov 03 03:12:51.519229 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:51.519133 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6899acd9-62fa-46f2-8682-fc9fce7e5d06" containerName="revoke-keys" Nov 03 03:12:51.519229 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:51.519194 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="6899acd9-62fa-46f2-8682-fc9fce7e5d06" containerName="revoke-keys" Nov 03 03:12:51.519229 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:51.519206 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="754ba27d-cd77-4f65-a165-bb6f79b49664" containerName="payload-processing" Nov 03 03:12:51.522701 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:51.522680 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5517bac8-nhgwv" Nov 03 03:12:51.573316 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:51.573292 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:12:51.573474 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:51.573377 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:12:51.619885 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:51.619845 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:12:51.619885 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:51.619871 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 03:12:51.622083 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:51.622056 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5517bac8-nhgwv"] Nov 03 03:12:51.647396 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:51.647343 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad0e4039-aeec-4828-80de-2e3490319eec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5517bac8-nhgwv\" (UID: \"ad0e4039-aeec-4828-80de-2e3490319eec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5517bac8-nhgwv" Nov 03 03:12:51.647522 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:51.647503 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad0e4039-aeec-4828-80de-2e3490319eec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5517bac8-nhgwv\" (UID: \"ad0e4039-aeec-4828-80de-2e3490319eec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5517bac8-nhgwv" Nov 03 03:12:51.647576 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:51.647557 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cbb7\" (UniqueName: \"kubernetes.io/projected/ad0e4039-aeec-4828-80de-2e3490319eec-kube-api-access-6cbb7\") pod \"maas-api-revoke-sub-5517bac8-nhgwv\" (UID: \"ad0e4039-aeec-4828-80de-2e3490319eec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5517bac8-nhgwv" Nov 03 03:12:51.748722 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:51.748689 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad0e4039-aeec-4828-80de-2e3490319eec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5517bac8-nhgwv\" (UID: \"ad0e4039-aeec-4828-80de-2e3490319eec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5517bac8-nhgwv" Nov 03 03:12:51.748928 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:51.748853 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad0e4039-aeec-4828-80de-2e3490319eec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5517bac8-nhgwv\" (UID: \"ad0e4039-aeec-4828-80de-2e3490319eec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5517bac8-nhgwv" Nov 03 03:12:51.748928 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:51.748886 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cbb7\" (UniqueName: \"kubernetes.io/projected/ad0e4039-aeec-4828-80de-2e3490319eec-kube-api-access-6cbb7\") pod \"maas-api-revoke-sub-5517bac8-nhgwv\" (UID: \"ad0e4039-aeec-4828-80de-2e3490319eec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5517bac8-nhgwv" Nov 03 03:12:51.749495 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:51.749467 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad0e4039-aeec-4828-80de-2e3490319eec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5517bac8-nhgwv\" (UID: \"ad0e4039-aeec-4828-80de-2e3490319eec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5517bac8-nhgwv" Nov 03 03:12:51.751275 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:51.751251 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad0e4039-aeec-4828-80de-2e3490319eec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5517bac8-nhgwv\" (UID: \"ad0e4039-aeec-4828-80de-2e3490319eec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5517bac8-nhgwv" Nov 03 03:12:51.802093 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:51.802064 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cbb7\" (UniqueName: \"kubernetes.io/projected/ad0e4039-aeec-4828-80de-2e3490319eec-kube-api-access-6cbb7\") pod \"maas-api-revoke-sub-5517bac8-nhgwv\" (UID: \"ad0e4039-aeec-4828-80de-2e3490319eec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5517bac8-nhgwv" Nov 03 03:12:51.835595 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:51.835569 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5517bac8-nhgwv" Nov 03 03:12:52.201205 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:52.201124 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5517bac8-nhgwv"] Nov 03 03:12:52.207422 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:12:52.207390 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad0e4039_aeec_4828_80de_2e3490319eec.slice/crio-7896271c353fa6802afeffe1a4aca25b8042a3b9431e37e5dec35a4172c23273 WatchSource:0}: Error finding container 7896271c353fa6802afeffe1a4aca25b8042a3b9431e37e5dec35a4172c23273: Status 404 returned error can't find the container with id 7896271c353fa6802afeffe1a4aca25b8042a3b9431e37e5dec35a4172c23273 Nov 03 03:12:52.793735 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:52.793642 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5517bac8-nhgwv" event={"ID":"ad0e4039-aeec-4828-80de-2e3490319eec","Type":"ContainerStarted","Data":"ce21f876def0f1608d7b67a44826dc7200a28faf6a659f0eecf8d29d2e365bdd"} Nov 03 03:12:52.793735 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:52.793709 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5517bac8-nhgwv" event={"ID":"ad0e4039-aeec-4828-80de-2e3490319eec","Type":"ContainerStarted","Data":"7896271c353fa6802afeffe1a4aca25b8042a3b9431e37e5dec35a4172c23273"} Nov 03 03:12:52.904294 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:52.904232 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5517bac8-nhgwv" podStartSLOduration=1.904215558 podStartE2EDuration="1.904215558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:12:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:12:52.901120743 +0000 UTC m=+4850.160318899" watchObservedRunningTime="2026-11-03 03:12:52.904215558 +0000 UTC m=+4850.163413713" Nov 03 03:12:53.800100 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:53.800062 5605 generic.go:356] "Generic (PLEG): container finished" podID="ad0e4039-aeec-4828-80de-2e3490319eec" containerID="ce21f876def0f1608d7b67a44826dc7200a28faf6a659f0eecf8d29d2e365bdd" exitCode=0 Nov 03 03:12:53.800530 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:53.800143 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5517bac8-nhgwv" event={"ID":"ad0e4039-aeec-4828-80de-2e3490319eec","Type":"ContainerDied","Data":"ce21f876def0f1608d7b67a44826dc7200a28faf6a659f0eecf8d29d2e365bdd"} Nov 03 03:12:53.914371 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:12:53.914313 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:12:54.938162 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:54.938133 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5517bac8-nhgwv" Nov 03 03:12:55.083083 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:55.083000 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad0e4039-aeec-4828-80de-2e3490319eec-kube-api-access-6cbb7\" (UniqueName: \"kubernetes.io/projected/ad0e4039-aeec-4828-80de-2e3490319eec-kube-api-access-6cbb7\") pod \"ad0e4039-aeec-4828-80de-2e3490319eec\" (UID: \"ad0e4039-aeec-4828-80de-2e3490319eec\") " Nov 03 03:12:55.083264 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:55.083108 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad0e4039-aeec-4828-80de-2e3490319eec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad0e4039-aeec-4828-80de-2e3490319eec-maas-api-xks-ca\") pod \"ad0e4039-aeec-4828-80de-2e3490319eec\" (UID: \"ad0e4039-aeec-4828-80de-2e3490319eec\") " Nov 03 03:12:55.083264 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:55.083149 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad0e4039-aeec-4828-80de-2e3490319eec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad0e4039-aeec-4828-80de-2e3490319eec-maas-api-openshift-ca\") pod \"ad0e4039-aeec-4828-80de-2e3490319eec\" (UID: \"ad0e4039-aeec-4828-80de-2e3490319eec\") " Nov 03 03:12:55.083584 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:55.083555 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad0e4039-aeec-4828-80de-2e3490319eec-maas-api-openshift-ca" pod "ad0e4039-aeec-4828-80de-2e3490319eec" (UID: "ad0e4039-aeec-4828-80de-2e3490319eec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:12:55.085174 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:55.085143 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad0e4039-aeec-4828-80de-2e3490319eec-kube-api-access-6cbb7" pod "ad0e4039-aeec-4828-80de-2e3490319eec" (UID: "ad0e4039-aeec-4828-80de-2e3490319eec"). InnerVolumeSpecName "kube-api-access-6cbb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:12:55.085260 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:55.085176 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad0e4039-aeec-4828-80de-2e3490319eec-maas-api-xks-ca" pod "ad0e4039-aeec-4828-80de-2e3490319eec" (UID: "ad0e4039-aeec-4828-80de-2e3490319eec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:12:55.183812 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:55.183776 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad0e4039-aeec-4828-80de-2e3490319eec-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:55.183812 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:55.183809 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad0e4039-aeec-4828-80de-2e3490319eec-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:55.183812 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:55.183819 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cbb7\" (UniqueName: \"kubernetes.io/projected/ad0e4039-aeec-4828-80de-2e3490319eec-kube-api-access-6cbb7\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:55.811210 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:55.811173 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5517bac8-nhgwv" event={"ID":"ad0e4039-aeec-4828-80de-2e3490319eec","Type":"ContainerDied","Data":"7896271c353fa6802afeffe1a4aca25b8042a3b9431e37e5dec35a4172c23273"} Nov 03 03:12:55.811451 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:55.811236 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7896271c353fa6802afeffe1a4aca25b8042a3b9431e37e5dec35a4172c23273" Nov 03 03:12:55.811451 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:55.811191 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5517bac8-nhgwv" Nov 03 03:12:56.419939 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:56.419901 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr" podUID="8282371b-c42d-43ab-a5c9-e07f2464dcdc" containerName="main" containerID="cri-o://480a71ce8070590f25882499d2be0a4c3fb366ef13abf4514f5982aa74a059fb" gracePeriod=45 Nov 03 03:12:56.465622 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:56.465590 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f7d9bb/simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls" podUID="9173364c-14ba-4287-8587-45f25118e3e4" containerName="main" containerID="cri-o://da057a3221ca59a939aed416d0fbc48f65c67c1cc1ab277bb4d75939b0b9c09b" gracePeriod=45 Nov 03 03:12:56.704254 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:56.704229 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr" Nov 03 03:12:56.728030 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:56.728006 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f7d9bb/simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls" Nov 03 03:12:56.798528 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:56.798499 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8282371b-c42d-43ab-a5c9-e07f2464dcdc-kube-api-access-bx29r\" (UniqueName: \"kubernetes.io/projected/8282371b-c42d-43ab-a5c9-e07f2464dcdc-kube-api-access-bx29r\") pod \"8282371b-c42d-43ab-a5c9-e07f2464dcdc\" (UID: \"8282371b-c42d-43ab-a5c9-e07f2464dcdc\") " Nov 03 03:12:56.798648 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:56.798548 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9173364c-14ba-4287-8587-45f25118e3e4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9173364c-14ba-4287-8587-45f25118e3e4-dshm\") pod \"9173364c-14ba-4287-8587-45f25118e3e4\" (UID: \"9173364c-14ba-4287-8587-45f25118e3e4\") " Nov 03 03:12:56.798648 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:56.798578 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9173364c-14ba-4287-8587-45f25118e3e4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9173364c-14ba-4287-8587-45f25118e3e4-tmp-dir\") pod \"9173364c-14ba-4287-8587-45f25118e3e4\" (UID: \"9173364c-14ba-4287-8587-45f25118e3e4\") " Nov 03 03:12:56.798648 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:56.798615 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9173364c-14ba-4287-8587-45f25118e3e4-tls-certs\" (UniqueName: \"kubernetes.io/secret/9173364c-14ba-4287-8587-45f25118e3e4-tls-certs\") pod \"9173364c-14ba-4287-8587-45f25118e3e4\" (UID: \"9173364c-14ba-4287-8587-45f25118e3e4\") " Nov 03 03:12:56.798761 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:56.798686 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9173364c-14ba-4287-8587-45f25118e3e4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9173364c-14ba-4287-8587-45f25118e3e4-model-cache\") pod \"9173364c-14ba-4287-8587-45f25118e3e4\" (UID: \"9173364c-14ba-4287-8587-45f25118e3e4\") " Nov 03 03:12:56.798761 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:56.798721 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9173364c-14ba-4287-8587-45f25118e3e4-kube-api-access-d5c5v\" (UniqueName: \"kubernetes.io/projected/9173364c-14ba-4287-8587-45f25118e3e4-kube-api-access-d5c5v\") pod \"9173364c-14ba-4287-8587-45f25118e3e4\" (UID: \"9173364c-14ba-4287-8587-45f25118e3e4\") " Nov 03 03:12:56.798761 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:56.798752 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8282371b-c42d-43ab-a5c9-e07f2464dcdc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8282371b-c42d-43ab-a5c9-e07f2464dcdc-model-cache\") pod \"8282371b-c42d-43ab-a5c9-e07f2464dcdc\" (UID: \"8282371b-c42d-43ab-a5c9-e07f2464dcdc\") " Nov 03 03:12:56.798922 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:56.798846 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8282371b-c42d-43ab-a5c9-e07f2464dcdc-tls-certs\" (UniqueName: \"kubernetes.io/secret/8282371b-c42d-43ab-a5c9-e07f2464dcdc-tls-certs\") pod \"8282371b-c42d-43ab-a5c9-e07f2464dcdc\" (UID: \"8282371b-c42d-43ab-a5c9-e07f2464dcdc\") " Nov 03 03:12:56.798922 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:56.798875 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8282371b-c42d-43ab-a5c9-e07f2464dcdc-home\" (UniqueName: \"kubernetes.io/empty-dir/8282371b-c42d-43ab-a5c9-e07f2464dcdc-home\") pod \"8282371b-c42d-43ab-a5c9-e07f2464dcdc\" (UID: \"8282371b-c42d-43ab-a5c9-e07f2464dcdc\") " Nov 03 03:12:56.798922 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:56.798879 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9173364c-14ba-4287-8587-45f25118e3e4-tmp-dir" pod "9173364c-14ba-4287-8587-45f25118e3e4" (UID: "9173364c-14ba-4287-8587-45f25118e3e4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:12:56.798922 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:56.798904 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9173364c-14ba-4287-8587-45f25118e3e4-home\" (UniqueName: \"kubernetes.io/empty-dir/9173364c-14ba-4287-8587-45f25118e3e4-home\") pod \"9173364c-14ba-4287-8587-45f25118e3e4\" (UID: \"9173364c-14ba-4287-8587-45f25118e3e4\") " Nov 03 03:12:56.799132 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:56.798995 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9173364c-14ba-4287-8587-45f25118e3e4-model-cache" pod "9173364c-14ba-4287-8587-45f25118e3e4" (UID: "9173364c-14ba-4287-8587-45f25118e3e4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:12:56.799132 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:56.799112 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9173364c-14ba-4287-8587-45f25118e3e4-home" pod "9173364c-14ba-4287-8587-45f25118e3e4" (UID: "9173364c-14ba-4287-8587-45f25118e3e4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:12:56.799246 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:56.799123 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8282371b-c42d-43ab-a5c9-e07f2464dcdc-model-cache" pod "8282371b-c42d-43ab-a5c9-e07f2464dcdc" (UID: "8282371b-c42d-43ab-a5c9-e07f2464dcdc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:12:56.799246 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:56.799231 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8282371b-c42d-43ab-a5c9-e07f2464dcdc-home" pod "8282371b-c42d-43ab-a5c9-e07f2464dcdc" (UID: "8282371b-c42d-43ab-a5c9-e07f2464dcdc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:12:56.799386 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:56.799237 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8282371b-c42d-43ab-a5c9-e07f2464dcdc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8282371b-c42d-43ab-a5c9-e07f2464dcdc-tmp-dir\") pod \"8282371b-c42d-43ab-a5c9-e07f2464dcdc\" (UID: \"8282371b-c42d-43ab-a5c9-e07f2464dcdc\") " Nov 03 03:12:56.799386 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:56.799295 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8282371b-c42d-43ab-a5c9-e07f2464dcdc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8282371b-c42d-43ab-a5c9-e07f2464dcdc-dshm\") pod \"8282371b-c42d-43ab-a5c9-e07f2464dcdc\" (UID: \"8282371b-c42d-43ab-a5c9-e07f2464dcdc\") " Nov 03 03:12:56.799505 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:56.799420 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8282371b-c42d-43ab-a5c9-e07f2464dcdc-tmp-dir" pod "8282371b-c42d-43ab-a5c9-e07f2464dcdc" (UID: "8282371b-c42d-43ab-a5c9-e07f2464dcdc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:12:56.799732 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:56.799696 5605 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8282371b-c42d-43ab-a5c9-e07f2464dcdc-model-cache\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:56.799732 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:56.799720 5605 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8282371b-c42d-43ab-a5c9-e07f2464dcdc-home\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:56.799732 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:56.799734 5605 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9173364c-14ba-4287-8587-45f25118e3e4-home\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:56.799992 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:56.799747 5605 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8282371b-c42d-43ab-a5c9-e07f2464dcdc-tmp-dir\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:56.799992 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:56.799759 5605 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9173364c-14ba-4287-8587-45f25118e3e4-tmp-dir\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:56.799992 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:56.799771 5605 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9173364c-14ba-4287-8587-45f25118e3e4-model-cache\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:56.801032 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:56.801007 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8282371b-c42d-43ab-a5c9-e07f2464dcdc-tls-certs" pod "8282371b-c42d-43ab-a5c9-e07f2464dcdc" (UID: "8282371b-c42d-43ab-a5c9-e07f2464dcdc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:12:56.801215 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:56.801194 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9173364c-14ba-4287-8587-45f25118e3e4-dshm" pod "9173364c-14ba-4287-8587-45f25118e3e4" (UID: "9173364c-14ba-4287-8587-45f25118e3e4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:12:56.801279 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:56.801196 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9173364c-14ba-4287-8587-45f25118e3e4-tls-certs" pod "9173364c-14ba-4287-8587-45f25118e3e4" (UID: "9173364c-14ba-4287-8587-45f25118e3e4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:12:56.801279 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:56.801225 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8282371b-c42d-43ab-a5c9-e07f2464dcdc-kube-api-access-bx29r" pod "8282371b-c42d-43ab-a5c9-e07f2464dcdc" (UID: "8282371b-c42d-43ab-a5c9-e07f2464dcdc"). InnerVolumeSpecName "kube-api-access-bx29r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:12:56.801279 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:56.801259 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8282371b-c42d-43ab-a5c9-e07f2464dcdc-dshm" pod "8282371b-c42d-43ab-a5c9-e07f2464dcdc" (UID: "8282371b-c42d-43ab-a5c9-e07f2464dcdc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:12:56.801575 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:56.801560 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9173364c-14ba-4287-8587-45f25118e3e4-kube-api-access-d5c5v" pod "9173364c-14ba-4287-8587-45f25118e3e4" (UID: "9173364c-14ba-4287-8587-45f25118e3e4"). InnerVolumeSpecName "kube-api-access-d5c5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:12:56.818503 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:56.818480 5605 generic.go:356] "Generic (PLEG): container finished" podID="9173364c-14ba-4287-8587-45f25118e3e4" containerID="da057a3221ca59a939aed416d0fbc48f65c67c1cc1ab277bb4d75939b0b9c09b" exitCode=0 Nov 03 03:12:56.818623 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:56.818520 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f7d9bb/simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls" Nov 03 03:12:56.818623 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:56.818566 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f7d9bb/simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls" event={"ID":"9173364c-14ba-4287-8587-45f25118e3e4","Type":"ContainerDied","Data":"da057a3221ca59a939aed416d0fbc48f65c67c1cc1ab277bb4d75939b0b9c09b"} Nov 03 03:12:56.818623 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:56.818613 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f7d9bb/simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls" event={"ID":"9173364c-14ba-4287-8587-45f25118e3e4","Type":"ContainerDied","Data":"0b47d7fc7966b0833b713c5a694705501f0ed4e82d42fe3c9202d2a1a0839852"} Nov 03 03:12:56.818806 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:56.818636 5605 scope.go:122] "RemoveContainer" containerID="da057a3221ca59a939aed416d0fbc48f65c67c1cc1ab277bb4d75939b0b9c09b" Nov 03 03:12:56.820086 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:56.820066 5605 generic.go:356] "Generic (PLEG): container finished" podID="8282371b-c42d-43ab-a5c9-e07f2464dcdc" containerID="480a71ce8070590f25882499d2be0a4c3fb366ef13abf4514f5982aa74a059fb" exitCode=0 Nov 03 03:12:56.820191 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:56.820114 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr" Nov 03 03:12:56.820191 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:56.820142 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr" event={"ID":"8282371b-c42d-43ab-a5c9-e07f2464dcdc","Type":"ContainerDied","Data":"480a71ce8070590f25882499d2be0a4c3fb366ef13abf4514f5982aa74a059fb"} Nov 03 03:12:56.820191 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:56.820168 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr" event={"ID":"8282371b-c42d-43ab-a5c9-e07f2464dcdc","Type":"ContainerDied","Data":"e219fd27167f74f99b4c15bb989a0c138ece678ec7484e623e208af985d622fc"} Nov 03 03:12:56.828313 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:56.828283 5605 scope.go:122] "RemoveContainer" containerID="da057a3221ca59a939aed416d0fbc48f65c67c1cc1ab277bb4d75939b0b9c09b" Nov 03 03:12:56.828709 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:12:56.828691 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da057a3221ca59a939aed416d0fbc48f65c67c1cc1ab277bb4d75939b0b9c09b\": container with ID starting with da057a3221ca59a939aed416d0fbc48f65c67c1cc1ab277bb4d75939b0b9c09b not found: ID does not exist" containerID="da057a3221ca59a939aed416d0fbc48f65c67c1cc1ab277bb4d75939b0b9c09b" Nov 03 03:12:56.828785 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:56.828717 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da057a3221ca59a939aed416d0fbc48f65c67c1cc1ab277bb4d75939b0b9c09b"} err="failed to get container status \"da057a3221ca59a939aed416d0fbc48f65c67c1cc1ab277bb4d75939b0b9c09b\": rpc error: code = NotFound desc = could not find container \"da057a3221ca59a939aed416d0fbc48f65c67c1cc1ab277bb4d75939b0b9c09b\": container with ID starting with da057a3221ca59a939aed416d0fbc48f65c67c1cc1ab277bb4d75939b0b9c09b not found: ID does not exist" Nov 03 03:12:56.828785 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:56.828732 5605 scope.go:122] "RemoveContainer" containerID="480a71ce8070590f25882499d2be0a4c3fb366ef13abf4514f5982aa74a059fb" Nov 03 03:12:56.836307 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:56.836291 5605 scope.go:122] "RemoveContainer" containerID="480a71ce8070590f25882499d2be0a4c3fb366ef13abf4514f5982aa74a059fb" Nov 03 03:12:56.836582 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:12:56.836564 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"480a71ce8070590f25882499d2be0a4c3fb366ef13abf4514f5982aa74a059fb\": container with ID starting with 480a71ce8070590f25882499d2be0a4c3fb366ef13abf4514f5982aa74a059fb not found: ID does not exist" containerID="480a71ce8070590f25882499d2be0a4c3fb366ef13abf4514f5982aa74a059fb" Nov 03 03:12:56.836631 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:56.836588 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"480a71ce8070590f25882499d2be0a4c3fb366ef13abf4514f5982aa74a059fb"} err="failed to get container status \"480a71ce8070590f25882499d2be0a4c3fb366ef13abf4514f5982aa74a059fb\": rpc error: code = NotFound desc = could not find container \"480a71ce8070590f25882499d2be0a4c3fb366ef13abf4514f5982aa74a059fb\": container with ID starting with 480a71ce8070590f25882499d2be0a4c3fb366ef13abf4514f5982aa74a059fb not found: ID does not exist" Nov 03 03:12:56.901059 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:56.901036 5605 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9173364c-14ba-4287-8587-45f25118e3e4-dshm\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:56.901059 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:56.901058 5605 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9173364c-14ba-4287-8587-45f25118e3e4-tls-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:56.901222 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:56.901068 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5c5v\" (UniqueName: \"kubernetes.io/projected/9173364c-14ba-4287-8587-45f25118e3e4-kube-api-access-d5c5v\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:56.901222 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:56.901080 5605 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8282371b-c42d-43ab-a5c9-e07f2464dcdc-tls-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:56.901222 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:56.901089 5605 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8282371b-c42d-43ab-a5c9-e07f2464dcdc-dshm\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:56.901222 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:56.901098 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx29r\" (UniqueName: \"kubernetes.io/projected/8282371b-c42d-43ab-a5c9-e07f2464dcdc-kube-api-access-bx29r\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:57.026978 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:57.026904 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f7d9bb/simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls"] Nov 03 03:12:57.030455 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:57.030425 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f7d9bb/simulator-w0-f7d9bb-kserve-5f57d66f99-zpjls"] Nov 03 03:12:57.131834 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:57.131806 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f7d9bb/distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr"] Nov 03 03:12:57.135151 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:57.135126 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f7d9bb/distinct-w0-f7d9bb-kserve-688c46cfdf-g7zqr"] Nov 03 03:12:57.295247 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:57.295181 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdfaf0b9-6n8kr"] Nov 03 03:12:57.295724 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:57.295702 5605 cpu_manager.go:414] "containerMap: removing container" podUID="8282371b-c42d-43ab-a5c9-e07f2464dcdc" containerName="main" Nov 03 03:12:57.295724 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:57.295724 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8282371b-c42d-43ab-a5c9-e07f2464dcdc" containerName="main" Nov 03 03:12:57.295892 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:57.295753 5605 cpu_manager.go:414] "containerMap: removing container" podUID="9173364c-14ba-4287-8587-45f25118e3e4" containerName="main" Nov 03 03:12:57.295892 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:57.295762 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9173364c-14ba-4287-8587-45f25118e3e4" containerName="main" Nov 03 03:12:57.295892 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:57.295798 5605 cpu_manager.go:414] "containerMap: removing container" podUID="ad0e4039-aeec-4828-80de-2e3490319eec" containerName="revoke-keys" Nov 03 03:12:57.295892 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:57.295806 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad0e4039-aeec-4828-80de-2e3490319eec" containerName="revoke-keys" Nov 03 03:12:57.296046 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:57.295900 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="9173364c-14ba-4287-8587-45f25118e3e4" containerName="main" Nov 03 03:12:57.296046 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:57.295921 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad0e4039-aeec-4828-80de-2e3490319eec" containerName="revoke-keys" Nov 03 03:12:57.296046 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:57.295930 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="8282371b-c42d-43ab-a5c9-e07f2464dcdc" containerName="main" Nov 03 03:12:57.300465 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:57.300444 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdfaf0b9-6n8kr" Nov 03 03:12:57.350199 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:57.350175 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:12:57.350343 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:57.350264 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 03:12:57.354937 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:57.352829 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdfaf0b9-6n8kr"] Nov 03 03:12:57.405477 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:57.405445 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b106f86-5a55-4614-9373-6a3f898d369f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cdfaf0b9-6n8kr\" (UID: \"8b106f86-5a55-4614-9373-6a3f898d369f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdfaf0b9-6n8kr" Nov 03 03:12:57.405606 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:57.405571 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndp6p\" (UniqueName: \"kubernetes.io/projected/8b106f86-5a55-4614-9373-6a3f898d369f-kube-api-access-ndp6p\") pod \"maas-api-revoke-sub-cdfaf0b9-6n8kr\" (UID: \"8b106f86-5a55-4614-9373-6a3f898d369f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdfaf0b9-6n8kr" Nov 03 03:12:57.405673 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:57.405608 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b106f86-5a55-4614-9373-6a3f898d369f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cdfaf0b9-6n8kr\" (UID: \"8b106f86-5a55-4614-9373-6a3f898d369f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdfaf0b9-6n8kr" Nov 03 03:12:57.415683 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:57.415660 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:12:57.415772 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:57.415688 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:12:57.506307 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:57.506283 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b106f86-5a55-4614-9373-6a3f898d369f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cdfaf0b9-6n8kr\" (UID: \"8b106f86-5a55-4614-9373-6a3f898d369f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdfaf0b9-6n8kr" Nov 03 03:12:57.506611 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:57.506326 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndp6p\" (UniqueName: \"kubernetes.io/projected/8b106f86-5a55-4614-9373-6a3f898d369f-kube-api-access-ndp6p\") pod \"maas-api-revoke-sub-cdfaf0b9-6n8kr\" (UID: \"8b106f86-5a55-4614-9373-6a3f898d369f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdfaf0b9-6n8kr" Nov 03 03:12:57.506611 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:57.506347 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b106f86-5a55-4614-9373-6a3f898d369f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cdfaf0b9-6n8kr\" (UID: \"8b106f86-5a55-4614-9373-6a3f898d369f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdfaf0b9-6n8kr" Nov 03 03:12:57.506940 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:57.506919 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b106f86-5a55-4614-9373-6a3f898d369f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cdfaf0b9-6n8kr\" (UID: \"8b106f86-5a55-4614-9373-6a3f898d369f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdfaf0b9-6n8kr" Nov 03 03:12:57.508608 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:57.508587 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b106f86-5a55-4614-9373-6a3f898d369f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cdfaf0b9-6n8kr\" (UID: \"8b106f86-5a55-4614-9373-6a3f898d369f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdfaf0b9-6n8kr" Nov 03 03:12:57.559553 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:57.559522 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndp6p\" (UniqueName: \"kubernetes.io/projected/8b106f86-5a55-4614-9373-6a3f898d369f-kube-api-access-ndp6p\") pod \"maas-api-revoke-sub-cdfaf0b9-6n8kr\" (UID: \"8b106f86-5a55-4614-9373-6a3f898d369f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdfaf0b9-6n8kr" Nov 03 03:12:57.614387 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:57.614346 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdfaf0b9-6n8kr" Nov 03 03:12:57.983846 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:57.983773 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdfaf0b9-6n8kr"] Nov 03 03:12:57.989601 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:12:57.989571 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b106f86_5a55_4614_9373_6a3f898d369f.slice/crio-ba94d1dc57575c4c53593b9bf764eaebae6a7efd901e37416f1f7b7125b84262 WatchSource:0}: Error finding container ba94d1dc57575c4c53593b9bf764eaebae6a7efd901e37416f1f7b7125b84262: Status 404 returned error can't find the container with id ba94d1dc57575c4c53593b9bf764eaebae6a7efd901e37416f1f7b7125b84262 Nov 03 03:12:58.403539 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:58.403501 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-957b33/sub-test-model-957b33-kserve-5888c7bc5-tb6t5" podUID="392beb71-3497-4141-b8cb-5485f7b9c238" containerName="main" containerID="cri-o://f1a6c614119a4cdfb200ca40945310bd36d390881fa844f82d59ae6cbe52f7c7" gracePeriod=45 Nov 03 03:12:58.407413 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:58.407375 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-957b33/sub-test-model-957b33-kserve-5888c7bc5-tb6t5" podUID="392beb71-3497-4141-b8cb-5485f7b9c238" containerName="main" probeResult="failure" output="Get \"https://10.134.1.9:8000/ready\": EOF" Nov 03 03:12:58.646411 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:58.646382 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-957b33/sub-test-model-957b33-kserve-5888c7bc5-tb6t5" Nov 03 03:12:58.725346 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:58.725305 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/392beb71-3497-4141-b8cb-5485f7b9c238-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/392beb71-3497-4141-b8cb-5485f7b9c238-tmp-dir\") pod \"392beb71-3497-4141-b8cb-5485f7b9c238\" (UID: \"392beb71-3497-4141-b8cb-5485f7b9c238\") " Nov 03 03:12:58.725586 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:58.725396 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/392beb71-3497-4141-b8cb-5485f7b9c238-home\" (UniqueName: \"kubernetes.io/empty-dir/392beb71-3497-4141-b8cb-5485f7b9c238-home\") pod \"392beb71-3497-4141-b8cb-5485f7b9c238\" (UID: \"392beb71-3497-4141-b8cb-5485f7b9c238\") " Nov 03 03:12:58.725586 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:58.725415 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/392beb71-3497-4141-b8cb-5485f7b9c238-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/392beb71-3497-4141-b8cb-5485f7b9c238-model-cache\") pod \"392beb71-3497-4141-b8cb-5485f7b9c238\" (UID: \"392beb71-3497-4141-b8cb-5485f7b9c238\") " Nov 03 03:12:58.725586 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:58.725433 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/392beb71-3497-4141-b8cb-5485f7b9c238-dshm\" (UniqueName: \"kubernetes.io/empty-dir/392beb71-3497-4141-b8cb-5485f7b9c238-dshm\") pod \"392beb71-3497-4141-b8cb-5485f7b9c238\" (UID: \"392beb71-3497-4141-b8cb-5485f7b9c238\") " Nov 03 03:12:58.725586 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:58.725479 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/392beb71-3497-4141-b8cb-5485f7b9c238-tls-certs\" (UniqueName: \"kubernetes.io/secret/392beb71-3497-4141-b8cb-5485f7b9c238-tls-certs\") pod \"392beb71-3497-4141-b8cb-5485f7b9c238\" (UID: \"392beb71-3497-4141-b8cb-5485f7b9c238\") " Nov 03 03:12:58.725586 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:58.725521 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/392beb71-3497-4141-b8cb-5485f7b9c238-kube-api-access-lk9qv\" (UniqueName: \"kubernetes.io/projected/392beb71-3497-4141-b8cb-5485f7b9c238-kube-api-access-lk9qv\") pod \"392beb71-3497-4141-b8cb-5485f7b9c238\" (UID: \"392beb71-3497-4141-b8cb-5485f7b9c238\") " Nov 03 03:12:58.725848 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:58.725679 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/392beb71-3497-4141-b8cb-5485f7b9c238-tmp-dir" pod "392beb71-3497-4141-b8cb-5485f7b9c238" (UID: "392beb71-3497-4141-b8cb-5485f7b9c238"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:12:58.725848 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:58.725739 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/392beb71-3497-4141-b8cb-5485f7b9c238-home" pod "392beb71-3497-4141-b8cb-5485f7b9c238" (UID: "392beb71-3497-4141-b8cb-5485f7b9c238"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:12:58.725848 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:58.725756 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/392beb71-3497-4141-b8cb-5485f7b9c238-model-cache" pod "392beb71-3497-4141-b8cb-5485f7b9c238" (UID: "392beb71-3497-4141-b8cb-5485f7b9c238"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:12:58.726008 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:58.725887 5605 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/392beb71-3497-4141-b8cb-5485f7b9c238-tmp-dir\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:58.726008 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:58.725906 5605 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/392beb71-3497-4141-b8cb-5485f7b9c238-home\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:58.726008 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:58.725916 5605 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/392beb71-3497-4141-b8cb-5485f7b9c238-model-cache\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:58.727732 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:58.727701 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/392beb71-3497-4141-b8cb-5485f7b9c238-tls-certs" pod "392beb71-3497-4141-b8cb-5485f7b9c238" (UID: "392beb71-3497-4141-b8cb-5485f7b9c238"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:12:58.727850 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:58.727788 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/392beb71-3497-4141-b8cb-5485f7b9c238-dshm" pod "392beb71-3497-4141-b8cb-5485f7b9c238" (UID: "392beb71-3497-4141-b8cb-5485f7b9c238"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:12:58.727850 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:58.727811 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/392beb71-3497-4141-b8cb-5485f7b9c238-kube-api-access-lk9qv" pod "392beb71-3497-4141-b8cb-5485f7b9c238" (UID: "392beb71-3497-4141-b8cb-5485f7b9c238"). InnerVolumeSpecName "kube-api-access-lk9qv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:12:58.827244 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:58.827210 5605 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/392beb71-3497-4141-b8cb-5485f7b9c238-dshm\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:58.827244 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:58.827238 5605 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/392beb71-3497-4141-b8cb-5485f7b9c238-tls-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:58.827244 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:58.827250 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk9qv\" (UniqueName: \"kubernetes.io/projected/392beb71-3497-4141-b8cb-5485f7b9c238-kube-api-access-lk9qv\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:58.832926 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:58.832892 5605 generic.go:356] "Generic (PLEG): container finished" podID="392beb71-3497-4141-b8cb-5485f7b9c238" containerID="f1a6c614119a4cdfb200ca40945310bd36d390881fa844f82d59ae6cbe52f7c7" exitCode=0 Nov 03 03:12:58.833089 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:58.832944 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-957b33/sub-test-model-957b33-kserve-5888c7bc5-tb6t5" Nov 03 03:12:58.833089 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:58.832972 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-957b33/sub-test-model-957b33-kserve-5888c7bc5-tb6t5" event={"ID":"392beb71-3497-4141-b8cb-5485f7b9c238","Type":"ContainerDied","Data":"f1a6c614119a4cdfb200ca40945310bd36d390881fa844f82d59ae6cbe52f7c7"} Nov 03 03:12:58.833089 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:58.833017 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-957b33/sub-test-model-957b33-kserve-5888c7bc5-tb6t5" event={"ID":"392beb71-3497-4141-b8cb-5485f7b9c238","Type":"ContainerDied","Data":"3e5a4450e37aa2a247de72efa90f2cee2f435ac3db01b64ffd0b20be9e1d86bd"} Nov 03 03:12:58.833089 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:58.833041 5605 scope.go:122] "RemoveContainer" containerID="f1a6c614119a4cdfb200ca40945310bd36d390881fa844f82d59ae6cbe52f7c7" Nov 03 03:12:58.834806 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:58.834774 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdfaf0b9-6n8kr" event={"ID":"8b106f86-5a55-4614-9373-6a3f898d369f","Type":"ContainerStarted","Data":"eb9795ec9e0abd928d836f52258bccb337c9caee3e1839e2bc25ea997f931035"} Nov 03 03:12:58.834969 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:58.834826 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdfaf0b9-6n8kr" event={"ID":"8b106f86-5a55-4614-9373-6a3f898d369f","Type":"ContainerStarted","Data":"ba94d1dc57575c4c53593b9bf764eaebae6a7efd901e37416f1f7b7125b84262"} Nov 03 03:12:58.843547 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:58.843528 5605 scope.go:122] "RemoveContainer" containerID="f1a6c614119a4cdfb200ca40945310bd36d390881fa844f82d59ae6cbe52f7c7" Nov 03 03:12:58.843857 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:12:58.843832 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1a6c614119a4cdfb200ca40945310bd36d390881fa844f82d59ae6cbe52f7c7\": container with ID starting with f1a6c614119a4cdfb200ca40945310bd36d390881fa844f82d59ae6cbe52f7c7 not found: ID does not exist" containerID="f1a6c614119a4cdfb200ca40945310bd36d390881fa844f82d59ae6cbe52f7c7" Nov 03 03:12:58.843920 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:58.843872 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1a6c614119a4cdfb200ca40945310bd36d390881fa844f82d59ae6cbe52f7c7"} err="failed to get container status \"f1a6c614119a4cdfb200ca40945310bd36d390881fa844f82d59ae6cbe52f7c7\": rpc error: code = NotFound desc = could not find container \"f1a6c614119a4cdfb200ca40945310bd36d390881fa844f82d59ae6cbe52f7c7\": container with ID starting with f1a6c614119a4cdfb200ca40945310bd36d390881fa844f82d59ae6cbe52f7c7 not found: ID does not exist" Nov 03 03:12:58.848812 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:58.848789 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8282371b-c42d-43ab-a5c9-e07f2464dcdc" path="/var/lib/kubelet/pods/8282371b-c42d-43ab-a5c9-e07f2464dcdc/volumes" Nov 03 03:12:58.849199 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:58.849187 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9173364c-14ba-4287-8587-45f25118e3e4" path="/var/lib/kubelet/pods/9173364c-14ba-4287-8587-45f25118e3e4/volumes" Nov 03 03:12:58.992857 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:58.992801 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdfaf0b9-6n8kr" podStartSLOduration=1.992786814 podStartE2EDuration="1.992786814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:12:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:12:58.990630085 +0000 UTC m=+4856.249828240" watchObservedRunningTime="2026-11-03 03:12:58.992786814 +0000 UTC m=+4856.251984968" Nov 03 03:12:59.095815 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:59.095735 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-957b33/sub-test-model-957b33-kserve-5888c7bc5-tb6t5"] Nov 03 03:12:59.101243 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:59.101214 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-957b33/sub-test-model-957b33-kserve-5888c7bc5-tb6t5"] Nov 03 03:12:59.842142 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:59.842103 5605 generic.go:356] "Generic (PLEG): container finished" podID="8b106f86-5a55-4614-9373-6a3f898d369f" containerID="eb9795ec9e0abd928d836f52258bccb337c9caee3e1839e2bc25ea997f931035" exitCode=0 Nov 03 03:12:59.842614 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:12:59.842189 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdfaf0b9-6n8kr" event={"ID":"8b106f86-5a55-4614-9373-6a3f898d369f","Type":"ContainerDied","Data":"eb9795ec9e0abd928d836f52258bccb337c9caee3e1839e2bc25ea997f931035"} Nov 03 03:13:00.848874 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:00.848837 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="392beb71-3497-4141-b8cb-5485f7b9c238" path="/var/lib/kubelet/pods/392beb71-3497-4141-b8cb-5485f7b9c238/volumes" Nov 03 03:13:00.980145 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:00.980116 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdfaf0b9-6n8kr" Nov 03 03:13:01.080195 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:01.080165 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1243713b-skfvd"] Nov 03 03:13:01.082166 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:01.082143 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1243713b-skfvd"] Nov 03 03:13:01.148391 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:01.148286 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b106f86-5a55-4614-9373-6a3f898d369f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b106f86-5a55-4614-9373-6a3f898d369f-maas-api-openshift-ca\") pod \"8b106f86-5a55-4614-9373-6a3f898d369f\" (UID: \"8b106f86-5a55-4614-9373-6a3f898d369f\") " Nov 03 03:13:01.148547 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:01.148439 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b106f86-5a55-4614-9373-6a3f898d369f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b106f86-5a55-4614-9373-6a3f898d369f-maas-api-xks-ca\") pod \"8b106f86-5a55-4614-9373-6a3f898d369f\" (UID: \"8b106f86-5a55-4614-9373-6a3f898d369f\") " Nov 03 03:13:01.148547 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:01.148487 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b106f86-5a55-4614-9373-6a3f898d369f-kube-api-access-ndp6p\" (UniqueName: \"kubernetes.io/projected/8b106f86-5a55-4614-9373-6a3f898d369f-kube-api-access-ndp6p\") pod \"8b106f86-5a55-4614-9373-6a3f898d369f\" (UID: \"8b106f86-5a55-4614-9373-6a3f898d369f\") " Nov 03 03:13:01.148732 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:01.148701 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b106f86-5a55-4614-9373-6a3f898d369f-maas-api-openshift-ca" pod "8b106f86-5a55-4614-9373-6a3f898d369f" (UID: "8b106f86-5a55-4614-9373-6a3f898d369f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:13:01.150581 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:01.150557 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b106f86-5a55-4614-9373-6a3f898d369f-maas-api-xks-ca" pod "8b106f86-5a55-4614-9373-6a3f898d369f" (UID: "8b106f86-5a55-4614-9373-6a3f898d369f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:13:01.150691 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:01.150647 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b106f86-5a55-4614-9373-6a3f898d369f-kube-api-access-ndp6p" pod "8b106f86-5a55-4614-9373-6a3f898d369f" (UID: "8b106f86-5a55-4614-9373-6a3f898d369f"). InnerVolumeSpecName "kube-api-access-ndp6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:13:01.250109 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:01.250067 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b106f86-5a55-4614-9373-6a3f898d369f-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:01.250109 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:01.250101 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndp6p\" (UniqueName: \"kubernetes.io/projected/8b106f86-5a55-4614-9373-6a3f898d369f-kube-api-access-ndp6p\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:01.250109 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:01.250117 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b106f86-5a55-4614-9373-6a3f898d369f-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:01.564673 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:01.564629 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-1bccbd/distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg"] Nov 03 03:13:01.569523 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:01.569491 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-1bccbd/distinct-w1-1bccbd-kserve-5dfc85555f-tkvg6"] Nov 03 03:13:01.854194 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:01.854115 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdfaf0b9-6n8kr" Nov 03 03:13:01.854194 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:01.854145 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdfaf0b9-6n8kr" event={"ID":"8b106f86-5a55-4614-9373-6a3f898d369f","Type":"ContainerDied","Data":"ba94d1dc57575c4c53593b9bf764eaebae6a7efd901e37416f1f7b7125b84262"} Nov 03 03:13:01.854194 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:01.854190 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba94d1dc57575c4c53593b9bf764eaebae6a7efd901e37416f1f7b7125b84262" Nov 03 03:13:02.850086 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:02.850052 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1439cf8d-1d92-42ba-80ca-eb5c781fa883" path="/var/lib/kubelet/pods/1439cf8d-1d92-42ba-80ca-eb5c781fa883/volumes" Nov 03 03:13:03.052838 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:03.052801 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:13:05.347414 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:05.347379 5605 scope.go:122] "RemoveContainer" containerID="4d38a44a7076ec4c8b85134bab414ba905ad81e0f639a1bdd3ff20265224513e" Nov 03 03:13:12.048337 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:12.048253 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dc834fd-hh4fr"] Nov 03 03:13:12.095216 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:12.095179 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dc834fd-hh4fr"] Nov 03 03:13:12.850037 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:12.850002 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8a5223d-e4dd-4c99-8042-825c94b85ba4" path="/var/lib/kubelet/pods/e8a5223d-e4dd-4c99-8042-825c94b85ba4/volumes" Nov 03 03:13:13.141531 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:13.141446 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6406d0c-gv425"] Nov 03 03:13:13.141911 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:13.141796 5605 cpu_manager.go:414] "containerMap: removing container" podUID="8b106f86-5a55-4614-9373-6a3f898d369f" containerName="revoke-keys" Nov 03 03:13:13.141911 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:13.141807 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b106f86-5a55-4614-9373-6a3f898d369f" containerName="revoke-keys" Nov 03 03:13:13.141911 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:13.141834 5605 cpu_manager.go:414] "containerMap: removing container" podUID="392beb71-3497-4141-b8cb-5485f7b9c238" containerName="main" Nov 03 03:13:13.141911 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:13.141839 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="392beb71-3497-4141-b8cb-5485f7b9c238" containerName="main" Nov 03 03:13:13.141911 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:13.141893 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b106f86-5a55-4614-9373-6a3f898d369f" containerName="revoke-keys" Nov 03 03:13:13.141911 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:13.141908 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="392beb71-3497-4141-b8cb-5485f7b9c238" containerName="main" Nov 03 03:13:13.144965 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:13.144946 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6406d0c-gv425" Nov 03 03:13:13.160436 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:13.160405 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55eeee4d-8327-4e10-836e-d6609ad111a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6406d0c-gv425\" (UID: \"55eeee4d-8327-4e10-836e-d6609ad111a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6406d0c-gv425" Nov 03 03:13:13.160566 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:13.160483 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55eeee4d-8327-4e10-836e-d6609ad111a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6406d0c-gv425\" (UID: \"55eeee4d-8327-4e10-836e-d6609ad111a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6406d0c-gv425" Nov 03 03:13:13.160639 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:13.160583 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtm2q\" (UniqueName: \"kubernetes.io/projected/55eeee4d-8327-4e10-836e-d6609ad111a3-kube-api-access-wtm2q\") pod \"maas-api-revoke-sub-b6406d0c-gv425\" (UID: \"55eeee4d-8327-4e10-836e-d6609ad111a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6406d0c-gv425" Nov 03 03:13:13.202727 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:13.202693 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:13:13.202727 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:13.202718 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:13:13.242701 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:13.242666 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 03:13:13.242701 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:13.242668 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:13:13.244703 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:13.244683 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6406d0c-gv425"] Nov 03 03:13:13.262099 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:13.261293 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtm2q\" (UniqueName: \"kubernetes.io/projected/55eeee4d-8327-4e10-836e-d6609ad111a3-kube-api-access-wtm2q\") pod \"maas-api-revoke-sub-b6406d0c-gv425\" (UID: \"55eeee4d-8327-4e10-836e-d6609ad111a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6406d0c-gv425" Nov 03 03:13:13.262099 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:13.261399 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55eeee4d-8327-4e10-836e-d6609ad111a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6406d0c-gv425\" (UID: \"55eeee4d-8327-4e10-836e-d6609ad111a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6406d0c-gv425" Nov 03 03:13:13.262099 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:13.261517 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55eeee4d-8327-4e10-836e-d6609ad111a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6406d0c-gv425\" (UID: \"55eeee4d-8327-4e10-836e-d6609ad111a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6406d0c-gv425" Nov 03 03:13:13.262678 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:13.262427 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55eeee4d-8327-4e10-836e-d6609ad111a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6406d0c-gv425\" (UID: \"55eeee4d-8327-4e10-836e-d6609ad111a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6406d0c-gv425" Nov 03 03:13:13.265795 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:13.265767 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55eeee4d-8327-4e10-836e-d6609ad111a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6406d0c-gv425\" (UID: \"55eeee4d-8327-4e10-836e-d6609ad111a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6406d0c-gv425" Nov 03 03:13:13.314866 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:13.314832 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtm2q\" (UniqueName: \"kubernetes.io/projected/55eeee4d-8327-4e10-836e-d6609ad111a3-kube-api-access-wtm2q\") pod \"maas-api-revoke-sub-b6406d0c-gv425\" (UID: \"55eeee4d-8327-4e10-836e-d6609ad111a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6406d0c-gv425" Nov 03 03:13:13.457678 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:13.457601 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6406d0c-gv425" Nov 03 03:13:13.824809 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:13.824775 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6406d0c-gv425"] Nov 03 03:13:13.831066 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:13:13.831034 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55eeee4d_8327_4e10_836e_d6609ad111a3.slice/crio-606a53450e41631cf05ac7f636e7656728371333c312a62feea690e80153127d WatchSource:0}: Error finding container 606a53450e41631cf05ac7f636e7656728371333c312a62feea690e80153127d: Status 404 returned error can't find the container with id 606a53450e41631cf05ac7f636e7656728371333c312a62feea690e80153127d Nov 03 03:13:13.916261 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:13.916230 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6406d0c-gv425" event={"ID":"55eeee4d-8327-4e10-836e-d6609ad111a3","Type":"ContainerStarted","Data":"606a53450e41631cf05ac7f636e7656728371333c312a62feea690e80153127d"} Nov 03 03:13:14.923087 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:14.923053 5605 generic.go:356] "Generic (PLEG): container finished" podID="55eeee4d-8327-4e10-836e-d6609ad111a3" containerID="8190cd1b0cfee93a5c3d708ca74e0325b844b6f74ba256463212bdc25e072524" exitCode=0 Nov 03 03:13:14.923518 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:14.923138 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6406d0c-gv425" event={"ID":"55eeee4d-8327-4e10-836e-d6609ad111a3","Type":"ContainerDied","Data":"8190cd1b0cfee93a5c3d708ca74e0325b844b6f74ba256463212bdc25e072524"} Nov 03 03:13:16.067196 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.067170 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6406d0c-gv425" Nov 03 03:13:16.086876 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.086850 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55eeee4d-8327-4e10-836e-d6609ad111a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55eeee4d-8327-4e10-836e-d6609ad111a3-maas-api-xks-ca\") pod \"55eeee4d-8327-4e10-836e-d6609ad111a3\" (UID: \"55eeee4d-8327-4e10-836e-d6609ad111a3\") " Nov 03 03:13:16.087026 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.086913 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55eeee4d-8327-4e10-836e-d6609ad111a3-kube-api-access-wtm2q\" (UniqueName: \"kubernetes.io/projected/55eeee4d-8327-4e10-836e-d6609ad111a3-kube-api-access-wtm2q\") pod \"55eeee4d-8327-4e10-836e-d6609ad111a3\" (UID: \"55eeee4d-8327-4e10-836e-d6609ad111a3\") " Nov 03 03:13:16.087026 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.086978 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55eeee4d-8327-4e10-836e-d6609ad111a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55eeee4d-8327-4e10-836e-d6609ad111a3-maas-api-openshift-ca\") pod \"55eeee4d-8327-4e10-836e-d6609ad111a3\" (UID: \"55eeee4d-8327-4e10-836e-d6609ad111a3\") " Nov 03 03:13:16.087528 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.087465 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55eeee4d-8327-4e10-836e-d6609ad111a3-maas-api-openshift-ca" pod "55eeee4d-8327-4e10-836e-d6609ad111a3" (UID: "55eeee4d-8327-4e10-836e-d6609ad111a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:13:16.089052 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.089025 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55eeee4d-8327-4e10-836e-d6609ad111a3-maas-api-xks-ca" pod "55eeee4d-8327-4e10-836e-d6609ad111a3" (UID: "55eeee4d-8327-4e10-836e-d6609ad111a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:13:16.089158 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.089091 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55eeee4d-8327-4e10-836e-d6609ad111a3-kube-api-access-wtm2q" pod "55eeee4d-8327-4e10-836e-d6609ad111a3" (UID: "55eeee4d-8327-4e10-836e-d6609ad111a3"). InnerVolumeSpecName "kube-api-access-wtm2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:13:16.188063 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.187994 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtm2q\" (UniqueName: \"kubernetes.io/projected/55eeee4d-8327-4e10-836e-d6609ad111a3-kube-api-access-wtm2q\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:16.188063 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.188021 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55eeee4d-8327-4e10-836e-d6609ad111a3-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:16.188063 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.188033 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55eeee4d-8327-4e10-836e-d6609ad111a3-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:16.584990 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.584946 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-1bccbd/distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg" podUID="5a71cdee-425e-4787-9217-5fb6b97e232b" containerName="main" containerID="cri-o://f0d955ba3038db02d8d86da70f8cc3f3c58807b67f406e3e41f5f6cde9f182fb" gracePeriod=45 Nov 03 03:13:16.588554 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.588513 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-1bccbd/distinct-w1-1bccbd-kserve-5dfc85555f-tkvg6" podUID="dd4a9d44-c63e-4f67-bcac-a4f98ecf1796" containerName="main" containerID="cri-o://55a2db3860d5cfe05593d9fc454af1847915448d03c9d747e72ae4acaa9a4c32" gracePeriod=45 Nov 03 03:13:16.833573 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.833546 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1bccbd/distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg" Nov 03 03:13:16.840339 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.840286 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1bccbd/distinct-w1-1bccbd-kserve-5dfc85555f-tkvg6" Nov 03 03:13:16.894837 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.894810 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd4a9d44-c63e-4f67-bcac-a4f98ecf1796-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd4a9d44-c63e-4f67-bcac-a4f98ecf1796-model-cache\") pod \"dd4a9d44-c63e-4f67-bcac-a4f98ecf1796\" (UID: \"dd4a9d44-c63e-4f67-bcac-a4f98ecf1796\") " Nov 03 03:13:16.894961 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.894846 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd4a9d44-c63e-4f67-bcac-a4f98ecf1796-kube-api-access-cfg2z\" (UniqueName: \"kubernetes.io/projected/dd4a9d44-c63e-4f67-bcac-a4f98ecf1796-kube-api-access-cfg2z\") pod \"dd4a9d44-c63e-4f67-bcac-a4f98ecf1796\" (UID: \"dd4a9d44-c63e-4f67-bcac-a4f98ecf1796\") " Nov 03 03:13:16.894961 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.894876 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a71cdee-425e-4787-9217-5fb6b97e232b-tls-certs\" (UniqueName: \"kubernetes.io/secret/5a71cdee-425e-4787-9217-5fb6b97e232b-tls-certs\") pod \"5a71cdee-425e-4787-9217-5fb6b97e232b\" (UID: \"5a71cdee-425e-4787-9217-5fb6b97e232b\") " Nov 03 03:13:16.894961 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.894899 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd4a9d44-c63e-4f67-bcac-a4f98ecf1796-home\" (UniqueName: \"kubernetes.io/empty-dir/dd4a9d44-c63e-4f67-bcac-a4f98ecf1796-home\") pod \"dd4a9d44-c63e-4f67-bcac-a4f98ecf1796\" (UID: \"dd4a9d44-c63e-4f67-bcac-a4f98ecf1796\") " Nov 03 03:13:16.894961 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.894943 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd4a9d44-c63e-4f67-bcac-a4f98ecf1796-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd4a9d44-c63e-4f67-bcac-a4f98ecf1796-dshm\") pod \"dd4a9d44-c63e-4f67-bcac-a4f98ecf1796\" (UID: \"dd4a9d44-c63e-4f67-bcac-a4f98ecf1796\") " Nov 03 03:13:16.894961 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.894962 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a71cdee-425e-4787-9217-5fb6b97e232b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a71cdee-425e-4787-9217-5fb6b97e232b-dshm\") pod \"5a71cdee-425e-4787-9217-5fb6b97e232b\" (UID: \"5a71cdee-425e-4787-9217-5fb6b97e232b\") " Nov 03 03:13:16.895263 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.895000 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a71cdee-425e-4787-9217-5fb6b97e232b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a71cdee-425e-4787-9217-5fb6b97e232b-tmp-dir\") pod \"5a71cdee-425e-4787-9217-5fb6b97e232b\" (UID: \"5a71cdee-425e-4787-9217-5fb6b97e232b\") " Nov 03 03:13:16.895263 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.895062 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a71cdee-425e-4787-9217-5fb6b97e232b-kube-api-access-gdcmr\" (UniqueName: \"kubernetes.io/projected/5a71cdee-425e-4787-9217-5fb6b97e232b-kube-api-access-gdcmr\") pod \"5a71cdee-425e-4787-9217-5fb6b97e232b\" (UID: \"5a71cdee-425e-4787-9217-5fb6b97e232b\") " Nov 03 03:13:16.895263 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.895141 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd4a9d44-c63e-4f67-bcac-a4f98ecf1796-model-cache" pod "dd4a9d44-c63e-4f67-bcac-a4f98ecf1796" (UID: "dd4a9d44-c63e-4f67-bcac-a4f98ecf1796"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:13:16.895263 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.895150 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a71cdee-425e-4787-9217-5fb6b97e232b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a71cdee-425e-4787-9217-5fb6b97e232b-model-cache\") pod \"5a71cdee-425e-4787-9217-5fb6b97e232b\" (UID: \"5a71cdee-425e-4787-9217-5fb6b97e232b\") " Nov 03 03:13:16.895263 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.895192 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd4a9d44-c63e-4f67-bcac-a4f98ecf1796-home" pod "dd4a9d44-c63e-4f67-bcac-a4f98ecf1796" (UID: "dd4a9d44-c63e-4f67-bcac-a4f98ecf1796"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:13:16.895263 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.895210 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd4a9d44-c63e-4f67-bcac-a4f98ecf1796-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd4a9d44-c63e-4f67-bcac-a4f98ecf1796-tmp-dir\") pod \"dd4a9d44-c63e-4f67-bcac-a4f98ecf1796\" (UID: \"dd4a9d44-c63e-4f67-bcac-a4f98ecf1796\") " Nov 03 03:13:16.895263 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.895261 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a71cdee-425e-4787-9217-5fb6b97e232b-home\" (UniqueName: \"kubernetes.io/empty-dir/5a71cdee-425e-4787-9217-5fb6b97e232b-home\") pod \"5a71cdee-425e-4787-9217-5fb6b97e232b\" (UID: \"5a71cdee-425e-4787-9217-5fb6b97e232b\") " Nov 03 03:13:16.895693 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.895327 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a71cdee-425e-4787-9217-5fb6b97e232b-model-cache" pod "5a71cdee-425e-4787-9217-5fb6b97e232b" (UID: "5a71cdee-425e-4787-9217-5fb6b97e232b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:13:16.895693 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.895331 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd4a9d44-c63e-4f67-bcac-a4f98ecf1796-tls-certs\" (UniqueName: \"kubernetes.io/secret/dd4a9d44-c63e-4f67-bcac-a4f98ecf1796-tls-certs\") pod \"dd4a9d44-c63e-4f67-bcac-a4f98ecf1796\" (UID: \"dd4a9d44-c63e-4f67-bcac-a4f98ecf1796\") " Nov 03 03:13:16.895693 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.895346 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a71cdee-425e-4787-9217-5fb6b97e232b-tmp-dir" pod "5a71cdee-425e-4787-9217-5fb6b97e232b" (UID: "5a71cdee-425e-4787-9217-5fb6b97e232b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:13:16.895693 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.895573 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd4a9d44-c63e-4f67-bcac-a4f98ecf1796-tmp-dir" pod "dd4a9d44-c63e-4f67-bcac-a4f98ecf1796" (UID: "dd4a9d44-c63e-4f67-bcac-a4f98ecf1796"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:13:16.895924 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.895719 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a71cdee-425e-4787-9217-5fb6b97e232b-home" pod "5a71cdee-425e-4787-9217-5fb6b97e232b" (UID: "5a71cdee-425e-4787-9217-5fb6b97e232b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:13:16.895924 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.895785 5605 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a71cdee-425e-4787-9217-5fb6b97e232b-tmp-dir\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:16.895924 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.895806 5605 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a71cdee-425e-4787-9217-5fb6b97e232b-model-cache\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:16.895924 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.895822 5605 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd4a9d44-c63e-4f67-bcac-a4f98ecf1796-tmp-dir\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:16.895924 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.895838 5605 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd4a9d44-c63e-4f67-bcac-a4f98ecf1796-model-cache\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:16.895924 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.895851 5605 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd4a9d44-c63e-4f67-bcac-a4f98ecf1796-home\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:16.897537 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.897516 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd4a9d44-c63e-4f67-bcac-a4f98ecf1796-tls-certs" pod "dd4a9d44-c63e-4f67-bcac-a4f98ecf1796" (UID: "dd4a9d44-c63e-4f67-bcac-a4f98ecf1796"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:13:16.897643 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.897561 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a71cdee-425e-4787-9217-5fb6b97e232b-tls-certs" pod "5a71cdee-425e-4787-9217-5fb6b97e232b" (UID: "5a71cdee-425e-4787-9217-5fb6b97e232b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:13:16.897643 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.897611 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd4a9d44-c63e-4f67-bcac-a4f98ecf1796-dshm" pod "dd4a9d44-c63e-4f67-bcac-a4f98ecf1796" (UID: "dd4a9d44-c63e-4f67-bcac-a4f98ecf1796"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:13:16.897733 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.897671 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a71cdee-425e-4787-9217-5fb6b97e232b-dshm" pod "5a71cdee-425e-4787-9217-5fb6b97e232b" (UID: "5a71cdee-425e-4787-9217-5fb6b97e232b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:13:16.897733 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.897680 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd4a9d44-c63e-4f67-bcac-a4f98ecf1796-kube-api-access-cfg2z" pod "dd4a9d44-c63e-4f67-bcac-a4f98ecf1796" (UID: "dd4a9d44-c63e-4f67-bcac-a4f98ecf1796"). InnerVolumeSpecName "kube-api-access-cfg2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:13:16.898020 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.897999 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a71cdee-425e-4787-9217-5fb6b97e232b-kube-api-access-gdcmr" pod "5a71cdee-425e-4787-9217-5fb6b97e232b" (UID: "5a71cdee-425e-4787-9217-5fb6b97e232b"). InnerVolumeSpecName "kube-api-access-gdcmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:13:16.934156 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.934126 5605 generic.go:356] "Generic (PLEG): container finished" podID="dd4a9d44-c63e-4f67-bcac-a4f98ecf1796" containerID="55a2db3860d5cfe05593d9fc454af1847915448d03c9d747e72ae4acaa9a4c32" exitCode=0 Nov 03 03:13:16.934300 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.934173 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1bccbd/distinct-w1-1bccbd-kserve-5dfc85555f-tkvg6" Nov 03 03:13:16.934300 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.934211 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1bccbd/distinct-w1-1bccbd-kserve-5dfc85555f-tkvg6" event={"ID":"dd4a9d44-c63e-4f67-bcac-a4f98ecf1796","Type":"ContainerDied","Data":"55a2db3860d5cfe05593d9fc454af1847915448d03c9d747e72ae4acaa9a4c32"} Nov 03 03:13:16.934300 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.934259 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1bccbd/distinct-w1-1bccbd-kserve-5dfc85555f-tkvg6" event={"ID":"dd4a9d44-c63e-4f67-bcac-a4f98ecf1796","Type":"ContainerDied","Data":"b5078dae62b1d5c8cbe557dfb99d179b013b733b96a9a2e2fb3ac7a3bb99cb7b"} Nov 03 03:13:16.934300 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.934281 5605 scope.go:122] "RemoveContainer" containerID="55a2db3860d5cfe05593d9fc454af1847915448d03c9d747e72ae4acaa9a4c32" Nov 03 03:13:16.935635 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.935590 5605 generic.go:356] "Generic (PLEG): container finished" podID="5a71cdee-425e-4787-9217-5fb6b97e232b" containerID="f0d955ba3038db02d8d86da70f8cc3f3c58807b67f406e3e41f5f6cde9f182fb" exitCode=0 Nov 03 03:13:16.935707 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.935639 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1bccbd/distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg" Nov 03 03:13:16.935707 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.935633 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1bccbd/distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg" event={"ID":"5a71cdee-425e-4787-9217-5fb6b97e232b","Type":"ContainerDied","Data":"f0d955ba3038db02d8d86da70f8cc3f3c58807b67f406e3e41f5f6cde9f182fb"} Nov 03 03:13:16.935707 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.935675 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1bccbd/distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg" event={"ID":"5a71cdee-425e-4787-9217-5fb6b97e232b","Type":"ContainerDied","Data":"8c7389012f8a44ee77bb2d25aa0f46430e56462c717d1e476f0697bbcdb2f2c0"} Nov 03 03:13:16.937332 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.937313 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6406d0c-gv425" Nov 03 03:13:16.937461 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.937316 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6406d0c-gv425" event={"ID":"55eeee4d-8327-4e10-836e-d6609ad111a3","Type":"ContainerDied","Data":"606a53450e41631cf05ac7f636e7656728371333c312a62feea690e80153127d"} Nov 03 03:13:16.937461 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.937411 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="606a53450e41631cf05ac7f636e7656728371333c312a62feea690e80153127d" Nov 03 03:13:16.948773 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.947710 5605 scope.go:122] "RemoveContainer" containerID="55a2db3860d5cfe05593d9fc454af1847915448d03c9d747e72ae4acaa9a4c32" Nov 03 03:13:16.948773 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:13:16.948064 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55a2db3860d5cfe05593d9fc454af1847915448d03c9d747e72ae4acaa9a4c32\": container with ID starting with 55a2db3860d5cfe05593d9fc454af1847915448d03c9d747e72ae4acaa9a4c32 not found: ID does not exist" containerID="55a2db3860d5cfe05593d9fc454af1847915448d03c9d747e72ae4acaa9a4c32" Nov 03 03:13:16.948773 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.948093 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55a2db3860d5cfe05593d9fc454af1847915448d03c9d747e72ae4acaa9a4c32"} err="failed to get container status \"55a2db3860d5cfe05593d9fc454af1847915448d03c9d747e72ae4acaa9a4c32\": rpc error: code = NotFound desc = could not find container \"55a2db3860d5cfe05593d9fc454af1847915448d03c9d747e72ae4acaa9a4c32\": container with ID starting with 55a2db3860d5cfe05593d9fc454af1847915448d03c9d747e72ae4acaa9a4c32 not found: ID does not exist" Nov 03 03:13:16.948773 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.948115 5605 scope.go:122] "RemoveContainer" containerID="f0d955ba3038db02d8d86da70f8cc3f3c58807b67f406e3e41f5f6cde9f182fb" Nov 03 03:13:16.960523 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.960501 5605 scope.go:122] "RemoveContainer" containerID="f0d955ba3038db02d8d86da70f8cc3f3c58807b67f406e3e41f5f6cde9f182fb" Nov 03 03:13:16.960805 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:13:16.960787 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0d955ba3038db02d8d86da70f8cc3f3c58807b67f406e3e41f5f6cde9f182fb\": container with ID starting with f0d955ba3038db02d8d86da70f8cc3f3c58807b67f406e3e41f5f6cde9f182fb not found: ID does not exist" containerID="f0d955ba3038db02d8d86da70f8cc3f3c58807b67f406e3e41f5f6cde9f182fb" Nov 03 03:13:16.960860 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.960814 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0d955ba3038db02d8d86da70f8cc3f3c58807b67f406e3e41f5f6cde9f182fb"} err="failed to get container status \"f0d955ba3038db02d8d86da70f8cc3f3c58807b67f406e3e41f5f6cde9f182fb\": rpc error: code = NotFound desc = could not find container \"f0d955ba3038db02d8d86da70f8cc3f3c58807b67f406e3e41f5f6cde9f182fb\": container with ID starting with f0d955ba3038db02d8d86da70f8cc3f3c58807b67f406e3e41f5f6cde9f182fb not found: ID does not exist" Nov 03 03:13:16.996460 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.996429 5605 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a71cdee-425e-4787-9217-5fb6b97e232b-home\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:16.996460 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.996457 5605 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd4a9d44-c63e-4f67-bcac-a4f98ecf1796-tls-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:16.996673 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.996468 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfg2z\" (UniqueName: \"kubernetes.io/projected/dd4a9d44-c63e-4f67-bcac-a4f98ecf1796-kube-api-access-cfg2z\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:16.996673 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.996478 5605 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a71cdee-425e-4787-9217-5fb6b97e232b-tls-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:16.996673 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.996487 5605 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd4a9d44-c63e-4f67-bcac-a4f98ecf1796-dshm\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:16.996673 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.996495 5605 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a71cdee-425e-4787-9217-5fb6b97e232b-dshm\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:16.996673 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:16.996506 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdcmr\" (UniqueName: \"kubernetes.io/projected/5a71cdee-425e-4787-9217-5fb6b97e232b-kube-api-access-gdcmr\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:17.299754 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:17.299727 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-1bccbd/distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg"] Nov 03 03:13:17.301777 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:17.301744 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-1bccbd/distinct-2-w1-1bccbd-kserve-5f4b77cbd-7zzkg"] Nov 03 03:13:17.403560 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:17.403529 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-1bccbd/distinct-w1-1bccbd-kserve-5dfc85555f-tkvg6"] Nov 03 03:13:17.408516 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:17.408491 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-1bccbd/distinct-w1-1bccbd-kserve-5dfc85555f-tkvg6"] Nov 03 03:13:18.041934 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:18.041898 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa18948d-pmch6"] Nov 03 03:13:18.044279 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:18.044250 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa18948d-pmch6"] Nov 03 03:13:18.848842 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:18.848813 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a71cdee-425e-4787-9217-5fb6b97e232b" path="/var/lib/kubelet/pods/5a71cdee-425e-4787-9217-5fb6b97e232b/volumes" Nov 03 03:13:18.849209 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:18.849192 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd4a9d44-c63e-4f67-bcac-a4f98ecf1796" path="/var/lib/kubelet/pods/dd4a9d44-c63e-4f67-bcac-a4f98ecf1796/volumes" Nov 03 03:13:18.849600 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:18.849585 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f58b31f9-f1f8-49c9-bd84-dfc5230ae8ea" path="/var/lib/kubelet/pods/f58b31f9-f1f8-49c9-bd84-dfc5230ae8ea/volumes" Nov 03 03:13:23.491184 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:23.491147 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt"] Nov 03 03:13:23.491738 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:23.491428 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt" podUID="f563fffd-fb25-4763-98f7-fdec204a5637" containerName="istio-proxy" containerID="cri-o://d68d37c09aa82fbbacd883e8ddc8b0f128922599bb32494745c7d246118ff3fb" gracePeriod=30 Nov 03 03:13:23.914523 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:13:23.914488 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:13:28.741829 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:28.741801 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt" Nov 03 03:13:28.811460 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:28.811422 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f563fffd-fb25-4763-98f7-fdec204a5637-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f563fffd-fb25-4763-98f7-fdec204a5637-istiod-ca-cert\") pod \"f563fffd-fb25-4763-98f7-fdec204a5637\" (UID: \"f563fffd-fb25-4763-98f7-fdec204a5637\") " Nov 03 03:13:28.811622 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:28.811488 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f563fffd-fb25-4763-98f7-fdec204a5637-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f563fffd-fb25-4763-98f7-fdec204a5637-workload-certs\") pod \"f563fffd-fb25-4763-98f7-fdec204a5637\" (UID: \"f563fffd-fb25-4763-98f7-fdec204a5637\") " Nov 03 03:13:28.811622 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:28.811554 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f563fffd-fb25-4763-98f7-fdec204a5637-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f563fffd-fb25-4763-98f7-fdec204a5637-istio-podinfo\") pod \"f563fffd-fb25-4763-98f7-fdec204a5637\" (UID: \"f563fffd-fb25-4763-98f7-fdec204a5637\") " Nov 03 03:13:28.811622 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:28.811608 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f563fffd-fb25-4763-98f7-fdec204a5637-kube-api-access-jf8ht\" (UniqueName: \"kubernetes.io/projected/f563fffd-fb25-4763-98f7-fdec204a5637-kube-api-access-jf8ht\") pod \"f563fffd-fb25-4763-98f7-fdec204a5637\" (UID: \"f563fffd-fb25-4763-98f7-fdec204a5637\") " Nov 03 03:13:28.811793 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:28.811643 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f563fffd-fb25-4763-98f7-fdec204a5637-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f563fffd-fb25-4763-98f7-fdec204a5637-istio-envoy\") pod \"f563fffd-fb25-4763-98f7-fdec204a5637\" (UID: \"f563fffd-fb25-4763-98f7-fdec204a5637\") " Nov 03 03:13:28.811793 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:28.811679 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f563fffd-fb25-4763-98f7-fdec204a5637-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f563fffd-fb25-4763-98f7-fdec204a5637-workload-socket\") pod \"f563fffd-fb25-4763-98f7-fdec204a5637\" (UID: \"f563fffd-fb25-4763-98f7-fdec204a5637\") " Nov 03 03:13:28.811793 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:28.811756 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f563fffd-fb25-4763-98f7-fdec204a5637-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f563fffd-fb25-4763-98f7-fdec204a5637-istio-data\") pod \"f563fffd-fb25-4763-98f7-fdec204a5637\" (UID: \"f563fffd-fb25-4763-98f7-fdec204a5637\") " Nov 03 03:13:28.811793 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:28.811789 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f563fffd-fb25-4763-98f7-fdec204a5637-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f563fffd-fb25-4763-98f7-fdec204a5637-credential-socket\") pod \"f563fffd-fb25-4763-98f7-fdec204a5637\" (UID: \"f563fffd-fb25-4763-98f7-fdec204a5637\") " Nov 03 03:13:28.811993 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:28.811823 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f563fffd-fb25-4763-98f7-fdec204a5637-istio-token\" (UniqueName: \"kubernetes.io/projected/f563fffd-fb25-4763-98f7-fdec204a5637-istio-token\") pod \"f563fffd-fb25-4763-98f7-fdec204a5637\" (UID: \"f563fffd-fb25-4763-98f7-fdec204a5637\") " Nov 03 03:13:28.811993 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:28.811835 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f563fffd-fb25-4763-98f7-fdec204a5637-istiod-ca-cert" pod "f563fffd-fb25-4763-98f7-fdec204a5637" (UID: "f563fffd-fb25-4763-98f7-fdec204a5637"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:13:28.811993 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:28.811842 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f563fffd-fb25-4763-98f7-fdec204a5637-workload-certs" pod "f563fffd-fb25-4763-98f7-fdec204a5637" (UID: "f563fffd-fb25-4763-98f7-fdec204a5637"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:13:28.812162 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:28.812101 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f563fffd-fb25-4763-98f7-fdec204a5637-workload-socket" pod "f563fffd-fb25-4763-98f7-fdec204a5637" (UID: "f563fffd-fb25-4763-98f7-fdec204a5637"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:13:28.812162 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:28.812117 5605 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f563fffd-fb25-4763-98f7-fdec204a5637-istiod-ca-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:28.812162 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:28.812136 5605 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f563fffd-fb25-4763-98f7-fdec204a5637-workload-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:28.812338 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:28.812178 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f563fffd-fb25-4763-98f7-fdec204a5637-istio-data" pod "f563fffd-fb25-4763-98f7-fdec204a5637" (UID: "f563fffd-fb25-4763-98f7-fdec204a5637"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:13:28.812338 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:28.812296 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f563fffd-fb25-4763-98f7-fdec204a5637-credential-socket" pod "f563fffd-fb25-4763-98f7-fdec204a5637" (UID: "f563fffd-fb25-4763-98f7-fdec204a5637"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:13:28.814007 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:28.813973 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f563fffd-fb25-4763-98f7-fdec204a5637-istio-envoy" pod "f563fffd-fb25-4763-98f7-fdec204a5637" (UID: "f563fffd-fb25-4763-98f7-fdec204a5637"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:13:28.814169 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:28.814152 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f563fffd-fb25-4763-98f7-fdec204a5637-istio-token" pod "f563fffd-fb25-4763-98f7-fdec204a5637" (UID: "f563fffd-fb25-4763-98f7-fdec204a5637"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:13:28.814297 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:28.814280 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f563fffd-fb25-4763-98f7-fdec204a5637-istio-podinfo" pod "f563fffd-fb25-4763-98f7-fdec204a5637" (UID: "f563fffd-fb25-4763-98f7-fdec204a5637"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:13:28.814397 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:28.814374 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f563fffd-fb25-4763-98f7-fdec204a5637-kube-api-access-jf8ht" pod "f563fffd-fb25-4763-98f7-fdec204a5637" (UID: "f563fffd-fb25-4763-98f7-fdec204a5637"). InnerVolumeSpecName "kube-api-access-jf8ht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:13:28.912636 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:28.912593 5605 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f563fffd-fb25-4763-98f7-fdec204a5637-istio-data\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:28.912636 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:28.912622 5605 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f563fffd-fb25-4763-98f7-fdec204a5637-credential-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:28.912636 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:28.912631 5605 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f563fffd-fb25-4763-98f7-fdec204a5637-istio-token\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:28.912636 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:28.912639 5605 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f563fffd-fb25-4763-98f7-fdec204a5637-istio-podinfo\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:28.912636 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:28.912648 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jf8ht\" (UniqueName: \"kubernetes.io/projected/f563fffd-fb25-4763-98f7-fdec204a5637-kube-api-access-jf8ht\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:28.913088 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:28.912657 5605 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f563fffd-fb25-4763-98f7-fdec204a5637-istio-envoy\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:28.913088 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:28.912668 5605 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f563fffd-fb25-4763-98f7-fdec204a5637-workload-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:29.001119 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:29.001078 5605 generic.go:356] "Generic (PLEG): container finished" podID="f563fffd-fb25-4763-98f7-fdec204a5637" containerID="d68d37c09aa82fbbacd883e8ddc8b0f128922599bb32494745c7d246118ff3fb" exitCode=0 Nov 03 03:13:29.001318 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:29.001141 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt" Nov 03 03:13:29.001318 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:29.001163 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt" event={"ID":"f563fffd-fb25-4763-98f7-fdec204a5637","Type":"ContainerDied","Data":"d68d37c09aa82fbbacd883e8ddc8b0f128922599bb32494745c7d246118ff3fb"} Nov 03 03:13:29.001318 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:29.001217 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt" event={"ID":"f563fffd-fb25-4763-98f7-fdec204a5637","Type":"ContainerDied","Data":"7b6dec0065781bd1ef27a8fbfbc707a16b5b03b1d20895b36d76f3ae03a79bb3"} Nov 03 03:13:29.001318 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:29.001239 5605 scope.go:122] "RemoveContainer" containerID="d68d37c09aa82fbbacd883e8ddc8b0f128922599bb32494745c7d246118ff3fb" Nov 03 03:13:29.010569 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:29.010517 5605 scope.go:122] "RemoveContainer" containerID="d68d37c09aa82fbbacd883e8ddc8b0f128922599bb32494745c7d246118ff3fb" Nov 03 03:13:29.010867 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:13:29.010843 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d68d37c09aa82fbbacd883e8ddc8b0f128922599bb32494745c7d246118ff3fb\": container with ID starting with d68d37c09aa82fbbacd883e8ddc8b0f128922599bb32494745c7d246118ff3fb not found: ID does not exist" containerID="d68d37c09aa82fbbacd883e8ddc8b0f128922599bb32494745c7d246118ff3fb" Nov 03 03:13:29.010941 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:29.010882 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d68d37c09aa82fbbacd883e8ddc8b0f128922599bb32494745c7d246118ff3fb"} err="failed to get container status \"d68d37c09aa82fbbacd883e8ddc8b0f128922599bb32494745c7d246118ff3fb\": rpc error: code = NotFound desc = could not find container \"d68d37c09aa82fbbacd883e8ddc8b0f128922599bb32494745c7d246118ff3fb\": container with ID starting with d68d37c09aa82fbbacd883e8ddc8b0f128922599bb32494745c7d246118ff3fb not found: ID does not exist" Nov 03 03:13:29.155602 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:29.155516 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt"] Nov 03 03:13:29.159089 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:29.159059 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-kkjtt"] Nov 03 03:13:30.048690 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:30.048653 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af76fd17-lhvd7"] Nov 03 03:13:30.050816 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:30.050790 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af76fd17-lhvd7"] Nov 03 03:13:30.848914 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:30.848882 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b9772d3-3595-4ac3-b747-d7cbaae23c08" path="/var/lib/kubelet/pods/6b9772d3-3595-4ac3-b747-d7cbaae23c08/volumes" Nov 03 03:13:30.849334 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:30.849320 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f563fffd-fb25-4763-98f7-fdec204a5637" path="/var/lib/kubelet/pods/f563fffd-fb25-4763-98f7-fdec204a5637/volumes" Nov 03 03:13:43.618866 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:43.618827 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz"] Nov 03 03:13:43.619443 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:43.619037 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz" podUID="f014c618-b1ce-44b0-b042-f1234c247228" containerName="istio-proxy" containerID="cri-o://b9928e7eb339922c3dd179c1542692e0e9f79374d5ba19da140c7c3bceaa757f" gracePeriod=30 Nov 03 03:13:47.574604 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:47.574566 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:13:48.873129 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:48.873106 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz" Nov 03 03:13:49.004277 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:49.004196 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f014c618-b1ce-44b0-b042-f1234c247228-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f014c618-b1ce-44b0-b042-f1234c247228-istio-data\") pod \"f014c618-b1ce-44b0-b042-f1234c247228\" (UID: \"f014c618-b1ce-44b0-b042-f1234c247228\") " Nov 03 03:13:49.004448 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:49.004310 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f014c618-b1ce-44b0-b042-f1234c247228-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f014c618-b1ce-44b0-b042-f1234c247228-workload-socket\") pod \"f014c618-b1ce-44b0-b042-f1234c247228\" (UID: \"f014c618-b1ce-44b0-b042-f1234c247228\") " Nov 03 03:13:49.004448 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:49.004341 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f014c618-b1ce-44b0-b042-f1234c247228-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f014c618-b1ce-44b0-b042-f1234c247228-istio-podinfo\") pod \"f014c618-b1ce-44b0-b042-f1234c247228\" (UID: \"f014c618-b1ce-44b0-b042-f1234c247228\") " Nov 03 03:13:49.004448 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:49.004402 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f014c618-b1ce-44b0-b042-f1234c247228-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f014c618-b1ce-44b0-b042-f1234c247228-credential-socket\") pod \"f014c618-b1ce-44b0-b042-f1234c247228\" (UID: \"f014c618-b1ce-44b0-b042-f1234c247228\") " Nov 03 03:13:49.004631 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:49.004438 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f014c618-b1ce-44b0-b042-f1234c247228-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f014c618-b1ce-44b0-b042-f1234c247228-istio-envoy\") pod \"f014c618-b1ce-44b0-b042-f1234c247228\" (UID: \"f014c618-b1ce-44b0-b042-f1234c247228\") " Nov 03 03:13:49.004631 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:49.004486 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f014c618-b1ce-44b0-b042-f1234c247228-kube-api-access-jjzf7\" (UniqueName: \"kubernetes.io/projected/f014c618-b1ce-44b0-b042-f1234c247228-kube-api-access-jjzf7\") pod \"f014c618-b1ce-44b0-b042-f1234c247228\" (UID: \"f014c618-b1ce-44b0-b042-f1234c247228\") " Nov 03 03:13:49.004631 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:49.004546 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f014c618-b1ce-44b0-b042-f1234c247228-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f014c618-b1ce-44b0-b042-f1234c247228-workload-certs\") pod \"f014c618-b1ce-44b0-b042-f1234c247228\" (UID: \"f014c618-b1ce-44b0-b042-f1234c247228\") " Nov 03 03:13:49.004631 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:49.004580 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f014c618-b1ce-44b0-b042-f1234c247228-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f014c618-b1ce-44b0-b042-f1234c247228-istiod-ca-cert\") pod \"f014c618-b1ce-44b0-b042-f1234c247228\" (UID: \"f014c618-b1ce-44b0-b042-f1234c247228\") " Nov 03 03:13:49.004631 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:49.004541 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f014c618-b1ce-44b0-b042-f1234c247228-istio-data" pod "f014c618-b1ce-44b0-b042-f1234c247228" (UID: "f014c618-b1ce-44b0-b042-f1234c247228"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:13:49.004631 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:49.004617 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f014c618-b1ce-44b0-b042-f1234c247228-istio-token\" (UniqueName: \"kubernetes.io/projected/f014c618-b1ce-44b0-b042-f1234c247228-istio-token\") pod \"f014c618-b1ce-44b0-b042-f1234c247228\" (UID: \"f014c618-b1ce-44b0-b042-f1234c247228\") " Nov 03 03:13:49.005033 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:49.004643 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f014c618-b1ce-44b0-b042-f1234c247228-workload-socket" pod "f014c618-b1ce-44b0-b042-f1234c247228" (UID: "f014c618-b1ce-44b0-b042-f1234c247228"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:13:49.005033 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:49.004720 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f014c618-b1ce-44b0-b042-f1234c247228-credential-socket" pod "f014c618-b1ce-44b0-b042-f1234c247228" (UID: "f014c618-b1ce-44b0-b042-f1234c247228"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:13:49.005033 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:49.004933 5605 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f014c618-b1ce-44b0-b042-f1234c247228-istio-data\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:49.005033 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:49.004955 5605 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f014c618-b1ce-44b0-b042-f1234c247228-workload-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:49.005033 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:49.004969 5605 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f014c618-b1ce-44b0-b042-f1234c247228-credential-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:49.005033 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:49.004975 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f014c618-b1ce-44b0-b042-f1234c247228-istiod-ca-cert" pod "f014c618-b1ce-44b0-b042-f1234c247228" (UID: "f014c618-b1ce-44b0-b042-f1234c247228"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:13:49.005393 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:49.005218 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f014c618-b1ce-44b0-b042-f1234c247228-workload-certs" pod "f014c618-b1ce-44b0-b042-f1234c247228" (UID: "f014c618-b1ce-44b0-b042-f1234c247228"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:13:49.006773 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:49.006743 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f014c618-b1ce-44b0-b042-f1234c247228-istio-podinfo" pod "f014c618-b1ce-44b0-b042-f1234c247228" (UID: "f014c618-b1ce-44b0-b042-f1234c247228"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:13:49.006927 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:49.006904 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f014c618-b1ce-44b0-b042-f1234c247228-istio-envoy" pod "f014c618-b1ce-44b0-b042-f1234c247228" (UID: "f014c618-b1ce-44b0-b042-f1234c247228"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:13:49.007099 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:49.007083 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f014c618-b1ce-44b0-b042-f1234c247228-istio-token" pod "f014c618-b1ce-44b0-b042-f1234c247228" (UID: "f014c618-b1ce-44b0-b042-f1234c247228"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:13:49.007234 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:49.007212 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f014c618-b1ce-44b0-b042-f1234c247228-kube-api-access-jjzf7" pod "f014c618-b1ce-44b0-b042-f1234c247228" (UID: "f014c618-b1ce-44b0-b042-f1234c247228"). InnerVolumeSpecName "kube-api-access-jjzf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:13:49.104039 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:49.104002 5605 generic.go:356] "Generic (PLEG): container finished" podID="f014c618-b1ce-44b0-b042-f1234c247228" containerID="b9928e7eb339922c3dd179c1542692e0e9f79374d5ba19da140c7c3bceaa757f" exitCode=0 Nov 03 03:13:49.104207 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:49.104057 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz" Nov 03 03:13:49.104207 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:49.104094 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz" event={"ID":"f014c618-b1ce-44b0-b042-f1234c247228","Type":"ContainerDied","Data":"b9928e7eb339922c3dd179c1542692e0e9f79374d5ba19da140c7c3bceaa757f"} Nov 03 03:13:49.104207 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:49.104156 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz" event={"ID":"f014c618-b1ce-44b0-b042-f1234c247228","Type":"ContainerDied","Data":"9d70cda70786af9f333fbbdef619e3dbc281427a6556b61e636956ef0f3c8b5d"} Nov 03 03:13:49.104207 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:49.104187 5605 scope.go:122] "RemoveContainer" containerID="b9928e7eb339922c3dd179c1542692e0e9f79374d5ba19da140c7c3bceaa757f" Nov 03 03:13:49.105519 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:49.105499 5605 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f014c618-b1ce-44b0-b042-f1234c247228-istio-podinfo\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:49.105631 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:49.105524 5605 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f014c618-b1ce-44b0-b042-f1234c247228-istio-envoy\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:49.105631 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:49.105539 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjzf7\" (UniqueName: \"kubernetes.io/projected/f014c618-b1ce-44b0-b042-f1234c247228-kube-api-access-jjzf7\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:49.105631 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:49.105553 5605 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f014c618-b1ce-44b0-b042-f1234c247228-workload-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:49.105631 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:49.105566 5605 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f014c618-b1ce-44b0-b042-f1234c247228-istiod-ca-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:49.105631 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:49.105579 5605 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f014c618-b1ce-44b0-b042-f1234c247228-istio-token\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:49.113832 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:49.113814 5605 scope.go:122] "RemoveContainer" containerID="b9928e7eb339922c3dd179c1542692e0e9f79374d5ba19da140c7c3bceaa757f" Nov 03 03:13:49.114106 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:13:49.114085 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9928e7eb339922c3dd179c1542692e0e9f79374d5ba19da140c7c3bceaa757f\": container with ID starting with b9928e7eb339922c3dd179c1542692e0e9f79374d5ba19da140c7c3bceaa757f not found: ID does not exist" containerID="b9928e7eb339922c3dd179c1542692e0e9f79374d5ba19da140c7c3bceaa757f" Nov 03 03:13:49.114174 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:49.114116 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9928e7eb339922c3dd179c1542692e0e9f79374d5ba19da140c7c3bceaa757f"} err="failed to get container status \"b9928e7eb339922c3dd179c1542692e0e9f79374d5ba19da140c7c3bceaa757f\": rpc error: code = NotFound desc = could not find container \"b9928e7eb339922c3dd179c1542692e0e9f79374d5ba19da140c7c3bceaa757f\": container with ID starting with b9928e7eb339922c3dd179c1542692e0e9f79374d5ba19da140c7c3bceaa757f not found: ID does not exist" Nov 03 03:13:49.258596 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:49.258513 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz"] Nov 03 03:13:49.261990 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:49.261970 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4zgkz"] Nov 03 03:13:50.697958 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:50.697925 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2"] Nov 03 03:13:50.698404 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:50.698131 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2" podUID="ccc49b02-1ca4-421c-8f34-320fbeb0e0ac" containerName="istio-proxy" containerID="cri-o://1d6ad52f6912003baf1ae6708aa92db4a64e74ba3548d859b3c990514dfb2c7e" gracePeriod=30 Nov 03 03:13:50.848813 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:50.848776 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f014c618-b1ce-44b0-b042-f1234c247228" path="/var/lib/kubelet/pods/f014c618-b1ce-44b0-b042-f1234c247228/volumes" Nov 03 03:13:53.911747 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:13:53.911699 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:13:55.942456 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:55.942428 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2" Nov 03 03:13:55.965243 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:55.965189 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccc49b02-1ca4-421c-8f34-320fbeb0e0ac-kube-api-access-lpz68\" (UniqueName: \"kubernetes.io/projected/ccc49b02-1ca4-421c-8f34-320fbeb0e0ac-kube-api-access-lpz68\") pod \"ccc49b02-1ca4-421c-8f34-320fbeb0e0ac\" (UID: \"ccc49b02-1ca4-421c-8f34-320fbeb0e0ac\") " Nov 03 03:13:55.965243 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:55.965231 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccc49b02-1ca4-421c-8f34-320fbeb0e0ac-istio-token\" (UniqueName: \"kubernetes.io/projected/ccc49b02-1ca4-421c-8f34-320fbeb0e0ac-istio-token\") pod \"ccc49b02-1ca4-421c-8f34-320fbeb0e0ac\" (UID: \"ccc49b02-1ca4-421c-8f34-320fbeb0e0ac\") " Nov 03 03:13:55.965450 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:55.965256 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccc49b02-1ca4-421c-8f34-320fbeb0e0ac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc49b02-1ca4-421c-8f34-320fbeb0e0ac-workload-socket\") pod \"ccc49b02-1ca4-421c-8f34-320fbeb0e0ac\" (UID: \"ccc49b02-1ca4-421c-8f34-320fbeb0e0ac\") " Nov 03 03:13:55.965450 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:55.965284 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccc49b02-1ca4-421c-8f34-320fbeb0e0ac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccc49b02-1ca4-421c-8f34-320fbeb0e0ac-istiod-ca-cert\") pod \"ccc49b02-1ca4-421c-8f34-320fbeb0e0ac\" (UID: \"ccc49b02-1ca4-421c-8f34-320fbeb0e0ac\") " Nov 03 03:13:55.965450 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:55.965346 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccc49b02-1ca4-421c-8f34-320fbeb0e0ac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccc49b02-1ca4-421c-8f34-320fbeb0e0ac-istio-data\") pod \"ccc49b02-1ca4-421c-8f34-320fbeb0e0ac\" (UID: \"ccc49b02-1ca4-421c-8f34-320fbeb0e0ac\") " Nov 03 03:13:55.965450 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:55.965410 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccc49b02-1ca4-421c-8f34-320fbeb0e0ac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccc49b02-1ca4-421c-8f34-320fbeb0e0ac-istio-envoy\") pod \"ccc49b02-1ca4-421c-8f34-320fbeb0e0ac\" (UID: \"ccc49b02-1ca4-421c-8f34-320fbeb0e0ac\") " Nov 03 03:13:55.965682 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:55.965477 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccc49b02-1ca4-421c-8f34-320fbeb0e0ac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccc49b02-1ca4-421c-8f34-320fbeb0e0ac-workload-certs\") pod \"ccc49b02-1ca4-421c-8f34-320fbeb0e0ac\" (UID: \"ccc49b02-1ca4-421c-8f34-320fbeb0e0ac\") " Nov 03 03:13:55.965682 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:55.965534 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccc49b02-1ca4-421c-8f34-320fbeb0e0ac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc49b02-1ca4-421c-8f34-320fbeb0e0ac-credential-socket\") pod \"ccc49b02-1ca4-421c-8f34-320fbeb0e0ac\" (UID: \"ccc49b02-1ca4-421c-8f34-320fbeb0e0ac\") " Nov 03 03:13:55.965682 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:55.965568 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccc49b02-1ca4-421c-8f34-320fbeb0e0ac-workload-socket" pod "ccc49b02-1ca4-421c-8f34-320fbeb0e0ac" (UID: "ccc49b02-1ca4-421c-8f34-320fbeb0e0ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:13:55.965682 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:55.965617 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ccc49b02-1ca4-421c-8f34-320fbeb0e0ac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccc49b02-1ca4-421c-8f34-320fbeb0e0ac-istio-podinfo\") pod \"ccc49b02-1ca4-421c-8f34-320fbeb0e0ac\" (UID: \"ccc49b02-1ca4-421c-8f34-320fbeb0e0ac\") " Nov 03 03:13:55.965682 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:55.965653 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccc49b02-1ca4-421c-8f34-320fbeb0e0ac-istio-data" pod "ccc49b02-1ca4-421c-8f34-320fbeb0e0ac" (UID: "ccc49b02-1ca4-421c-8f34-320fbeb0e0ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:13:55.965960 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:55.965714 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccc49b02-1ca4-421c-8f34-320fbeb0e0ac-istiod-ca-cert" pod "ccc49b02-1ca4-421c-8f34-320fbeb0e0ac" (UID: "ccc49b02-1ca4-421c-8f34-320fbeb0e0ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:13:55.965960 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:55.965794 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccc49b02-1ca4-421c-8f34-320fbeb0e0ac-workload-certs" pod "ccc49b02-1ca4-421c-8f34-320fbeb0e0ac" (UID: "ccc49b02-1ca4-421c-8f34-320fbeb0e0ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:13:55.965960 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:55.965830 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccc49b02-1ca4-421c-8f34-320fbeb0e0ac-credential-socket" pod "ccc49b02-1ca4-421c-8f34-320fbeb0e0ac" (UID: "ccc49b02-1ca4-421c-8f34-320fbeb0e0ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:13:55.965960 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:55.965905 5605 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc49b02-1ca4-421c-8f34-320fbeb0e0ac-workload-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:55.965960 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:55.965926 5605 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccc49b02-1ca4-421c-8f34-320fbeb0e0ac-istiod-ca-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:55.965960 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:55.965940 5605 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccc49b02-1ca4-421c-8f34-320fbeb0e0ac-istio-data\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:55.965960 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:55.965954 5605 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccc49b02-1ca4-421c-8f34-320fbeb0e0ac-workload-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:55.966349 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:55.965970 5605 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc49b02-1ca4-421c-8f34-320fbeb0e0ac-credential-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:55.967588 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:55.967564 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccc49b02-1ca4-421c-8f34-320fbeb0e0ac-kube-api-access-lpz68" pod "ccc49b02-1ca4-421c-8f34-320fbeb0e0ac" (UID: "ccc49b02-1ca4-421c-8f34-320fbeb0e0ac"). InnerVolumeSpecName "kube-api-access-lpz68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:13:55.967704 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:55.967585 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccc49b02-1ca4-421c-8f34-320fbeb0e0ac-istio-envoy" pod "ccc49b02-1ca4-421c-8f34-320fbeb0e0ac" (UID: "ccc49b02-1ca4-421c-8f34-320fbeb0e0ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:13:55.967704 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:55.967657 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccc49b02-1ca4-421c-8f34-320fbeb0e0ac-istio-token" pod "ccc49b02-1ca4-421c-8f34-320fbeb0e0ac" (UID: "ccc49b02-1ca4-421c-8f34-320fbeb0e0ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:13:55.967799 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:55.967699 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ccc49b02-1ca4-421c-8f34-320fbeb0e0ac-istio-podinfo" pod "ccc49b02-1ca4-421c-8f34-320fbeb0e0ac" (UID: "ccc49b02-1ca4-421c-8f34-320fbeb0e0ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:13:56.066342 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:56.066320 5605 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccc49b02-1ca4-421c-8f34-320fbeb0e0ac-istio-podinfo\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:56.066342 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:56.066342 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpz68\" (UniqueName: \"kubernetes.io/projected/ccc49b02-1ca4-421c-8f34-320fbeb0e0ac-kube-api-access-lpz68\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:56.066503 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:56.066352 5605 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccc49b02-1ca4-421c-8f34-320fbeb0e0ac-istio-token\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:56.066503 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:56.066387 5605 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccc49b02-1ca4-421c-8f34-320fbeb0e0ac-istio-envoy\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:56.140010 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:56.139975 5605 generic.go:356] "Generic (PLEG): container finished" podID="ccc49b02-1ca4-421c-8f34-320fbeb0e0ac" containerID="1d6ad52f6912003baf1ae6708aa92db4a64e74ba3548d859b3c990514dfb2c7e" exitCode=0 Nov 03 03:13:56.140170 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:56.140025 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2" Nov 03 03:13:56.140170 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:56.140070 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2" event={"ID":"ccc49b02-1ca4-421c-8f34-320fbeb0e0ac","Type":"ContainerDied","Data":"1d6ad52f6912003baf1ae6708aa92db4a64e74ba3548d859b3c990514dfb2c7e"} Nov 03 03:13:56.140170 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:56.140133 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2" event={"ID":"ccc49b02-1ca4-421c-8f34-320fbeb0e0ac","Type":"ContainerDied","Data":"92a4a59c3d3b7e252db55b69f6b4056db9b068d6bcaef7cd0559a0ffd844f709"} Nov 03 03:13:56.140170 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:56.140162 5605 scope.go:122] "RemoveContainer" containerID="1d6ad52f6912003baf1ae6708aa92db4a64e74ba3548d859b3c990514dfb2c7e" Nov 03 03:13:56.150321 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:56.150296 5605 scope.go:122] "RemoveContainer" containerID="1d6ad52f6912003baf1ae6708aa92db4a64e74ba3548d859b3c990514dfb2c7e" Nov 03 03:13:56.150750 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:13:56.150723 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d6ad52f6912003baf1ae6708aa92db4a64e74ba3548d859b3c990514dfb2c7e\": container with ID starting with 1d6ad52f6912003baf1ae6708aa92db4a64e74ba3548d859b3c990514dfb2c7e not found: ID does not exist" containerID="1d6ad52f6912003baf1ae6708aa92db4a64e74ba3548d859b3c990514dfb2c7e" Nov 03 03:13:56.150844 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:56.150760 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d6ad52f6912003baf1ae6708aa92db4a64e74ba3548d859b3c990514dfb2c7e"} err="failed to get container status \"1d6ad52f6912003baf1ae6708aa92db4a64e74ba3548d859b3c990514dfb2c7e\": rpc error: code = NotFound desc = could not find container \"1d6ad52f6912003baf1ae6708aa92db4a64e74ba3548d859b3c990514dfb2c7e\": container with ID starting with 1d6ad52f6912003baf1ae6708aa92db4a64e74ba3548d859b3c990514dfb2c7e not found: ID does not exist" Nov 03 03:13:56.294090 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:56.294023 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2"] Nov 03 03:13:56.297659 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:56.297635 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-ns4c2"] Nov 03 03:13:56.726677 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:56.726643 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-957b33-54c6fcf67c-td8mv"] Nov 03 03:13:56.848873 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:13:56.848843 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccc49b02-1ca4-421c-8f34-320fbeb0e0ac" path="/var/lib/kubelet/pods/ccc49b02-1ca4-421c-8f34-320fbeb0e0ac/volumes" Nov 03 03:14:01.727813 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:01.727768 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-957b33-54c6fcf67c-td8mv" podUID="cb7f4ac2-ff63-48b0-ab55-74bf25ffbb6a" containerName="payload-processing" containerID="cri-o://7c6c22377f9718b4f4aa017a0b52be07ac5c57f59c30e4a8a3721cb6c6d0f280" gracePeriod=25 Nov 03 03:14:05.384574 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:05.384543 5605 scope.go:122] "RemoveContainer" containerID="849a340013366e838d2ac2b3a3bebeb1186590dffe14fa226f3c08fb2973937b" Nov 03 03:14:08.043257 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:08.043221 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-957b33-54c6fcf67c-td8mv" podUID="cb7f4ac2-ff63-48b0-ab55-74bf25ffbb6a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:14:15.530403 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:15.530367 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:14:18.043022 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:18.042981 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-957b33-54c6fcf67c-td8mv" podUID="cb7f4ac2-ff63-48b0-ab55-74bf25ffbb6a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:14:20.390740 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:20.390701 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6aba08c1-hndzn"] Nov 03 03:14:20.391133 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:20.391012 5605 cpu_manager.go:414] "containerMap: removing container" podUID="5a71cdee-425e-4787-9217-5fb6b97e232b" containerName="main" Nov 03 03:14:20.391133 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:20.391024 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a71cdee-425e-4787-9217-5fb6b97e232b" containerName="main" Nov 03 03:14:20.391133 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:20.391033 5605 cpu_manager.go:414] "containerMap: removing container" podUID="55eeee4d-8327-4e10-836e-d6609ad111a3" containerName="revoke-keys" Nov 03 03:14:20.391133 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:20.391039 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55eeee4d-8327-4e10-836e-d6609ad111a3" containerName="revoke-keys" Nov 03 03:14:20.391133 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:20.391047 5605 cpu_manager.go:414] "containerMap: removing container" podUID="f563fffd-fb25-4763-98f7-fdec204a5637" containerName="istio-proxy" Nov 03 03:14:20.391133 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:20.391052 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f563fffd-fb25-4763-98f7-fdec204a5637" containerName="istio-proxy" Nov 03 03:14:20.391133 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:20.391060 5605 cpu_manager.go:414] "containerMap: removing container" podUID="ccc49b02-1ca4-421c-8f34-320fbeb0e0ac" containerName="istio-proxy" Nov 03 03:14:20.391133 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:20.391067 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccc49b02-1ca4-421c-8f34-320fbeb0e0ac" containerName="istio-proxy" Nov 03 03:14:20.391133 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:20.391079 5605 cpu_manager.go:414] "containerMap: removing container" podUID="dd4a9d44-c63e-4f67-bcac-a4f98ecf1796" containerName="main" Nov 03 03:14:20.391133 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:20.391084 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd4a9d44-c63e-4f67-bcac-a4f98ecf1796" containerName="main" Nov 03 03:14:20.391133 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:20.391101 5605 cpu_manager.go:414] "containerMap: removing container" podUID="f014c618-b1ce-44b0-b042-f1234c247228" containerName="istio-proxy" Nov 03 03:14:20.391133 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:20.391106 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f014c618-b1ce-44b0-b042-f1234c247228" containerName="istio-proxy" Nov 03 03:14:20.391559 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:20.391174 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a71cdee-425e-4787-9217-5fb6b97e232b" containerName="main" Nov 03 03:14:20.391559 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:20.391182 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="55eeee4d-8327-4e10-836e-d6609ad111a3" containerName="revoke-keys" Nov 03 03:14:20.391559 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:20.391189 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="f563fffd-fb25-4763-98f7-fdec204a5637" containerName="istio-proxy" Nov 03 03:14:20.391559 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:20.391195 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccc49b02-1ca4-421c-8f34-320fbeb0e0ac" containerName="istio-proxy" Nov 03 03:14:20.391559 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:20.391202 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd4a9d44-c63e-4f67-bcac-a4f98ecf1796" containerName="main" Nov 03 03:14:20.391559 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:20.391209 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="f014c618-b1ce-44b0-b042-f1234c247228" containerName="istio-proxy" Nov 03 03:14:20.394348 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:20.394328 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aba08c1-hndzn" Nov 03 03:14:20.443587 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:20.443568 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:14:20.443714 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:20.443603 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:14:20.459013 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:20.458984 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09fa4230-1064-47be-9d05-34232e5cb1e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6aba08c1-hndzn\" (UID: \"09fa4230-1064-47be-9d05-34232e5cb1e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aba08c1-hndzn" Nov 03 03:14:20.459129 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:20.459087 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09fa4230-1064-47be-9d05-34232e5cb1e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6aba08c1-hndzn\" (UID: \"09fa4230-1064-47be-9d05-34232e5cb1e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aba08c1-hndzn" Nov 03 03:14:20.459172 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:20.459128 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7zpv\" (UniqueName: \"kubernetes.io/projected/09fa4230-1064-47be-9d05-34232e5cb1e3-kube-api-access-b7zpv\") pod \"maas-api-revoke-sub-6aba08c1-hndzn\" (UID: \"09fa4230-1064-47be-9d05-34232e5cb1e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aba08c1-hndzn" Nov 03 03:14:20.490294 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:20.490269 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:14:20.490552 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:20.490535 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 03:14:20.492294 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:20.492273 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6aba08c1-hndzn"] Nov 03 03:14:20.560470 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:20.560446 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09fa4230-1064-47be-9d05-34232e5cb1e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6aba08c1-hndzn\" (UID: \"09fa4230-1064-47be-9d05-34232e5cb1e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aba08c1-hndzn" Nov 03 03:14:20.560582 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:20.560492 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09fa4230-1064-47be-9d05-34232e5cb1e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6aba08c1-hndzn\" (UID: \"09fa4230-1064-47be-9d05-34232e5cb1e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aba08c1-hndzn" Nov 03 03:14:20.560582 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:20.560520 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7zpv\" (UniqueName: \"kubernetes.io/projected/09fa4230-1064-47be-9d05-34232e5cb1e3-kube-api-access-b7zpv\") pod \"maas-api-revoke-sub-6aba08c1-hndzn\" (UID: \"09fa4230-1064-47be-9d05-34232e5cb1e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aba08c1-hndzn" Nov 03 03:14:20.561063 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:20.561043 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09fa4230-1064-47be-9d05-34232e5cb1e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6aba08c1-hndzn\" (UID: \"09fa4230-1064-47be-9d05-34232e5cb1e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aba08c1-hndzn" Nov 03 03:14:20.562804 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:20.562784 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09fa4230-1064-47be-9d05-34232e5cb1e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6aba08c1-hndzn\" (UID: \"09fa4230-1064-47be-9d05-34232e5cb1e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aba08c1-hndzn" Nov 03 03:14:20.613902 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:20.613877 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7zpv\" (UniqueName: \"kubernetes.io/projected/09fa4230-1064-47be-9d05-34232e5cb1e3-kube-api-access-b7zpv\") pod \"maas-api-revoke-sub-6aba08c1-hndzn\" (UID: \"09fa4230-1064-47be-9d05-34232e5cb1e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aba08c1-hndzn" Nov 03 03:14:20.706766 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:20.706717 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aba08c1-hndzn" Nov 03 03:14:21.072307 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:21.072275 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6aba08c1-hndzn"] Nov 03 03:14:21.078218 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:14:21.078187 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09fa4230_1064_47be_9d05_34232e5cb1e3.slice/crio-a9b5060249b33207b636d0065ab1016d5d4fddc882baffaa945206424ddd2ae6 WatchSource:0}: Error finding container a9b5060249b33207b636d0065ab1016d5d4fddc882baffaa945206424ddd2ae6: Status 404 returned error can't find the container with id a9b5060249b33207b636d0065ab1016d5d4fddc882baffaa945206424ddd2ae6 Nov 03 03:14:21.269918 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:21.269887 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aba08c1-hndzn" event={"ID":"09fa4230-1064-47be-9d05-34232e5cb1e3","Type":"ContainerStarted","Data":"a9b5060249b33207b636d0065ab1016d5d4fddc882baffaa945206424ddd2ae6"} Nov 03 03:14:21.967801 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:21.967777 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-957b33-54c6fcf67c-td8mv" Nov 03 03:14:22.073043 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:22.073010 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb7f4ac2-ff63-48b0-ab55-74bf25ffbb6a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb7f4ac2-ff63-48b0-ab55-74bf25ffbb6a-plugins-config-volume\") pod \"cb7f4ac2-ff63-48b0-ab55-74bf25ffbb6a\" (UID: \"cb7f4ac2-ff63-48b0-ab55-74bf25ffbb6a\") " Nov 03 03:14:22.073166 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:22.073044 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb7f4ac2-ff63-48b0-ab55-74bf25ffbb6a-kube-api-access-9wbhb\" (UniqueName: \"kubernetes.io/projected/cb7f4ac2-ff63-48b0-ab55-74bf25ffbb6a-kube-api-access-9wbhb\") pod \"cb7f4ac2-ff63-48b0-ab55-74bf25ffbb6a\" (UID: \"cb7f4ac2-ff63-48b0-ab55-74bf25ffbb6a\") " Nov 03 03:14:22.073368 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:22.073335 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb7f4ac2-ff63-48b0-ab55-74bf25ffbb6a-plugins-config-volume" pod "cb7f4ac2-ff63-48b0-ab55-74bf25ffbb6a" (UID: "cb7f4ac2-ff63-48b0-ab55-74bf25ffbb6a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:14:22.076333 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:22.076285 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb7f4ac2-ff63-48b0-ab55-74bf25ffbb6a-kube-api-access-9wbhb" pod "cb7f4ac2-ff63-48b0-ab55-74bf25ffbb6a" (UID: "cb7f4ac2-ff63-48b0-ab55-74bf25ffbb6a"). InnerVolumeSpecName "kube-api-access-9wbhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:14:22.173863 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:22.173835 5605 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb7f4ac2-ff63-48b0-ab55-74bf25ffbb6a-plugins-config-volume\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:14:22.173863 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:22.173859 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wbhb\" (UniqueName: \"kubernetes.io/projected/cb7f4ac2-ff63-48b0-ab55-74bf25ffbb6a-kube-api-access-9wbhb\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:14:22.276587 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:22.276557 5605 generic.go:356] "Generic (PLEG): container finished" podID="09fa4230-1064-47be-9d05-34232e5cb1e3" containerID="89c8710d210d554210b63762285e262e0c59494c7b556b68b3e1031ebad8d3eb" exitCode=0 Nov 03 03:14:22.276701 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:22.276646 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aba08c1-hndzn" event={"ID":"09fa4230-1064-47be-9d05-34232e5cb1e3","Type":"ContainerDied","Data":"89c8710d210d554210b63762285e262e0c59494c7b556b68b3e1031ebad8d3eb"} Nov 03 03:14:22.277926 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:22.277906 5605 generic.go:356] "Generic (PLEG): container finished" podID="cb7f4ac2-ff63-48b0-ab55-74bf25ffbb6a" containerID="7c6c22377f9718b4f4aa017a0b52be07ac5c57f59c30e4a8a3721cb6c6d0f280" exitCode=0 Nov 03 03:14:22.278026 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:22.277944 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-957b33-54c6fcf67c-td8mv" Nov 03 03:14:22.278026 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:22.277945 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-957b33-54c6fcf67c-td8mv" event={"ID":"cb7f4ac2-ff63-48b0-ab55-74bf25ffbb6a","Type":"ContainerDied","Data":"7c6c22377f9718b4f4aa017a0b52be07ac5c57f59c30e4a8a3721cb6c6d0f280"} Nov 03 03:14:22.278107 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:22.278063 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-957b33-54c6fcf67c-td8mv" event={"ID":"cb7f4ac2-ff63-48b0-ab55-74bf25ffbb6a","Type":"ContainerDied","Data":"144e6723be0c9f8b70ee7c32971a268f29eddfdc2cb0da04022922b4f35d6c79"} Nov 03 03:14:22.278107 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:22.278092 5605 scope.go:122] "RemoveContainer" containerID="7c6c22377f9718b4f4aa017a0b52be07ac5c57f59c30e4a8a3721cb6c6d0f280" Nov 03 03:14:22.287453 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:22.287432 5605 scope.go:122] "RemoveContainer" containerID="7c6c22377f9718b4f4aa017a0b52be07ac5c57f59c30e4a8a3721cb6c6d0f280" Nov 03 03:14:22.287699 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:14:22.287681 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c6c22377f9718b4f4aa017a0b52be07ac5c57f59c30e4a8a3721cb6c6d0f280\": container with ID starting with 7c6c22377f9718b4f4aa017a0b52be07ac5c57f59c30e4a8a3721cb6c6d0f280 not found: ID does not exist" containerID="7c6c22377f9718b4f4aa017a0b52be07ac5c57f59c30e4a8a3721cb6c6d0f280" Nov 03 03:14:22.287765 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:22.287709 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c6c22377f9718b4f4aa017a0b52be07ac5c57f59c30e4a8a3721cb6c6d0f280"} err="failed to get container status \"7c6c22377f9718b4f4aa017a0b52be07ac5c57f59c30e4a8a3721cb6c6d0f280\": rpc error: code = NotFound desc = could not find container \"7c6c22377f9718b4f4aa017a0b52be07ac5c57f59c30e4a8a3721cb6c6d0f280\": container with ID starting with 7c6c22377f9718b4f4aa017a0b52be07ac5c57f59c30e4a8a3721cb6c6d0f280 not found: ID does not exist" Nov 03 03:14:22.485695 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:22.485630 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-957b33-54c6fcf67c-td8mv"] Nov 03 03:14:22.493590 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:22.488987 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-957b33-54c6fcf67c-td8mv"] Nov 03 03:14:22.848935 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:22.848905 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb7f4ac2-ff63-48b0-ab55-74bf25ffbb6a" path="/var/lib/kubelet/pods/cb7f4ac2-ff63-48b0-ab55-74bf25ffbb6a/volumes" Nov 03 03:14:23.411140 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:23.411114 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aba08c1-hndzn" Nov 03 03:14:23.484118 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:23.484095 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09fa4230-1064-47be-9d05-34232e5cb1e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09fa4230-1064-47be-9d05-34232e5cb1e3-maas-api-xks-ca\") pod \"09fa4230-1064-47be-9d05-34232e5cb1e3\" (UID: \"09fa4230-1064-47be-9d05-34232e5cb1e3\") " Nov 03 03:14:23.484217 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:23.484151 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09fa4230-1064-47be-9d05-34232e5cb1e3-kube-api-access-b7zpv\" (UniqueName: \"kubernetes.io/projected/09fa4230-1064-47be-9d05-34232e5cb1e3-kube-api-access-b7zpv\") pod \"09fa4230-1064-47be-9d05-34232e5cb1e3\" (UID: \"09fa4230-1064-47be-9d05-34232e5cb1e3\") " Nov 03 03:14:23.484217 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:23.484189 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09fa4230-1064-47be-9d05-34232e5cb1e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09fa4230-1064-47be-9d05-34232e5cb1e3-maas-api-openshift-ca\") pod \"09fa4230-1064-47be-9d05-34232e5cb1e3\" (UID: \"09fa4230-1064-47be-9d05-34232e5cb1e3\") " Nov 03 03:14:23.484639 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:23.484607 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09fa4230-1064-47be-9d05-34232e5cb1e3-maas-api-openshift-ca" pod "09fa4230-1064-47be-9d05-34232e5cb1e3" (UID: "09fa4230-1064-47be-9d05-34232e5cb1e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:14:23.486215 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:23.486191 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09fa4230-1064-47be-9d05-34232e5cb1e3-maas-api-xks-ca" pod "09fa4230-1064-47be-9d05-34232e5cb1e3" (UID: "09fa4230-1064-47be-9d05-34232e5cb1e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:14:23.486310 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:23.486229 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09fa4230-1064-47be-9d05-34232e5cb1e3-kube-api-access-b7zpv" pod "09fa4230-1064-47be-9d05-34232e5cb1e3" (UID: "09fa4230-1064-47be-9d05-34232e5cb1e3"). InnerVolumeSpecName "kube-api-access-b7zpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:14:23.585338 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:23.585312 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09fa4230-1064-47be-9d05-34232e5cb1e3-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:14:23.585338 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:23.585337 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7zpv\" (UniqueName: \"kubernetes.io/projected/09fa4230-1064-47be-9d05-34232e5cb1e3-kube-api-access-b7zpv\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:14:23.585510 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:23.585346 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09fa4230-1064-47be-9d05-34232e5cb1e3-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:14:23.910425 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:14:23.910336 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:14:24.290538 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:24.290506 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aba08c1-hndzn" Nov 03 03:14:24.290688 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:24.290532 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aba08c1-hndzn" event={"ID":"09fa4230-1064-47be-9d05-34232e5cb1e3","Type":"ContainerDied","Data":"a9b5060249b33207b636d0065ab1016d5d4fddc882baffaa945206424ddd2ae6"} Nov 03 03:14:24.290688 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:24.290580 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9b5060249b33207b636d0065ab1016d5d4fddc882baffaa945206424ddd2ae6" Nov 03 03:14:24.761621 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:24.761579 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-521f1f/premium-w2-521f1f-kserve-7578fb7957-89wfb"] Nov 03 03:14:24.766041 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:24.766015 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-521f1f/simulator-w2-521f1f-kserve-5f47b86f6-mgksf"] Nov 03 03:14:30.475091 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:30.475057 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:14:32.155118 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:32.155087 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-796c034d-4lksf"] Nov 03 03:14:32.155563 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:32.155402 5605 cpu_manager.go:414] "containerMap: removing container" podUID="cb7f4ac2-ff63-48b0-ab55-74bf25ffbb6a" containerName="payload-processing" Nov 03 03:14:32.155563 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:32.155414 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb7f4ac2-ff63-48b0-ab55-74bf25ffbb6a" containerName="payload-processing" Nov 03 03:14:32.155563 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:32.155436 5605 cpu_manager.go:414] "containerMap: removing container" podUID="09fa4230-1064-47be-9d05-34232e5cb1e3" containerName="revoke-keys" Nov 03 03:14:32.155563 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:32.155441 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09fa4230-1064-47be-9d05-34232e5cb1e3" containerName="revoke-keys" Nov 03 03:14:32.155563 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:32.155513 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="09fa4230-1064-47be-9d05-34232e5cb1e3" containerName="revoke-keys" Nov 03 03:14:32.155563 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:32.155527 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb7f4ac2-ff63-48b0-ab55-74bf25ffbb6a" containerName="payload-processing" Nov 03 03:14:32.158731 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:32.158710 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-796c034d-4lksf" Nov 03 03:14:32.207300 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:32.207273 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:14:32.210183 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:32.210164 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 03:14:32.252517 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:32.252491 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e5078d7-c363-438d-891e-e03666ee4fb4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-796c034d-4lksf\" (UID: \"3e5078d7-c363-438d-891e-e03666ee4fb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-796c034d-4lksf" Nov 03 03:14:32.252635 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:32.252547 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e5078d7-c363-438d-891e-e03666ee4fb4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-796c034d-4lksf\" (UID: \"3e5078d7-c363-438d-891e-e03666ee4fb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-796c034d-4lksf" Nov 03 03:14:32.252635 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:32.252615 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92nl7\" (UniqueName: \"kubernetes.io/projected/3e5078d7-c363-438d-891e-e03666ee4fb4-kube-api-access-92nl7\") pod \"maas-api-revoke-sub-796c034d-4lksf\" (UID: \"3e5078d7-c363-438d-891e-e03666ee4fb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-796c034d-4lksf" Nov 03 03:14:32.260373 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:32.260310 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-796c034d-4lksf"] Nov 03 03:14:32.265824 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:32.265802 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:14:32.265905 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:32.265829 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:14:32.353706 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:32.353683 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e5078d7-c363-438d-891e-e03666ee4fb4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-796c034d-4lksf\" (UID: \"3e5078d7-c363-438d-891e-e03666ee4fb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-796c034d-4lksf" Nov 03 03:14:32.353802 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:32.353719 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92nl7\" (UniqueName: \"kubernetes.io/projected/3e5078d7-c363-438d-891e-e03666ee4fb4-kube-api-access-92nl7\") pod \"maas-api-revoke-sub-796c034d-4lksf\" (UID: \"3e5078d7-c363-438d-891e-e03666ee4fb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-796c034d-4lksf" Nov 03 03:14:32.353802 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:32.353759 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e5078d7-c363-438d-891e-e03666ee4fb4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-796c034d-4lksf\" (UID: \"3e5078d7-c363-438d-891e-e03666ee4fb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-796c034d-4lksf" Nov 03 03:14:32.354384 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:32.354339 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e5078d7-c363-438d-891e-e03666ee4fb4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-796c034d-4lksf\" (UID: \"3e5078d7-c363-438d-891e-e03666ee4fb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-796c034d-4lksf" Nov 03 03:14:32.356111 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:32.356094 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e5078d7-c363-438d-891e-e03666ee4fb4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-796c034d-4lksf\" (UID: \"3e5078d7-c363-438d-891e-e03666ee4fb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-796c034d-4lksf" Nov 03 03:14:32.407013 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:32.406951 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92nl7\" (UniqueName: \"kubernetes.io/projected/3e5078d7-c363-438d-891e-e03666ee4fb4-kube-api-access-92nl7\") pod \"maas-api-revoke-sub-796c034d-4lksf\" (UID: \"3e5078d7-c363-438d-891e-e03666ee4fb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-796c034d-4lksf" Nov 03 03:14:32.474510 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:32.474485 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-796c034d-4lksf" Nov 03 03:14:32.839085 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:32.839055 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-796c034d-4lksf"] Nov 03 03:14:32.844982 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:14:32.844949 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e5078d7_c363_438d_891e_e03666ee4fb4.slice/crio-3e4d842c9721762a83a5b7535cdd4959dfcd7dde111fe201303f2eccd57efb6d WatchSource:0}: Error finding container 3e4d842c9721762a83a5b7535cdd4959dfcd7dde111fe201303f2eccd57efb6d: Status 404 returned error can't find the container with id 3e4d842c9721762a83a5b7535cdd4959dfcd7dde111fe201303f2eccd57efb6d Nov 03 03:14:33.333423 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:33.333390 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-796c034d-4lksf" event={"ID":"3e5078d7-c363-438d-891e-e03666ee4fb4","Type":"ContainerStarted","Data":"c376e2e6dbebc0888b539f008362a003d55c5e433fd5ed9390ced812014a9a73"} Nov 03 03:14:33.333793 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:33.333448 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-796c034d-4lksf" event={"ID":"3e5078d7-c363-438d-891e-e03666ee4fb4","Type":"ContainerStarted","Data":"3e4d842c9721762a83a5b7535cdd4959dfcd7dde111fe201303f2eccd57efb6d"} Nov 03 03:14:33.492213 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:33.492170 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-796c034d-4lksf" podStartSLOduration=1.492157696 podStartE2EDuration="1.492157696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:14:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:14:33.444928552 +0000 UTC m=+4950.704126706" watchObservedRunningTime="2026-11-03 03:14:33.492157696 +0000 UTC m=+4950.751355847" Nov 03 03:14:34.340568 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:34.340532 5605 generic.go:356] "Generic (PLEG): container finished" podID="3e5078d7-c363-438d-891e-e03666ee4fb4" containerID="c376e2e6dbebc0888b539f008362a003d55c5e433fd5ed9390ced812014a9a73" exitCode=0 Nov 03 03:14:34.340568 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:34.340571 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-796c034d-4lksf" event={"ID":"3e5078d7-c363-438d-891e-e03666ee4fb4","Type":"ContainerDied","Data":"c376e2e6dbebc0888b539f008362a003d55c5e433fd5ed9390ced812014a9a73"} Nov 03 03:14:35.468868 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:35.468845 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-796c034d-4lksf" Nov 03 03:14:35.582046 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:35.582016 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e5078d7-c363-438d-891e-e03666ee4fb4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e5078d7-c363-438d-891e-e03666ee4fb4-maas-api-xks-ca\") pod \"3e5078d7-c363-438d-891e-e03666ee4fb4\" (UID: \"3e5078d7-c363-438d-891e-e03666ee4fb4\") " Nov 03 03:14:35.582253 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:35.582079 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e5078d7-c363-438d-891e-e03666ee4fb4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e5078d7-c363-438d-891e-e03666ee4fb4-maas-api-openshift-ca\") pod \"3e5078d7-c363-438d-891e-e03666ee4fb4\" (UID: \"3e5078d7-c363-438d-891e-e03666ee4fb4\") " Nov 03 03:14:35.582253 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:35.582120 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e5078d7-c363-438d-891e-e03666ee4fb4-kube-api-access-92nl7\" (UniqueName: \"kubernetes.io/projected/3e5078d7-c363-438d-891e-e03666ee4fb4-kube-api-access-92nl7\") pod \"3e5078d7-c363-438d-891e-e03666ee4fb4\" (UID: \"3e5078d7-c363-438d-891e-e03666ee4fb4\") " Nov 03 03:14:35.582482 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:35.582456 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e5078d7-c363-438d-891e-e03666ee4fb4-maas-api-openshift-ca" pod "3e5078d7-c363-438d-891e-e03666ee4fb4" (UID: "3e5078d7-c363-438d-891e-e03666ee4fb4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:14:35.584225 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:35.584199 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e5078d7-c363-438d-891e-e03666ee4fb4-kube-api-access-92nl7" pod "3e5078d7-c363-438d-891e-e03666ee4fb4" (UID: "3e5078d7-c363-438d-891e-e03666ee4fb4"). InnerVolumeSpecName "kube-api-access-92nl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:14:35.584398 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:35.584353 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e5078d7-c363-438d-891e-e03666ee4fb4-maas-api-xks-ca" pod "3e5078d7-c363-438d-891e-e03666ee4fb4" (UID: "3e5078d7-c363-438d-891e-e03666ee4fb4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:14:35.683063 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:35.682986 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e5078d7-c363-438d-891e-e03666ee4fb4-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:14:35.683063 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:35.683015 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e5078d7-c363-438d-891e-e03666ee4fb4-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:14:35.683063 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:35.683026 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92nl7\" (UniqueName: \"kubernetes.io/projected/3e5078d7-c363-438d-891e-e03666ee4fb4-kube-api-access-92nl7\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:14:36.351034 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:36.350999 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-796c034d-4lksf" event={"ID":"3e5078d7-c363-438d-891e-e03666ee4fb4","Type":"ContainerDied","Data":"3e4d842c9721762a83a5b7535cdd4959dfcd7dde111fe201303f2eccd57efb6d"} Nov 03 03:14:36.351219 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:36.351035 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-796c034d-4lksf" Nov 03 03:14:36.351219 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:36.351051 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e4d842c9721762a83a5b7535cdd4959dfcd7dde111fe201303f2eccd57efb6d" Nov 03 03:14:39.781803 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:39.781733 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-521f1f/premium-w2-521f1f-kserve-7578fb7957-89wfb" podUID="3d988573-d8b1-43ba-a423-51a63ca12337" containerName="main" containerID="cri-o://5a4a1e2f703733cb114e00e6a72cc2efdcf3f7c20aa0fd52ed7f7a9346ad1600" gracePeriod=45 Nov 03 03:14:39.785388 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:39.785338 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-521f1f/simulator-w2-521f1f-kserve-5f47b86f6-mgksf" podUID="6e1b7fda-0ade-4042-ad63-5115bc66b454" containerName="main" containerID="cri-o://c59287da96ff090ec1aaca274904ed223bc1b7851441c352b29e106b29a67337" gracePeriod=45 Nov 03 03:14:39.803911 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:39.803884 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-521f1f/simulator-w2-521f1f-kserve-5f47b86f6-mgksf" podUID="6e1b7fda-0ade-4042-ad63-5115bc66b454" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": dial tcp 10.134.0.37:8000: connect: connection refused" Nov 03 03:14:39.903624 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:39.903593 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-521f1f/premium-w2-521f1f-kserve-7578fb7957-89wfb" podUID="3d988573-d8b1-43ba-a423-51a63ca12337" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": dial tcp 10.134.0.39:8000: connect: connection refused" Nov 03 03:14:40.027906 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:40.027879 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-521f1f/premium-w2-521f1f-kserve-7578fb7957-89wfb" Nov 03 03:14:40.053455 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:40.053435 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-521f1f/simulator-w2-521f1f-kserve-5f47b86f6-mgksf" Nov 03 03:14:40.116768 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:40.116745 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d988573-d8b1-43ba-a423-51a63ca12337-home\" (UniqueName: \"kubernetes.io/empty-dir/3d988573-d8b1-43ba-a423-51a63ca12337-home\") pod \"3d988573-d8b1-43ba-a423-51a63ca12337\" (UID: \"3d988573-d8b1-43ba-a423-51a63ca12337\") " Nov 03 03:14:40.116897 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:40.116782 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d988573-d8b1-43ba-a423-51a63ca12337-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d988573-d8b1-43ba-a423-51a63ca12337-dshm\") pod \"3d988573-d8b1-43ba-a423-51a63ca12337\" (UID: \"3d988573-d8b1-43ba-a423-51a63ca12337\") " Nov 03 03:14:40.116897 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:40.116809 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d988573-d8b1-43ba-a423-51a63ca12337-kube-api-access-dxbg2\" (UniqueName: \"kubernetes.io/projected/3d988573-d8b1-43ba-a423-51a63ca12337-kube-api-access-dxbg2\") pod \"3d988573-d8b1-43ba-a423-51a63ca12337\" (UID: \"3d988573-d8b1-43ba-a423-51a63ca12337\") " Nov 03 03:14:40.116897 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:40.116843 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d988573-d8b1-43ba-a423-51a63ca12337-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d988573-d8b1-43ba-a423-51a63ca12337-tmp-dir\") pod \"3d988573-d8b1-43ba-a423-51a63ca12337\" (UID: \"3d988573-d8b1-43ba-a423-51a63ca12337\") " Nov 03 03:14:40.116897 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:40.116863 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d988573-d8b1-43ba-a423-51a63ca12337-tls-certs\" (UniqueName: \"kubernetes.io/secret/3d988573-d8b1-43ba-a423-51a63ca12337-tls-certs\") pod \"3d988573-d8b1-43ba-a423-51a63ca12337\" (UID: \"3d988573-d8b1-43ba-a423-51a63ca12337\") " Nov 03 03:14:40.117091 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:40.116926 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d988573-d8b1-43ba-a423-51a63ca12337-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d988573-d8b1-43ba-a423-51a63ca12337-model-cache\") pod \"3d988573-d8b1-43ba-a423-51a63ca12337\" (UID: \"3d988573-d8b1-43ba-a423-51a63ca12337\") " Nov 03 03:14:40.117091 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:40.117005 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d988573-d8b1-43ba-a423-51a63ca12337-home" pod "3d988573-d8b1-43ba-a423-51a63ca12337" (UID: "3d988573-d8b1-43ba-a423-51a63ca12337"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:14:40.117215 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:40.117189 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d988573-d8b1-43ba-a423-51a63ca12337-tmp-dir" pod "3d988573-d8b1-43ba-a423-51a63ca12337" (UID: "3d988573-d8b1-43ba-a423-51a63ca12337"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:14:40.117273 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:40.117228 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d988573-d8b1-43ba-a423-51a63ca12337-model-cache" pod "3d988573-d8b1-43ba-a423-51a63ca12337" (UID: "3d988573-d8b1-43ba-a423-51a63ca12337"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:14:40.117273 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:40.117252 5605 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d988573-d8b1-43ba-a423-51a63ca12337-home\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:14:40.118958 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:40.118935 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d988573-d8b1-43ba-a423-51a63ca12337-dshm" pod "3d988573-d8b1-43ba-a423-51a63ca12337" (UID: "3d988573-d8b1-43ba-a423-51a63ca12337"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:14:40.119070 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:40.118974 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d988573-d8b1-43ba-a423-51a63ca12337-tls-certs" pod "3d988573-d8b1-43ba-a423-51a63ca12337" (UID: "3d988573-d8b1-43ba-a423-51a63ca12337"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:14:40.119070 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:40.118998 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d988573-d8b1-43ba-a423-51a63ca12337-kube-api-access-dxbg2" pod "3d988573-d8b1-43ba-a423-51a63ca12337" (UID: "3d988573-d8b1-43ba-a423-51a63ca12337"). InnerVolumeSpecName "kube-api-access-dxbg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:14:40.218276 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:40.218246 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e1b7fda-0ade-4042-ad63-5115bc66b454-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e1b7fda-0ade-4042-ad63-5115bc66b454-model-cache\") pod \"6e1b7fda-0ade-4042-ad63-5115bc66b454\" (UID: \"6e1b7fda-0ade-4042-ad63-5115bc66b454\") " Nov 03 03:14:40.218481 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:40.218315 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e1b7fda-0ade-4042-ad63-5115bc66b454-tls-certs\" (UniqueName: \"kubernetes.io/secret/6e1b7fda-0ade-4042-ad63-5115bc66b454-tls-certs\") pod \"6e1b7fda-0ade-4042-ad63-5115bc66b454\" (UID: \"6e1b7fda-0ade-4042-ad63-5115bc66b454\") " Nov 03 03:14:40.218481 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:40.218344 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e1b7fda-0ade-4042-ad63-5115bc66b454-kube-api-access-52p49\" (UniqueName: \"kubernetes.io/projected/6e1b7fda-0ade-4042-ad63-5115bc66b454-kube-api-access-52p49\") pod \"6e1b7fda-0ade-4042-ad63-5115bc66b454\" (UID: \"6e1b7fda-0ade-4042-ad63-5115bc66b454\") " Nov 03 03:14:40.218481 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:40.218385 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e1b7fda-0ade-4042-ad63-5115bc66b454-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e1b7fda-0ade-4042-ad63-5115bc66b454-dshm\") pod \"6e1b7fda-0ade-4042-ad63-5115bc66b454\" (UID: \"6e1b7fda-0ade-4042-ad63-5115bc66b454\") " Nov 03 03:14:40.218481 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:40.218415 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e1b7fda-0ade-4042-ad63-5115bc66b454-home\" (UniqueName: \"kubernetes.io/empty-dir/6e1b7fda-0ade-4042-ad63-5115bc66b454-home\") pod \"6e1b7fda-0ade-4042-ad63-5115bc66b454\" (UID: \"6e1b7fda-0ade-4042-ad63-5115bc66b454\") " Nov 03 03:14:40.218728 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:40.218505 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e1b7fda-0ade-4042-ad63-5115bc66b454-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e1b7fda-0ade-4042-ad63-5115bc66b454-tmp-dir\") pod \"6e1b7fda-0ade-4042-ad63-5115bc66b454\" (UID: \"6e1b7fda-0ade-4042-ad63-5115bc66b454\") " Nov 03 03:14:40.218728 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:40.218666 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxbg2\" (UniqueName: \"kubernetes.io/projected/3d988573-d8b1-43ba-a423-51a63ca12337-kube-api-access-dxbg2\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:14:40.218728 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:40.218678 5605 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d988573-d8b1-43ba-a423-51a63ca12337-tmp-dir\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:14:40.218728 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:40.218690 5605 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d988573-d8b1-43ba-a423-51a63ca12337-tls-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:14:40.218728 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:40.218703 5605 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d988573-d8b1-43ba-a423-51a63ca12337-model-cache\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:14:40.218728 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:40.218716 5605 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d988573-d8b1-43ba-a423-51a63ca12337-dshm\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:14:40.218998 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:40.218927 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e1b7fda-0ade-4042-ad63-5115bc66b454-tmp-dir" pod "6e1b7fda-0ade-4042-ad63-5115bc66b454" (UID: "6e1b7fda-0ade-4042-ad63-5115bc66b454"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:14:40.219094 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:40.219069 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e1b7fda-0ade-4042-ad63-5115bc66b454-model-cache" pod "6e1b7fda-0ade-4042-ad63-5115bc66b454" (UID: "6e1b7fda-0ade-4042-ad63-5115bc66b454"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:14:40.219167 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:40.219097 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e1b7fda-0ade-4042-ad63-5115bc66b454-home" pod "6e1b7fda-0ade-4042-ad63-5115bc66b454" (UID: "6e1b7fda-0ade-4042-ad63-5115bc66b454"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:14:40.220826 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:40.220802 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e1b7fda-0ade-4042-ad63-5115bc66b454-dshm" pod "6e1b7fda-0ade-4042-ad63-5115bc66b454" (UID: "6e1b7fda-0ade-4042-ad63-5115bc66b454"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:14:40.221111 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:40.221090 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e1b7fda-0ade-4042-ad63-5115bc66b454-kube-api-access-52p49" pod "6e1b7fda-0ade-4042-ad63-5115bc66b454" (UID: "6e1b7fda-0ade-4042-ad63-5115bc66b454"). InnerVolumeSpecName "kube-api-access-52p49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:14:40.221666 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:40.221646 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e1b7fda-0ade-4042-ad63-5115bc66b454-tls-certs" pod "6e1b7fda-0ade-4042-ad63-5115bc66b454" (UID: "6e1b7fda-0ade-4042-ad63-5115bc66b454"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:14:40.319671 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:40.319642 5605 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e1b7fda-0ade-4042-ad63-5115bc66b454-tmp-dir\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:14:40.319671 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:40.319669 5605 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e1b7fda-0ade-4042-ad63-5115bc66b454-model-cache\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:14:40.319857 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:40.319682 5605 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e1b7fda-0ade-4042-ad63-5115bc66b454-tls-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:14:40.319857 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:40.319697 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52p49\" (UniqueName: \"kubernetes.io/projected/6e1b7fda-0ade-4042-ad63-5115bc66b454-kube-api-access-52p49\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:14:40.319857 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:40.319707 5605 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e1b7fda-0ade-4042-ad63-5115bc66b454-dshm\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:14:40.319857 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:40.319715 5605 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e1b7fda-0ade-4042-ad63-5115bc66b454-home\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:14:40.379218 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:40.379185 5605 generic.go:356] "Generic (PLEG): container finished" podID="3d988573-d8b1-43ba-a423-51a63ca12337" containerID="5a4a1e2f703733cb114e00e6a72cc2efdcf3f7c20aa0fd52ed7f7a9346ad1600" exitCode=0 Nov 03 03:14:40.379410 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:40.379236 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-521f1f/premium-w2-521f1f-kserve-7578fb7957-89wfb" Nov 03 03:14:40.379410 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:40.379275 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-521f1f/premium-w2-521f1f-kserve-7578fb7957-89wfb" event={"ID":"3d988573-d8b1-43ba-a423-51a63ca12337","Type":"ContainerDied","Data":"5a4a1e2f703733cb114e00e6a72cc2efdcf3f7c20aa0fd52ed7f7a9346ad1600"} Nov 03 03:14:40.379410 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:40.379316 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-521f1f/premium-w2-521f1f-kserve-7578fb7957-89wfb" event={"ID":"3d988573-d8b1-43ba-a423-51a63ca12337","Type":"ContainerDied","Data":"395898ff4fb26a6dc43f8c8c285ac2149c776476c94395e6e9982a100273e691"} Nov 03 03:14:40.379410 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:40.379344 5605 scope.go:122] "RemoveContainer" containerID="5a4a1e2f703733cb114e00e6a72cc2efdcf3f7c20aa0fd52ed7f7a9346ad1600" Nov 03 03:14:40.380836 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:40.380811 5605 generic.go:356] "Generic (PLEG): container finished" podID="6e1b7fda-0ade-4042-ad63-5115bc66b454" containerID="c59287da96ff090ec1aaca274904ed223bc1b7851441c352b29e106b29a67337" exitCode=0 Nov 03 03:14:40.380921 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:40.380849 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-521f1f/simulator-w2-521f1f-kserve-5f47b86f6-mgksf" Nov 03 03:14:40.380921 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:40.380893 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-521f1f/simulator-w2-521f1f-kserve-5f47b86f6-mgksf" event={"ID":"6e1b7fda-0ade-4042-ad63-5115bc66b454","Type":"ContainerDied","Data":"c59287da96ff090ec1aaca274904ed223bc1b7851441c352b29e106b29a67337"} Nov 03 03:14:40.381004 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:40.380932 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-521f1f/simulator-w2-521f1f-kserve-5f47b86f6-mgksf" event={"ID":"6e1b7fda-0ade-4042-ad63-5115bc66b454","Type":"ContainerDied","Data":"58ed42aa6e00103394a5308a75f9dcbfd569f40ef257032b4053d34d98fbb453"} Nov 03 03:14:40.390091 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:40.390009 5605 scope.go:122] "RemoveContainer" containerID="5a4a1e2f703733cb114e00e6a72cc2efdcf3f7c20aa0fd52ed7f7a9346ad1600" Nov 03 03:14:40.390396 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:14:40.390351 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a4a1e2f703733cb114e00e6a72cc2efdcf3f7c20aa0fd52ed7f7a9346ad1600\": container with ID starting with 5a4a1e2f703733cb114e00e6a72cc2efdcf3f7c20aa0fd52ed7f7a9346ad1600 not found: ID does not exist" containerID="5a4a1e2f703733cb114e00e6a72cc2efdcf3f7c20aa0fd52ed7f7a9346ad1600" Nov 03 03:14:40.390461 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:40.390406 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a4a1e2f703733cb114e00e6a72cc2efdcf3f7c20aa0fd52ed7f7a9346ad1600"} err="failed to get container status \"5a4a1e2f703733cb114e00e6a72cc2efdcf3f7c20aa0fd52ed7f7a9346ad1600\": rpc error: code = NotFound desc = could not find container \"5a4a1e2f703733cb114e00e6a72cc2efdcf3f7c20aa0fd52ed7f7a9346ad1600\": container with ID starting with 5a4a1e2f703733cb114e00e6a72cc2efdcf3f7c20aa0fd52ed7f7a9346ad1600 not found: ID does not exist" Nov 03 03:14:40.390461 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:40.390424 5605 scope.go:122] "RemoveContainer" containerID="c59287da96ff090ec1aaca274904ed223bc1b7851441c352b29e106b29a67337" Nov 03 03:14:40.397783 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:40.397767 5605 scope.go:122] "RemoveContainer" containerID="c59287da96ff090ec1aaca274904ed223bc1b7851441c352b29e106b29a67337" Nov 03 03:14:40.398018 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:14:40.398000 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c59287da96ff090ec1aaca274904ed223bc1b7851441c352b29e106b29a67337\": container with ID starting with c59287da96ff090ec1aaca274904ed223bc1b7851441c352b29e106b29a67337 not found: ID does not exist" containerID="c59287da96ff090ec1aaca274904ed223bc1b7851441c352b29e106b29a67337" Nov 03 03:14:40.398067 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:40.398023 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c59287da96ff090ec1aaca274904ed223bc1b7851441c352b29e106b29a67337"} err="failed to get container status \"c59287da96ff090ec1aaca274904ed223bc1b7851441c352b29e106b29a67337\": rpc error: code = NotFound desc = could not find container \"c59287da96ff090ec1aaca274904ed223bc1b7851441c352b29e106b29a67337\": container with ID starting with c59287da96ff090ec1aaca274904ed223bc1b7851441c352b29e106b29a67337 not found: ID does not exist" Nov 03 03:14:40.532169 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:40.532149 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-521f1f/simulator-w2-521f1f-kserve-5f47b86f6-mgksf"] Nov 03 03:14:40.536523 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:40.536503 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-521f1f/simulator-w2-521f1f-kserve-5f47b86f6-mgksf"] Nov 03 03:14:40.638898 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:40.638841 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-521f1f/premium-w2-521f1f-kserve-7578fb7957-89wfb"] Nov 03 03:14:40.641609 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:40.641587 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-521f1f/premium-w2-521f1f-kserve-7578fb7957-89wfb"] Nov 03 03:14:40.848037 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:40.847984 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d988573-d8b1-43ba-a423-51a63ca12337" path="/var/lib/kubelet/pods/3d988573-d8b1-43ba-a423-51a63ca12337/volumes" Nov 03 03:14:40.848395 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:40.848379 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e1b7fda-0ade-4042-ad63-5115bc66b454" path="/var/lib/kubelet/pods/6e1b7fda-0ade-4042-ad63-5115bc66b454/volumes" Nov 03 03:14:46.620502 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:46.620469 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:14:53.911283 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:14:53.911249 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:14:58.961626 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:14:58.961591 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:15:00.159825 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:00.159790 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894595-xl94r"] Nov 03 03:15:00.160234 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:00.160097 5605 cpu_manager.go:414] "containerMap: removing container" podUID="3d988573-d8b1-43ba-a423-51a63ca12337" containerName="main" Nov 03 03:15:00.160234 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:00.160108 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d988573-d8b1-43ba-a423-51a63ca12337" containerName="main" Nov 03 03:15:00.160234 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:00.160133 5605 cpu_manager.go:414] "containerMap: removing container" podUID="6e1b7fda-0ade-4042-ad63-5115bc66b454" containerName="main" Nov 03 03:15:00.160234 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:00.160138 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e1b7fda-0ade-4042-ad63-5115bc66b454" containerName="main" Nov 03 03:15:00.160234 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:00.160146 5605 cpu_manager.go:414] "containerMap: removing container" podUID="3e5078d7-c363-438d-891e-e03666ee4fb4" containerName="revoke-keys" Nov 03 03:15:00.160234 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:00.160152 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e5078d7-c363-438d-891e-e03666ee4fb4" containerName="revoke-keys" Nov 03 03:15:00.160234 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:00.160214 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d988573-d8b1-43ba-a423-51a63ca12337" containerName="main" Nov 03 03:15:00.160234 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:00.160226 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e1b7fda-0ade-4042-ad63-5115bc66b454" containerName="main" Nov 03 03:15:00.160234 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:00.160231 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e5078d7-c363-438d-891e-e03666ee4fb4" containerName="revoke-keys" Nov 03 03:15:00.163476 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:00.163455 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894595-xl94r" Nov 03 03:15:00.215180 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:00.215155 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:15:00.261775 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:00.261750 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:15:00.261891 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:00.261871 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 03:15:00.262984 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:00.262962 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmj62\" (UniqueName: \"kubernetes.io/projected/3393ca62-2683-40bb-8890-f739432dc510-kube-api-access-zmj62\") pod \"maas-api-key-cleanup-29894595-xl94r\" (UID: \"3393ca62-2683-40bb-8890-f739432dc510\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894595-xl94r" Nov 03 03:15:00.263045 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:00.263015 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3393ca62-2683-40bb-8890-f739432dc510-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894595-xl94r\" (UID: \"3393ca62-2683-40bb-8890-f739432dc510\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894595-xl94r" Nov 03 03:15:00.263555 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:00.263536 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894595-xl94r"] Nov 03 03:15:00.364353 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:00.364326 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3393ca62-2683-40bb-8890-f739432dc510-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894595-xl94r\" (UID: \"3393ca62-2683-40bb-8890-f739432dc510\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894595-xl94r" Nov 03 03:15:00.364480 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:00.364397 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmj62\" (UniqueName: \"kubernetes.io/projected/3393ca62-2683-40bb-8890-f739432dc510-kube-api-access-zmj62\") pod \"maas-api-key-cleanup-29894595-xl94r\" (UID: \"3393ca62-2683-40bb-8890-f739432dc510\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894595-xl94r" Nov 03 03:15:00.364905 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:00.364886 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3393ca62-2683-40bb-8890-f739432dc510-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894595-xl94r\" (UID: \"3393ca62-2683-40bb-8890-f739432dc510\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894595-xl94r" Nov 03 03:15:00.417786 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:00.417728 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmj62\" (UniqueName: \"kubernetes.io/projected/3393ca62-2683-40bb-8890-f739432dc510-kube-api-access-zmj62\") pod \"maas-api-key-cleanup-29894595-xl94r\" (UID: \"3393ca62-2683-40bb-8890-f739432dc510\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894595-xl94r" Nov 03 03:15:00.476219 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:00.476199 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894595-xl94r" Nov 03 03:15:00.848114 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:00.848083 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894595-xl94r"] Nov 03 03:15:00.852486 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:15:00.852457 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3393ca62_2683_40bb_8890_f739432dc510.slice/crio-0a54ca9ac35946961f4b29980392fd42dfe3573aabedf9a9d944d068a2a14001 WatchSource:0}: Error finding container 0a54ca9ac35946961f4b29980392fd42dfe3573aabedf9a9d944d068a2a14001: Status 404 returned error can't find the container with id 0a54ca9ac35946961f4b29980392fd42dfe3573aabedf9a9d944d068a2a14001 Nov 03 03:15:01.495992 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:01.495945 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894595-xl94r" event={"ID":"3393ca62-2683-40bb-8890-f739432dc510","Type":"ContainerStarted","Data":"8bafa331d4c06cf67cd2c0765258ec5836ffe480554749c8462ee57143dd3d66"} Nov 03 03:15:01.496616 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:01.496025 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894595-xl94r" event={"ID":"3393ca62-2683-40bb-8890-f739432dc510","Type":"ContainerStarted","Data":"0a54ca9ac35946961f4b29980392fd42dfe3573aabedf9a9d944d068a2a14001"} Nov 03 03:15:01.602972 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:01.602918 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894595-xl94r" podStartSLOduration=1.602905333 podStartE2EDuration="1.602905333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:15:01.601457021 +0000 UTC m=+4978.860655175" watchObservedRunningTime="2026-11-03 03:15:01.602905333 +0000 UTC m=+4978.862103487" Nov 03 03:15:02.501856 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:02.501823 5605 generic.go:356] "Generic (PLEG): container finished" podID="3393ca62-2683-40bb-8890-f739432dc510" containerID="8bafa331d4c06cf67cd2c0765258ec5836ffe480554749c8462ee57143dd3d66" exitCode=60 Nov 03 03:15:02.502272 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:02.501904 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894595-xl94r" event={"ID":"3393ca62-2683-40bb-8890-f739432dc510","Type":"ContainerDied","Data":"8bafa331d4c06cf67cd2c0765258ec5836ffe480554749c8462ee57143dd3d66"} Nov 03 03:15:02.502272 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:02.502074 5605 scope.go:122] "RemoveContainer" containerID="8bafa331d4c06cf67cd2c0765258ec5836ffe480554749c8462ee57143dd3d66" Nov 03 03:15:03.509063 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:03.509029 5605 generic.go:356] "Generic (PLEG): container finished" podID="3393ca62-2683-40bb-8890-f739432dc510" containerID="d33b61c6611193a21f1f4465f93c46be43b9a8fa790a125049724840433462c1" exitCode=60 Nov 03 03:15:03.509486 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:03.509067 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894595-xl94r" event={"ID":"3393ca62-2683-40bb-8890-f739432dc510","Type":"ContainerDied","Data":"d33b61c6611193a21f1f4465f93c46be43b9a8fa790a125049724840433462c1"} Nov 03 03:15:03.509486 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:03.509147 5605 scope.go:122] "RemoveContainer" containerID="8bafa331d4c06cf67cd2c0765258ec5836ffe480554749c8462ee57143dd3d66" Nov 03 03:15:03.509486 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:03.509287 5605 scope.go:122] "RemoveContainer" containerID="d33b61c6611193a21f1f4465f93c46be43b9a8fa790a125049724840433462c1" Nov 03 03:15:03.509486 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:15:03.509480 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894595-xl94r_odh-ai-gateway-infra(3393ca62-2683-40bb-8890-f739432dc510)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894595-xl94r" podUID="3393ca62-2683-40bb-8890-f739432dc510" Nov 03 03:15:04.517642 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:04.517609 5605 scope.go:122] "RemoveContainer" containerID="d33b61c6611193a21f1f4465f93c46be43b9a8fa790a125049724840433462c1" Nov 03 03:15:04.518128 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:15:04.517819 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894595-xl94r_odh-ai-gateway-infra(3393ca62-2683-40bb-8890-f739432dc510)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894595-xl94r" podUID="3393ca62-2683-40bb-8890-f739432dc510" Nov 03 03:15:05.423445 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:05.423413 5605 scope.go:122] "RemoveContainer" containerID="114ebee09aab92b2815ce5c7232f54ae0e5aaf077122ae644bc50db162642b7b" Nov 03 03:15:05.431925 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:05.431902 5605 scope.go:122] "RemoveContainer" containerID="1148f4805e7c8510e14cf6ffd3b3d352578f13a0810e32413f37f03a51acaf74" Nov 03 03:15:05.439843 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:05.439822 5605 scope.go:122] "RemoveContainer" containerID="57d83a43860b48d30eff86b838c613ff51c7a59efe27bdb7acb6a8997c7e0288" Nov 03 03:15:10.212873 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:10.212844 5605 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-skgg9_ce015707-8aae-4215-bb4b-6ebb7bef21c5/limitador/0.log" Nov 03 03:15:12.042265 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:12.042230 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-970fa6cc-jsdnt"] Nov 03 03:15:12.088953 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:12.088923 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-970fa6cc-jsdnt"] Nov 03 03:15:12.241083 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:12.241056 5605 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-rbsfv_fc905c8d-4bfa-49ee-b0b7-8c434727e284/manager/0.log" Nov 03 03:15:12.848523 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:12.848492 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffabd9df-96b6-4e1c-913c-cb29cbfbd83a" path="/var/lib/kubelet/pods/ffabd9df-96b6-4e1c-913c-cb29cbfbd83a/volumes" Nov 03 03:15:15.843117 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:15.843092 5605 scope.go:122] "RemoveContainer" containerID="d33b61c6611193a21f1f4465f93c46be43b9a8fa790a125049724840433462c1" Nov 03 03:15:16.572147 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:16.572122 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894595-xl94r" event={"ID":"3393ca62-2683-40bb-8890-f739432dc510","Type":"ContainerStarted","Data":"33922001f8c82c68f07b808494130a43626ea644110d79773ad8b3848d05f6a3"} Nov 03 03:15:16.572528 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:16.572503 5605 scope.go:122] "RemoveContainer" containerID="33922001f8c82c68f07b808494130a43626ea644110d79773ad8b3848d05f6a3" Nov 03 03:15:16.572748 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:15:16.572727 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894595-xl94r_odh-ai-gateway-infra(3393ca62-2683-40bb-8890-f739432dc510)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894595-xl94r" podUID="3393ca62-2683-40bb-8890-f739432dc510" Nov 03 03:15:17.578001 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:17.577966 5605 generic.go:356] "Generic (PLEG): container finished" podID="3393ca62-2683-40bb-8890-f739432dc510" containerID="33922001f8c82c68f07b808494130a43626ea644110d79773ad8b3848d05f6a3" exitCode=60 Nov 03 03:15:17.578482 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:17.578044 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894595-xl94r" event={"ID":"3393ca62-2683-40bb-8890-f739432dc510","Type":"ContainerDied","Data":"33922001f8c82c68f07b808494130a43626ea644110d79773ad8b3848d05f6a3"} Nov 03 03:15:17.578482 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:17.578119 5605 scope.go:122] "RemoveContainer" containerID="d33b61c6611193a21f1f4465f93c46be43b9a8fa790a125049724840433462c1" Nov 03 03:15:17.578482 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:17.578457 5605 scope.go:122] "RemoveContainer" containerID="33922001f8c82c68f07b808494130a43626ea644110d79773ad8b3848d05f6a3" Nov 03 03:15:17.578666 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:15:17.578647 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894595-xl94r_odh-ai-gateway-infra(3393ca62-2683-40bb-8890-f739432dc510)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894595-xl94r" podUID="3393ca62-2683-40bb-8890-f739432dc510" Nov 03 03:15:17.693490 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:17.693466 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894595-xl94r"] Nov 03 03:15:18.707398 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:18.707352 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894595-xl94r" Nov 03 03:15:18.800587 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:18.800564 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3393ca62-2683-40bb-8890-f739432dc510-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3393ca62-2683-40bb-8890-f739432dc510-maas-api-service-ca\") pod \"3393ca62-2683-40bb-8890-f739432dc510\" (UID: \"3393ca62-2683-40bb-8890-f739432dc510\") " Nov 03 03:15:18.800722 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:18.800636 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3393ca62-2683-40bb-8890-f739432dc510-kube-api-access-zmj62\" (UniqueName: \"kubernetes.io/projected/3393ca62-2683-40bb-8890-f739432dc510-kube-api-access-zmj62\") pod \"3393ca62-2683-40bb-8890-f739432dc510\" (UID: \"3393ca62-2683-40bb-8890-f739432dc510\") " Nov 03 03:15:18.800903 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:18.800881 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3393ca62-2683-40bb-8890-f739432dc510-maas-api-service-ca" pod "3393ca62-2683-40bb-8890-f739432dc510" (UID: "3393ca62-2683-40bb-8890-f739432dc510"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:15:18.802538 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:18.802516 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3393ca62-2683-40bb-8890-f739432dc510-kube-api-access-zmj62" pod "3393ca62-2683-40bb-8890-f739432dc510" (UID: "3393ca62-2683-40bb-8890-f739432dc510"). InnerVolumeSpecName "kube-api-access-zmj62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:15:18.901724 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:18.901699 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmj62\" (UniqueName: \"kubernetes.io/projected/3393ca62-2683-40bb-8890-f739432dc510-kube-api-access-zmj62\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:15:18.901724 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:18.901722 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3393ca62-2683-40bb-8890-f739432dc510-maas-api-service-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:15:19.590616 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:19.590586 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894595-xl94r" Nov 03 03:15:19.590616 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:19.590600 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894595-xl94r" event={"ID":"3393ca62-2683-40bb-8890-f739432dc510","Type":"ContainerDied","Data":"0a54ca9ac35946961f4b29980392fd42dfe3573aabedf9a9d944d068a2a14001"} Nov 03 03:15:19.590940 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:19.590655 5605 scope.go:122] "RemoveContainer" containerID="33922001f8c82c68f07b808494130a43626ea644110d79773ad8b3848d05f6a3" Nov 03 03:15:19.743432 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:19.743404 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894595-xl94r"] Nov 03 03:15:19.789849 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:19.789825 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894595-xl94r"] Nov 03 03:15:20.510368 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:20.510335 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg"] Nov 03 03:15:20.510656 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:20.510644 5605 cpu_manager.go:414] "containerMap: removing container" podUID="3393ca62-2683-40bb-8890-f739432dc510" containerName="cleanup" Nov 03 03:15:20.510710 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:20.510657 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3393ca62-2683-40bb-8890-f739432dc510" containerName="cleanup" Nov 03 03:15:20.510710 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:20.510670 5605 cpu_manager.go:414] "containerMap: removing container" podUID="3393ca62-2683-40bb-8890-f739432dc510" containerName="cleanup" Nov 03 03:15:20.510710 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:20.510675 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3393ca62-2683-40bb-8890-f739432dc510" containerName="cleanup" Nov 03 03:15:20.510809 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:20.510731 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="3393ca62-2683-40bb-8890-f739432dc510" containerName="cleanup" Nov 03 03:15:20.510809 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:20.510746 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="3393ca62-2683-40bb-8890-f739432dc510" containerName="cleanup" Nov 03 03:15:20.515051 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:20.515031 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg" Nov 03 03:15:20.563881 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:20.563838 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 03:15:20.612408 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:20.612378 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg"] Nov 03 03:15:20.618353 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:20.617849 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e785f8b2-3071-44c5-a171-e3cd9cf831b5-credential-socket\") pod \"e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg\" (UID: \"e785f8b2-3071-44c5-a171-e3cd9cf831b5\") " pod="openshift-ingress/e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg" Nov 03 03:15:20.618353 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:20.617899 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e785f8b2-3071-44c5-a171-e3cd9cf831b5-workload-socket\") pod \"e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg\" (UID: \"e785f8b2-3071-44c5-a171-e3cd9cf831b5\") " pod="openshift-ingress/e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg" Nov 03 03:15:20.618353 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:20.617956 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8tpd\" (UniqueName: \"kubernetes.io/projected/e785f8b2-3071-44c5-a171-e3cd9cf831b5-kube-api-access-f8tpd\") pod \"e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg\" (UID: \"e785f8b2-3071-44c5-a171-e3cd9cf831b5\") " pod="openshift-ingress/e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg" Nov 03 03:15:20.618353 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:20.617988 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e785f8b2-3071-44c5-a171-e3cd9cf831b5-workload-certs\") pod \"e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg\" (UID: \"e785f8b2-3071-44c5-a171-e3cd9cf831b5\") " pod="openshift-ingress/e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg" Nov 03 03:15:20.618353 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:20.618018 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e785f8b2-3071-44c5-a171-e3cd9cf831b5-istiod-ca-cert\") pod \"e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg\" (UID: \"e785f8b2-3071-44c5-a171-e3cd9cf831b5\") " pod="openshift-ingress/e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg" Nov 03 03:15:20.618353 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:20.618060 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e785f8b2-3071-44c5-a171-e3cd9cf831b5-istio-podinfo\") pod \"e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg\" (UID: \"e785f8b2-3071-44c5-a171-e3cd9cf831b5\") " pod="openshift-ingress/e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg" Nov 03 03:15:20.618353 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:20.618082 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e785f8b2-3071-44c5-a171-e3cd9cf831b5-istio-data\") pod \"e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg\" (UID: \"e785f8b2-3071-44c5-a171-e3cd9cf831b5\") " pod="openshift-ingress/e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg" Nov 03 03:15:20.618353 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:20.618114 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e785f8b2-3071-44c5-a171-e3cd9cf831b5-istio-envoy\") pod \"e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg\" (UID: \"e785f8b2-3071-44c5-a171-e3cd9cf831b5\") " pod="openshift-ingress/e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg" Nov 03 03:15:20.618353 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:20.618148 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e785f8b2-3071-44c5-a171-e3cd9cf831b5-istio-token\") pod \"e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg\" (UID: \"e785f8b2-3071-44c5-a171-e3cd9cf831b5\") " pod="openshift-ingress/e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg" Nov 03 03:15:20.643286 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:20.636562 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-0d474c52-openshift-default-dockercfg-jjxk8\"" Nov 03 03:15:20.719239 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:20.719214 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8tpd\" (UniqueName: \"kubernetes.io/projected/e785f8b2-3071-44c5-a171-e3cd9cf831b5-kube-api-access-f8tpd\") pod \"e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg\" (UID: \"e785f8b2-3071-44c5-a171-e3cd9cf831b5\") " pod="openshift-ingress/e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg" Nov 03 03:15:20.719431 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:20.719246 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e785f8b2-3071-44c5-a171-e3cd9cf831b5-workload-certs\") pod \"e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg\" (UID: \"e785f8b2-3071-44c5-a171-e3cd9cf831b5\") " pod="openshift-ingress/e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg" Nov 03 03:15:20.719431 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:20.719270 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e785f8b2-3071-44c5-a171-e3cd9cf831b5-istiod-ca-cert\") pod \"e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg\" (UID: \"e785f8b2-3071-44c5-a171-e3cd9cf831b5\") " pod="openshift-ingress/e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg" Nov 03 03:15:20.719431 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:20.719295 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e785f8b2-3071-44c5-a171-e3cd9cf831b5-istio-podinfo\") pod \"e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg\" (UID: \"e785f8b2-3071-44c5-a171-e3cd9cf831b5\") " pod="openshift-ingress/e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg" Nov 03 03:15:20.719431 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:20.719331 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e785f8b2-3071-44c5-a171-e3cd9cf831b5-istio-data\") pod \"e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg\" (UID: \"e785f8b2-3071-44c5-a171-e3cd9cf831b5\") " pod="openshift-ingress/e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg" Nov 03 03:15:20.719431 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:20.719382 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e785f8b2-3071-44c5-a171-e3cd9cf831b5-istio-envoy\") pod \"e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg\" (UID: \"e785f8b2-3071-44c5-a171-e3cd9cf831b5\") " pod="openshift-ingress/e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg" Nov 03 03:15:20.719727 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:20.719434 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e785f8b2-3071-44c5-a171-e3cd9cf831b5-istio-token\") pod \"e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg\" (UID: \"e785f8b2-3071-44c5-a171-e3cd9cf831b5\") " pod="openshift-ingress/e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg" Nov 03 03:15:20.719886 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:20.719857 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e785f8b2-3071-44c5-a171-e3cd9cf831b5-istio-data\") pod \"e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg\" (UID: \"e785f8b2-3071-44c5-a171-e3cd9cf831b5\") " pod="openshift-ingress/e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg" Nov 03 03:15:20.719992 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:20.719926 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e785f8b2-3071-44c5-a171-e3cd9cf831b5-workload-certs\") pod \"e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg\" (UID: \"e785f8b2-3071-44c5-a171-e3cd9cf831b5\") " pod="openshift-ingress/e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg" Nov 03 03:15:20.720051 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:20.719985 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e785f8b2-3071-44c5-a171-e3cd9cf831b5-credential-socket\") pod \"e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg\" (UID: \"e785f8b2-3071-44c5-a171-e3cd9cf831b5\") " pod="openshift-ingress/e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg" Nov 03 03:15:20.720051 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:20.720027 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e785f8b2-3071-44c5-a171-e3cd9cf831b5-workload-socket\") pod \"e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg\" (UID: \"e785f8b2-3071-44c5-a171-e3cd9cf831b5\") " pod="openshift-ingress/e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg" Nov 03 03:15:20.720173 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:20.720158 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e785f8b2-3071-44c5-a171-e3cd9cf831b5-istiod-ca-cert\") pod \"e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg\" (UID: \"e785f8b2-3071-44c5-a171-e3cd9cf831b5\") " pod="openshift-ingress/e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg" Nov 03 03:15:20.720285 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:20.720266 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e785f8b2-3071-44c5-a171-e3cd9cf831b5-credential-socket\") pod \"e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg\" (UID: \"e785f8b2-3071-44c5-a171-e3cd9cf831b5\") " pod="openshift-ingress/e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg" Nov 03 03:15:20.720399 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:20.720377 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e785f8b2-3071-44c5-a171-e3cd9cf831b5-workload-socket\") pod \"e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg\" (UID: \"e785f8b2-3071-44c5-a171-e3cd9cf831b5\") " pod="openshift-ingress/e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg" Nov 03 03:15:20.721739 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:20.721719 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e785f8b2-3071-44c5-a171-e3cd9cf831b5-istio-envoy\") pod \"e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg\" (UID: \"e785f8b2-3071-44c5-a171-e3cd9cf831b5\") " pod="openshift-ingress/e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg" Nov 03 03:15:20.721951 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:20.721931 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e785f8b2-3071-44c5-a171-e3cd9cf831b5-istio-podinfo\") pod \"e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg\" (UID: \"e785f8b2-3071-44c5-a171-e3cd9cf831b5\") " pod="openshift-ingress/e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg" Nov 03 03:15:20.775941 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:20.775883 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e785f8b2-3071-44c5-a171-e3cd9cf831b5-istio-token\") pod \"e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg\" (UID: \"e785f8b2-3071-44c5-a171-e3cd9cf831b5\") " pod="openshift-ingress/e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg" Nov 03 03:15:20.776229 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:20.776184 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8tpd\" (UniqueName: \"kubernetes.io/projected/e785f8b2-3071-44c5-a171-e3cd9cf831b5-kube-api-access-f8tpd\") pod \"e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg\" (UID: \"e785f8b2-3071-44c5-a171-e3cd9cf831b5\") " pod="openshift-ingress/e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg" Nov 03 03:15:20.826971 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:20.826941 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg" Nov 03 03:15:20.848535 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:20.848508 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3393ca62-2683-40bb-8890-f739432dc510" path="/var/lib/kubelet/pods/3393ca62-2683-40bb-8890-f739432dc510/volumes" Nov 03 03:15:21.246627 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:21.246597 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg"] Nov 03 03:15:21.252975 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:15:21.252951 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode785f8b2_3071_44c5_a171_e3cd9cf831b5.slice/crio-4fe39193acbb002d1628bef15723a4ece59c055705ba06c0c455e87041055c5e WatchSource:0}: Error finding container 4fe39193acbb002d1628bef15723a4ece59c055705ba06c0c455e87041055c5e: Status 404 returned error can't find the container with id 4fe39193acbb002d1628bef15723a4ece59c055705ba06c0c455e87041055c5e Nov 03 03:15:21.254916 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:21.254885 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 03:15:21.254988 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:21.254949 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 03:15:21.254988 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:21.254978 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 03:15:21.603205 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:21.603166 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg" event={"ID":"e785f8b2-3071-44c5-a171-e3cd9cf831b5","Type":"ContainerStarted","Data":"12b618e539b57777e3d9c2cc4d5b5141d66d70c78d9cec021c16ceef0158e0fa"} Nov 03 03:15:21.603428 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:21.603227 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg" event={"ID":"e785f8b2-3071-44c5-a171-e3cd9cf831b5","Type":"ContainerStarted","Data":"4fe39193acbb002d1628bef15723a4ece59c055705ba06c0c455e87041055c5e"} Nov 03 03:15:21.712414 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:21.712344 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg" podStartSLOduration=1.71233097 podStartE2EDuration="1.71233097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:15:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:15:21.712184236 +0000 UTC m=+4998.971382402" watchObservedRunningTime="2026-11-03 03:15:21.71233097 +0000 UTC m=+4998.971529115" Nov 03 03:15:21.828054 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:21.828024 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg" Nov 03 03:15:22.042716 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:22.042631 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08ba1427-r84b9"] Nov 03 03:15:22.045054 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:22.045031 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08ba1427-r84b9"] Nov 03 03:15:22.832197 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:22.832170 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg" Nov 03 03:15:22.848434 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:22.848403 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1c56256-530d-408e-b532-d54ea4ecfac6" path="/var/lib/kubelet/pods/d1c56256-530d-408e-b532-d54ea4ecfac6/volumes" Nov 03 03:15:23.614436 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:23.614409 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg" Nov 03 03:15:23.615304 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:23.615281 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg" Nov 03 03:15:23.911880 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:15:23.911799 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:15:27.376207 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:27.376172 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0d474c52-556c6597d7-298xk"] Nov 03 03:15:27.376628 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:27.376569 5605 cpu_manager.go:414] "containerMap: removing container" podUID="3393ca62-2683-40bb-8890-f739432dc510" containerName="cleanup" Nov 03 03:15:27.376628 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:27.376583 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3393ca62-2683-40bb-8890-f739432dc510" containerName="cleanup" Nov 03 03:15:27.376705 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:27.376642 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="3393ca62-2683-40bb-8890-f739432dc510" containerName="cleanup" Nov 03 03:15:27.380069 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:27.380052 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d474c52-556c6597d7-298xk" Nov 03 03:15:27.430560 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:27.430538 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-0d474c52\"" Nov 03 03:15:27.472779 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:27.472751 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d64ce3ec-5732-44f9-9183-20ae46505d60-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0d474c52-556c6597d7-298xk\" (UID: \"d64ce3ec-5732-44f9-9183-20ae46505d60\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0d474c52-556c6597d7-298xk" Nov 03 03:15:27.472889 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:27.472787 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drsft\" (UniqueName: \"kubernetes.io/projected/d64ce3ec-5732-44f9-9183-20ae46505d60-kube-api-access-drsft\") pod \"payload-pre-processing-e2e-ait-0d474c52-556c6597d7-298xk\" (UID: \"d64ce3ec-5732-44f9-9183-20ae46505d60\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0d474c52-556c6597d7-298xk" Nov 03 03:15:27.477001 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:27.476983 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-0d474c52-dockercfg-vn25w\"" Nov 03 03:15:27.478500 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:27.478481 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0d474c52-556c6597d7-298xk"] Nov 03 03:15:27.525219 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:27.525192 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0d474c52-5c46747f5c-26kzt"] Nov 03 03:15:27.528610 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:27.528593 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0d474c52-5c46747f5c-26kzt" Nov 03 03:15:27.574110 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:27.574085 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa18475a-f0e7-4e76-a046-74648c4092d7-plugins-config-volume\") pod \"payload-processing-e2e-ait-0d474c52-5c46747f5c-26kzt\" (UID: \"fa18475a-f0e7-4e76-a046-74648c4092d7\") " pod="openshift-ingress/payload-processing-e2e-ait-0d474c52-5c46747f5c-26kzt" Nov 03 03:15:27.574203 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:27.574127 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d64ce3ec-5732-44f9-9183-20ae46505d60-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0d474c52-556c6597d7-298xk\" (UID: \"d64ce3ec-5732-44f9-9183-20ae46505d60\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0d474c52-556c6597d7-298xk" Nov 03 03:15:27.574203 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:27.574156 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drsft\" (UniqueName: \"kubernetes.io/projected/d64ce3ec-5732-44f9-9183-20ae46505d60-kube-api-access-drsft\") pod \"payload-pre-processing-e2e-ait-0d474c52-556c6597d7-298xk\" (UID: \"d64ce3ec-5732-44f9-9183-20ae46505d60\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0d474c52-556c6597d7-298xk" Nov 03 03:15:27.574277 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:27.574240 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn5sm\" (UniqueName: \"kubernetes.io/projected/fa18475a-f0e7-4e76-a046-74648c4092d7-kube-api-access-nn5sm\") pod \"payload-processing-e2e-ait-0d474c52-5c46747f5c-26kzt\" (UID: \"fa18475a-f0e7-4e76-a046-74648c4092d7\") " pod="openshift-ingress/payload-processing-e2e-ait-0d474c52-5c46747f5c-26kzt" Nov 03 03:15:27.574741 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:27.574723 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d64ce3ec-5732-44f9-9183-20ae46505d60-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0d474c52-556c6597d7-298xk\" (UID: \"d64ce3ec-5732-44f9-9183-20ae46505d60\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0d474c52-556c6597d7-298xk" Nov 03 03:15:27.578112 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:27.578087 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0d474c52-5c46747f5c-26kzt"] Nov 03 03:15:27.628057 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:27.627988 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drsft\" (UniqueName: \"kubernetes.io/projected/d64ce3ec-5732-44f9-9183-20ae46505d60-kube-api-access-drsft\") pod \"payload-pre-processing-e2e-ait-0d474c52-556c6597d7-298xk\" (UID: \"d64ce3ec-5732-44f9-9183-20ae46505d60\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0d474c52-556c6597d7-298xk" Nov 03 03:15:27.675621 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:27.675596 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn5sm\" (UniqueName: \"kubernetes.io/projected/fa18475a-f0e7-4e76-a046-74648c4092d7-kube-api-access-nn5sm\") pod \"payload-processing-e2e-ait-0d474c52-5c46747f5c-26kzt\" (UID: \"fa18475a-f0e7-4e76-a046-74648c4092d7\") " pod="openshift-ingress/payload-processing-e2e-ait-0d474c52-5c46747f5c-26kzt" Nov 03 03:15:27.675739 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:27.675632 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa18475a-f0e7-4e76-a046-74648c4092d7-plugins-config-volume\") pod \"payload-processing-e2e-ait-0d474c52-5c46747f5c-26kzt\" (UID: \"fa18475a-f0e7-4e76-a046-74648c4092d7\") " pod="openshift-ingress/payload-processing-e2e-ait-0d474c52-5c46747f5c-26kzt" Nov 03 03:15:27.676290 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:27.676271 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa18475a-f0e7-4e76-a046-74648c4092d7-plugins-config-volume\") pod \"payload-processing-e2e-ait-0d474c52-5c46747f5c-26kzt\" (UID: \"fa18475a-f0e7-4e76-a046-74648c4092d7\") " pod="openshift-ingress/payload-processing-e2e-ait-0d474c52-5c46747f5c-26kzt" Nov 03 03:15:27.692514 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:27.692493 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d474c52-556c6597d7-298xk" Nov 03 03:15:27.730917 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:27.730890 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn5sm\" (UniqueName: \"kubernetes.io/projected/fa18475a-f0e7-4e76-a046-74648c4092d7-kube-api-access-nn5sm\") pod \"payload-processing-e2e-ait-0d474c52-5c46747f5c-26kzt\" (UID: \"fa18475a-f0e7-4e76-a046-74648c4092d7\") " pod="openshift-ingress/payload-processing-e2e-ait-0d474c52-5c46747f5c-26kzt" Nov 03 03:15:27.840325 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:27.840294 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0d474c52-5c46747f5c-26kzt" Nov 03 03:15:28.055216 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:28.055187 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0d474c52-556c6597d7-298xk"] Nov 03 03:15:28.061075 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:15:28.061048 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd64ce3ec_5732_44f9_9183_20ae46505d60.slice/crio-95476826a3dad5fd3662afc4f57ecb5461cddeb736a7d3fe6b0d767b5b5023e0 WatchSource:0}: Error finding container 95476826a3dad5fd3662afc4f57ecb5461cddeb736a7d3fe6b0d767b5b5023e0: Status 404 returned error can't find the container with id 95476826a3dad5fd3662afc4f57ecb5461cddeb736a7d3fe6b0d767b5b5023e0 Nov 03 03:15:28.205790 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:28.205736 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0d474c52-5c46747f5c-26kzt"] Nov 03 03:15:28.210730 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:15:28.210704 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa18475a_f0e7_4e76_a046_74648c4092d7.slice/crio-3575c1cb4c2c794a61ac6835f0ba98fb8c8c770c66a9e137f2c89909f4f20198 WatchSource:0}: Error finding container 3575c1cb4c2c794a61ac6835f0ba98fb8c8c770c66a9e137f2c89909f4f20198: Status 404 returned error can't find the container with id 3575c1cb4c2c794a61ac6835f0ba98fb8c8c770c66a9e137f2c89909f4f20198 Nov 03 03:15:28.639085 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:28.639055 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0d474c52-5c46747f5c-26kzt" event={"ID":"fa18475a-f0e7-4e76-a046-74648c4092d7","Type":"ContainerStarted","Data":"cc0fce323cf43c97e6d38f57cf767dc1c6697a2a14aaa743cb0295fca3a7c0d4"} Nov 03 03:15:28.639543 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:28.639105 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0d474c52-5c46747f5c-26kzt" event={"ID":"fa18475a-f0e7-4e76-a046-74648c4092d7","Type":"ContainerStarted","Data":"3575c1cb4c2c794a61ac6835f0ba98fb8c8c770c66a9e137f2c89909f4f20198"} Nov 03 03:15:28.639543 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:28.639133 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-0d474c52-5c46747f5c-26kzt" Nov 03 03:15:28.640629 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:28.640603 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d474c52-556c6597d7-298xk" event={"ID":"d64ce3ec-5732-44f9-9183-20ae46505d60","Type":"ContainerStarted","Data":"0760a5dfdb09626965d1b69512c653b353fca9b4119175bfabba00f4ec6c7f1e"} Nov 03 03:15:28.640715 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:28.640645 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d474c52-556c6597d7-298xk" event={"ID":"d64ce3ec-5732-44f9-9183-20ae46505d60","Type":"ContainerStarted","Data":"95476826a3dad5fd3662afc4f57ecb5461cddeb736a7d3fe6b0d767b5b5023e0"} Nov 03 03:15:28.640715 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:28.640681 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d474c52-556c6597d7-298xk" Nov 03 03:15:28.746605 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:28.746560 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-0d474c52-5c46747f5c-26kzt" podStartSLOduration=1.746546494 podStartE2EDuration="1.746546494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:15:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:15:28.746012263 +0000 UTC m=+5006.005210414" watchObservedRunningTime="2026-11-03 03:15:28.746546494 +0000 UTC m=+5006.005744651" Nov 03 03:15:28.862578 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:28.862530 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d474c52-556c6597d7-298xk" podStartSLOduration=1.862516055 podStartE2EDuration="1.862516055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:15:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:15:28.860791966 +0000 UTC m=+5006.119990120" watchObservedRunningTime="2026-11-03 03:15:28.862516055 +0000 UTC m=+5006.121714223" Nov 03 03:15:37.834572 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:37.834540 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:15:39.648463 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:39.648436 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d474c52-556c6597d7-298xk" Nov 03 03:15:39.648865 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:39.648499 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-0d474c52-5c46747f5c-26kzt" Nov 03 03:15:46.727104 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:46.727075 5605 generic.go:356] "Generic (PLEG): container finished" podID="e785f8b2-3071-44c5-a171-e3cd9cf831b5" containerID="12b618e539b57777e3d9c2cc4d5b5141d66d70c78d9cec021c16ceef0158e0fa" exitCode=0 Nov 03 03:15:46.727507 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:46.727119 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg" event={"ID":"e785f8b2-3071-44c5-a171-e3cd9cf831b5","Type":"ContainerDied","Data":"12b618e539b57777e3d9c2cc4d5b5141d66d70c78d9cec021c16ceef0158e0fa"} Nov 03 03:15:46.727507 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:46.727411 5605 scope.go:122] "RemoveContainer" containerID="12b618e539b57777e3d9c2cc4d5b5141d66d70c78d9cec021c16ceef0158e0fa" Nov 03 03:15:46.728521 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:46.728491 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 03:15:46.728605 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:46.728563 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 03:15:46.728605 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:46.728594 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 03:15:47.733958 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:47.733924 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg" event={"ID":"e785f8b2-3071-44c5-a171-e3cd9cf831b5","Type":"ContainerStarted","Data":"5ab280075a28ee53efbedde4e779ff708e8c85495eefee39d482e882c4ec8a81"} Nov 03 03:15:47.827813 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:47.827780 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg" Nov 03 03:15:47.827979 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:47.827961 5605 patch_prober.go:29] interesting pod/e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused" start-of-body= Nov 03 03:15:47.828040 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:47.828019 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg" podUID="e785f8b2-3071-44c5-a171-e3cd9cf831b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused" Nov 03 03:15:48.828212 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:48.828177 5605 patch_prober.go:29] interesting pod/e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused" start-of-body= Nov 03 03:15:48.828751 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:48.828234 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg" podUID="e785f8b2-3071-44c5-a171-e3cd9cf831b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused" Nov 03 03:15:49.828149 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:49.828116 5605 patch_prober.go:29] interesting pod/e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused" start-of-body= Nov 03 03:15:49.828331 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:49.828173 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg" podUID="e785f8b2-3071-44c5-a171-e3cd9cf831b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused" Nov 03 03:15:50.754313 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:50.754277 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d474c52-kbr9m"] Nov 03 03:15:50.781575 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:50.781541 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d474c52-kbr9m" Nov 03 03:15:50.811189 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:50.811163 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d474c52-kbr9m"] Nov 03 03:15:50.827389 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:50.827348 5605 patch_prober.go:29] interesting pod/e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused" start-of-body= Nov 03 03:15:50.827508 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:50.827408 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg" podUID="e785f8b2-3071-44c5-a171-e3cd9cf831b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused" Nov 03 03:15:50.877393 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:50.876625 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:15:50.877393 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:50.876717 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:15:50.877393 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:50.876790 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sb4qh\"" Nov 03 03:15:50.877393 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:50.876841 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:15:50.907826 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:50.907800 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzbxq\" (UniqueName: \"kubernetes.io/projected/39c8ea8d-397b-471f-8d0f-5a5009292526-kube-api-access-fzbxq\") pod \"maas-api-revoke-keys-e2e-ait-0d474c52-kbr9m\" (UID: \"39c8ea8d-397b-471f-8d0f-5a5009292526\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d474c52-kbr9m" Nov 03 03:15:50.907948 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:50.907838 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39c8ea8d-397b-471f-8d0f-5a5009292526-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-0d474c52-kbr9m\" (UID: \"39c8ea8d-397b-471f-8d0f-5a5009292526\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d474c52-kbr9m" Nov 03 03:15:50.907999 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:50.907959 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39c8ea8d-397b-471f-8d0f-5a5009292526-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-0d474c52-kbr9m\" (UID: \"39c8ea8d-397b-471f-8d0f-5a5009292526\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d474c52-kbr9m" Nov 03 03:15:51.008752 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:51.008655 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39c8ea8d-397b-471f-8d0f-5a5009292526-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-0d474c52-kbr9m\" (UID: \"39c8ea8d-397b-471f-8d0f-5a5009292526\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d474c52-kbr9m" Nov 03 03:15:51.008752 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:51.008720 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzbxq\" (UniqueName: \"kubernetes.io/projected/39c8ea8d-397b-471f-8d0f-5a5009292526-kube-api-access-fzbxq\") pod \"maas-api-revoke-keys-e2e-ait-0d474c52-kbr9m\" (UID: \"39c8ea8d-397b-471f-8d0f-5a5009292526\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d474c52-kbr9m" Nov 03 03:15:51.008752 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:51.008742 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39c8ea8d-397b-471f-8d0f-5a5009292526-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-0d474c52-kbr9m\" (UID: \"39c8ea8d-397b-471f-8d0f-5a5009292526\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d474c52-kbr9m" Nov 03 03:15:51.009431 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:51.009401 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39c8ea8d-397b-471f-8d0f-5a5009292526-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-0d474c52-kbr9m\" (UID: \"39c8ea8d-397b-471f-8d0f-5a5009292526\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d474c52-kbr9m" Nov 03 03:15:51.011048 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:51.011029 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39c8ea8d-397b-471f-8d0f-5a5009292526-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-0d474c52-kbr9m\" (UID: \"39c8ea8d-397b-471f-8d0f-5a5009292526\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d474c52-kbr9m" Nov 03 03:15:51.071388 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:51.069966 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzbxq\" (UniqueName: \"kubernetes.io/projected/39c8ea8d-397b-471f-8d0f-5a5009292526-kube-api-access-fzbxq\") pod \"maas-api-revoke-keys-e2e-ait-0d474c52-kbr9m\" (UID: \"39c8ea8d-397b-471f-8d0f-5a5009292526\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d474c52-kbr9m" Nov 03 03:15:51.093691 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:51.093657 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d474c52-kbr9m" Nov 03 03:15:51.466438 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:51.462407 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d474c52-kbr9m"] Nov 03 03:15:51.483506 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:15:51.483474 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39c8ea8d_397b_471f_8d0f_5a5009292526.slice/crio-9b55c2a4d60be7b2d071a97256aa88c294dd773d10200855f8ed8347f781d192 WatchSource:0}: Error finding container 9b55c2a4d60be7b2d071a97256aa88c294dd773d10200855f8ed8347f781d192: Status 404 returned error can't find the container with id 9b55c2a4d60be7b2d071a97256aa88c294dd773d10200855f8ed8347f781d192 Nov 03 03:15:51.754226 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:51.754143 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d474c52-kbr9m" event={"ID":"39c8ea8d-397b-471f-8d0f-5a5009292526","Type":"ContainerStarted","Data":"9b55c2a4d60be7b2d071a97256aa88c294dd773d10200855f8ed8347f781d192"} Nov 03 03:15:51.827646 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:51.827611 5605 patch_prober.go:29] interesting pod/e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused" start-of-body= Nov 03 03:15:51.827757 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:51.827659 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg" podUID="e785f8b2-3071-44c5-a171-e3cd9cf831b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused" Nov 03 03:15:52.760652 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:52.760561 5605 generic.go:356] "Generic (PLEG): container finished" podID="39c8ea8d-397b-471f-8d0f-5a5009292526" containerID="1a4ba13e508e0dbec61425dbbf7ccbd920a0ad12cb8df09a1b3bed2be6c59308" exitCode=0 Nov 03 03:15:52.761030 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:52.760645 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d474c52-kbr9m" event={"ID":"39c8ea8d-397b-471f-8d0f-5a5009292526","Type":"ContainerDied","Data":"1a4ba13e508e0dbec61425dbbf7ccbd920a0ad12cb8df09a1b3bed2be6c59308"} Nov 03 03:15:52.831438 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:52.831411 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg" Nov 03 03:15:52.831691 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:52.831676 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg" Nov 03 03:15:52.832439 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:52.832420 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg" Nov 03 03:15:53.890730 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:53.890706 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d474c52-kbr9m" Nov 03 03:15:53.909687 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:15:53.909659 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:15:54.036350 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:54.036285 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39c8ea8d-397b-471f-8d0f-5a5009292526-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39c8ea8d-397b-471f-8d0f-5a5009292526-maas-api-openshift-ca\") pod \"39c8ea8d-397b-471f-8d0f-5a5009292526\" (UID: \"39c8ea8d-397b-471f-8d0f-5a5009292526\") " Nov 03 03:15:54.036350 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:54.036327 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39c8ea8d-397b-471f-8d0f-5a5009292526-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39c8ea8d-397b-471f-8d0f-5a5009292526-maas-api-xks-ca\") pod \"39c8ea8d-397b-471f-8d0f-5a5009292526\" (UID: \"39c8ea8d-397b-471f-8d0f-5a5009292526\") " Nov 03 03:15:54.036571 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:54.036405 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39c8ea8d-397b-471f-8d0f-5a5009292526-kube-api-access-fzbxq\" (UniqueName: \"kubernetes.io/projected/39c8ea8d-397b-471f-8d0f-5a5009292526-kube-api-access-fzbxq\") pod \"39c8ea8d-397b-471f-8d0f-5a5009292526\" (UID: \"39c8ea8d-397b-471f-8d0f-5a5009292526\") " Nov 03 03:15:54.036717 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:54.036697 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39c8ea8d-397b-471f-8d0f-5a5009292526-maas-api-openshift-ca" pod "39c8ea8d-397b-471f-8d0f-5a5009292526" (UID: "39c8ea8d-397b-471f-8d0f-5a5009292526"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:15:54.038300 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:54.038276 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39c8ea8d-397b-471f-8d0f-5a5009292526-maas-api-xks-ca" pod "39c8ea8d-397b-471f-8d0f-5a5009292526" (UID: "39c8ea8d-397b-471f-8d0f-5a5009292526"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:15:54.038443 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:54.038348 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39c8ea8d-397b-471f-8d0f-5a5009292526-kube-api-access-fzbxq" pod "39c8ea8d-397b-471f-8d0f-5a5009292526" (UID: "39c8ea8d-397b-471f-8d0f-5a5009292526"). InnerVolumeSpecName "kube-api-access-fzbxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:15:54.137309 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:54.137285 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39c8ea8d-397b-471f-8d0f-5a5009292526-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:15:54.137309 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:54.137307 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39c8ea8d-397b-471f-8d0f-5a5009292526-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:15:54.137481 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:54.137316 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzbxq\" (UniqueName: \"kubernetes.io/projected/39c8ea8d-397b-471f-8d0f-5a5009292526-kube-api-access-fzbxq\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:15:54.771465 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:54.771440 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d474c52-kbr9m" Nov 03 03:15:54.771620 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:54.771485 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d474c52-kbr9m" event={"ID":"39c8ea8d-397b-471f-8d0f-5a5009292526","Type":"ContainerDied","Data":"9b55c2a4d60be7b2d071a97256aa88c294dd773d10200855f8ed8347f781d192"} Nov 03 03:15:54.771620 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:54.771556 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b55c2a4d60be7b2d071a97256aa88c294dd773d10200855f8ed8347f781d192" Nov 03 03:15:58.042217 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:58.042186 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b83899d3-v4qdb"] Nov 03 03:15:58.088971 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:58.088942 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b83899d3-v4qdb"] Nov 03 03:15:58.847268 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:15:58.847235 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a8641cb-d366-4df0-8ea4-c62b50b77428" path="/var/lib/kubelet/pods/2a8641cb-d366-4df0-8ea4-c62b50b77428/volumes" Nov 03 03:16:02.275371 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:02.275337 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0d474c52-5c46747f5c-26kzt"] Nov 03 03:16:02.382573 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:02.382540 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0d474c52-556c6597d7-298xk"] Nov 03 03:16:07.275864 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:07.275795 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-0d474c52-5c46747f5c-26kzt" podUID="fa18475a-f0e7-4e76-a046-74648c4092d7" containerName="payload-processing" containerID="cri-o://cc0fce323cf43c97e6d38f57cf767dc1c6697a2a14aaa743cb0295fca3a7c0d4" gracePeriod=25 Nov 03 03:16:07.383705 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:07.383656 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d474c52-556c6597d7-298xk" podUID="d64ce3ec-5732-44f9-9183-20ae46505d60" containerName="payload-pre-processing" containerID="cri-o://0760a5dfdb09626965d1b69512c653b353fca9b4119175bfabba00f4ec6c7f1e" gracePeriod=25 Nov 03 03:16:07.531411 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:07.531328 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0d474c52-5c46747f5c-26kzt" Nov 03 03:16:07.534325 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:07.534305 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa18475a-f0e7-4e76-a046-74648c4092d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa18475a-f0e7-4e76-a046-74648c4092d7-plugins-config-volume\") pod \"fa18475a-f0e7-4e76-a046-74648c4092d7\" (UID: \"fa18475a-f0e7-4e76-a046-74648c4092d7\") " Nov 03 03:16:07.534443 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:07.534342 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa18475a-f0e7-4e76-a046-74648c4092d7-kube-api-access-nn5sm\" (UniqueName: \"kubernetes.io/projected/fa18475a-f0e7-4e76-a046-74648c4092d7-kube-api-access-nn5sm\") pod \"fa18475a-f0e7-4e76-a046-74648c4092d7\" (UID: \"fa18475a-f0e7-4e76-a046-74648c4092d7\") " Nov 03 03:16:07.534789 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:07.534757 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa18475a-f0e7-4e76-a046-74648c4092d7-plugins-config-volume" pod "fa18475a-f0e7-4e76-a046-74648c4092d7" (UID: "fa18475a-f0e7-4e76-a046-74648c4092d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:16:07.536454 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:07.536414 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa18475a-f0e7-4e76-a046-74648c4092d7-kube-api-access-nn5sm" pod "fa18475a-f0e7-4e76-a046-74648c4092d7" (UID: "fa18475a-f0e7-4e76-a046-74648c4092d7"). InnerVolumeSpecName "kube-api-access-nn5sm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:16:07.621060 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:07.621034 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d474c52-556c6597d7-298xk" Nov 03 03:16:07.635453 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:07.635418 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d64ce3ec-5732-44f9-9183-20ae46505d60-kube-api-access-drsft\" (UniqueName: \"kubernetes.io/projected/d64ce3ec-5732-44f9-9183-20ae46505d60-kube-api-access-drsft\") pod \"d64ce3ec-5732-44f9-9183-20ae46505d60\" (UID: \"d64ce3ec-5732-44f9-9183-20ae46505d60\") " Nov 03 03:16:07.635618 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:07.635500 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d64ce3ec-5732-44f9-9183-20ae46505d60-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d64ce3ec-5732-44f9-9183-20ae46505d60-plugins-config-volume\") pod \"d64ce3ec-5732-44f9-9183-20ae46505d60\" (UID: \"d64ce3ec-5732-44f9-9183-20ae46505d60\") " Nov 03 03:16:07.635705 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:07.635650 5605 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa18475a-f0e7-4e76-a046-74648c4092d7-plugins-config-volume\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:16:07.635705 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:07.635661 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nn5sm\" (UniqueName: \"kubernetes.io/projected/fa18475a-f0e7-4e76-a046-74648c4092d7-kube-api-access-nn5sm\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:16:07.635824 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:07.635787 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d64ce3ec-5732-44f9-9183-20ae46505d60-plugins-config-volume" pod "d64ce3ec-5732-44f9-9183-20ae46505d60" (UID: "d64ce3ec-5732-44f9-9183-20ae46505d60"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:16:07.637394 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:07.637372 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d64ce3ec-5732-44f9-9183-20ae46505d60-kube-api-access-drsft" pod "d64ce3ec-5732-44f9-9183-20ae46505d60" (UID: "d64ce3ec-5732-44f9-9183-20ae46505d60"). InnerVolumeSpecName "kube-api-access-drsft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:16:07.736125 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:07.736091 5605 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d64ce3ec-5732-44f9-9183-20ae46505d60-plugins-config-volume\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:16:07.736125 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:07.736120 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drsft\" (UniqueName: \"kubernetes.io/projected/d64ce3ec-5732-44f9-9183-20ae46505d60-kube-api-access-drsft\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:16:07.834861 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:07.834827 5605 generic.go:356] "Generic (PLEG): container finished" podID="fa18475a-f0e7-4e76-a046-74648c4092d7" containerID="cc0fce323cf43c97e6d38f57cf767dc1c6697a2a14aaa743cb0295fca3a7c0d4" exitCode=0 Nov 03 03:16:07.835081 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:07.834872 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0d474c52-5c46747f5c-26kzt" Nov 03 03:16:07.835081 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:07.834910 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0d474c52-5c46747f5c-26kzt" event={"ID":"fa18475a-f0e7-4e76-a046-74648c4092d7","Type":"ContainerDied","Data":"cc0fce323cf43c97e6d38f57cf767dc1c6697a2a14aaa743cb0295fca3a7c0d4"} Nov 03 03:16:07.835081 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:07.834964 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0d474c52-5c46747f5c-26kzt" event={"ID":"fa18475a-f0e7-4e76-a046-74648c4092d7","Type":"ContainerDied","Data":"3575c1cb4c2c794a61ac6835f0ba98fb8c8c770c66a9e137f2c89909f4f20198"} Nov 03 03:16:07.835081 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:07.834995 5605 scope.go:122] "RemoveContainer" containerID="cc0fce323cf43c97e6d38f57cf767dc1c6697a2a14aaa743cb0295fca3a7c0d4" Nov 03 03:16:07.836335 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:07.836310 5605 generic.go:356] "Generic (PLEG): container finished" podID="d64ce3ec-5732-44f9-9183-20ae46505d60" containerID="0760a5dfdb09626965d1b69512c653b353fca9b4119175bfabba00f4ec6c7f1e" exitCode=0 Nov 03 03:16:07.836508 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:07.836377 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d474c52-556c6597d7-298xk" Nov 03 03:16:07.836508 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:07.836390 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d474c52-556c6597d7-298xk" event={"ID":"d64ce3ec-5732-44f9-9183-20ae46505d60","Type":"ContainerDied","Data":"0760a5dfdb09626965d1b69512c653b353fca9b4119175bfabba00f4ec6c7f1e"} Nov 03 03:16:07.836508 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:07.836456 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d474c52-556c6597d7-298xk" event={"ID":"d64ce3ec-5732-44f9-9183-20ae46505d60","Type":"ContainerDied","Data":"95476826a3dad5fd3662afc4f57ecb5461cddeb736a7d3fe6b0d767b5b5023e0"} Nov 03 03:16:07.845211 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:07.845195 5605 scope.go:122] "RemoveContainer" containerID="cc0fce323cf43c97e6d38f57cf767dc1c6697a2a14aaa743cb0295fca3a7c0d4" Nov 03 03:16:07.845519 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:16:07.845499 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc0fce323cf43c97e6d38f57cf767dc1c6697a2a14aaa743cb0295fca3a7c0d4\": container with ID starting with cc0fce323cf43c97e6d38f57cf767dc1c6697a2a14aaa743cb0295fca3a7c0d4 not found: ID does not exist" containerID="cc0fce323cf43c97e6d38f57cf767dc1c6697a2a14aaa743cb0295fca3a7c0d4" Nov 03 03:16:07.845588 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:07.845529 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc0fce323cf43c97e6d38f57cf767dc1c6697a2a14aaa743cb0295fca3a7c0d4"} err="failed to get container status \"cc0fce323cf43c97e6d38f57cf767dc1c6697a2a14aaa743cb0295fca3a7c0d4\": rpc error: code = NotFound desc = could not find container \"cc0fce323cf43c97e6d38f57cf767dc1c6697a2a14aaa743cb0295fca3a7c0d4\": container with ID starting with cc0fce323cf43c97e6d38f57cf767dc1c6697a2a14aaa743cb0295fca3a7c0d4 not found: ID does not exist" Nov 03 03:16:07.845588 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:07.845546 5605 scope.go:122] "RemoveContainer" containerID="0760a5dfdb09626965d1b69512c653b353fca9b4119175bfabba00f4ec6c7f1e" Nov 03 03:16:07.853679 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:07.853662 5605 scope.go:122] "RemoveContainer" containerID="0760a5dfdb09626965d1b69512c653b353fca9b4119175bfabba00f4ec6c7f1e" Nov 03 03:16:07.853949 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:16:07.853913 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0760a5dfdb09626965d1b69512c653b353fca9b4119175bfabba00f4ec6c7f1e\": container with ID starting with 0760a5dfdb09626965d1b69512c653b353fca9b4119175bfabba00f4ec6c7f1e not found: ID does not exist" containerID="0760a5dfdb09626965d1b69512c653b353fca9b4119175bfabba00f4ec6c7f1e" Nov 03 03:16:07.854000 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:07.853955 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0760a5dfdb09626965d1b69512c653b353fca9b4119175bfabba00f4ec6c7f1e"} err="failed to get container status \"0760a5dfdb09626965d1b69512c653b353fca9b4119175bfabba00f4ec6c7f1e\": rpc error: code = NotFound desc = could not find container \"0760a5dfdb09626965d1b69512c653b353fca9b4119175bfabba00f4ec6c7f1e\": container with ID starting with 0760a5dfdb09626965d1b69512c653b353fca9b4119175bfabba00f4ec6c7f1e not found: ID does not exist" Nov 03 03:16:07.987871 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:07.987838 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0d474c52-556c6597d7-298xk"] Nov 03 03:16:07.991400 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:07.991350 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0d474c52-556c6597d7-298xk"] Nov 03 03:16:08.042059 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:08.042021 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6c1354a-z88hg"] Nov 03 03:16:08.043670 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:08.043645 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6c1354a-z88hg"] Nov 03 03:16:08.094783 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:08.094693 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0d474c52-5c46747f5c-26kzt"] Nov 03 03:16:08.097721 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:08.097690 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0d474c52-5c46747f5c-26kzt"] Nov 03 03:16:08.847262 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:08.847231 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c62c4b3-d3df-4240-b233-5d38db1acee7" path="/var/lib/kubelet/pods/2c62c4b3-d3df-4240-b233-5d38db1acee7/volumes" Nov 03 03:16:08.847664 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:08.847608 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d64ce3ec-5732-44f9-9183-20ae46505d60" path="/var/lib/kubelet/pods/d64ce3ec-5732-44f9-9183-20ae46505d60/volumes" Nov 03 03:16:08.847916 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:08.847903 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa18475a-f0e7-4e76-a046-74648c4092d7" path="/var/lib/kubelet/pods/fa18475a-f0e7-4e76-a046-74648c4092d7/volumes" Nov 03 03:16:10.682624 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:10.682590 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d474c52-kbr9m"] Nov 03 03:16:10.729491 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:10.729461 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d474c52-kbr9m"] Nov 03 03:16:10.848867 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:10.848791 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39c8ea8d-397b-471f-8d0f-5a5009292526" path="/var/lib/kubelet/pods/39c8ea8d-397b-471f-8d0f-5a5009292526/volumes" Nov 03 03:16:11.910159 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:11.910129 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg"] Nov 03 03:16:11.910693 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:11.910335 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg" podUID="e785f8b2-3071-44c5-a171-e3cd9cf831b5" containerName="istio-proxy" containerID="cri-o://5ab280075a28ee53efbedde4e779ff708e8c85495eefee39d482e882c4ec8a81" gracePeriod=30 Nov 03 03:16:14.616611 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:14.616574 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:16:17.148054 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:17.148031 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg" Nov 03 03:16:17.209380 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:17.209310 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e785f8b2-3071-44c5-a171-e3cd9cf831b5-kube-api-access-f8tpd\" (UniqueName: \"kubernetes.io/projected/e785f8b2-3071-44c5-a171-e3cd9cf831b5-kube-api-access-f8tpd\") pod \"e785f8b2-3071-44c5-a171-e3cd9cf831b5\" (UID: \"e785f8b2-3071-44c5-a171-e3cd9cf831b5\") " Nov 03 03:16:17.209380 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:17.209346 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e785f8b2-3071-44c5-a171-e3cd9cf831b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e785f8b2-3071-44c5-a171-e3cd9cf831b5-istio-podinfo\") pod \"e785f8b2-3071-44c5-a171-e3cd9cf831b5\" (UID: \"e785f8b2-3071-44c5-a171-e3cd9cf831b5\") " Nov 03 03:16:17.209517 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:17.209385 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e785f8b2-3071-44c5-a171-e3cd9cf831b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e785f8b2-3071-44c5-a171-e3cd9cf831b5-istio-data\") pod \"e785f8b2-3071-44c5-a171-e3cd9cf831b5\" (UID: \"e785f8b2-3071-44c5-a171-e3cd9cf831b5\") " Nov 03 03:16:17.209575 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:17.209556 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e785f8b2-3071-44c5-a171-e3cd9cf831b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e785f8b2-3071-44c5-a171-e3cd9cf831b5-credential-socket\") pod \"e785f8b2-3071-44c5-a171-e3cd9cf831b5\" (UID: \"e785f8b2-3071-44c5-a171-e3cd9cf831b5\") " Nov 03 03:16:17.209617 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:17.209600 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e785f8b2-3071-44c5-a171-e3cd9cf831b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e785f8b2-3071-44c5-a171-e3cd9cf831b5-workload-socket\") pod \"e785f8b2-3071-44c5-a171-e3cd9cf831b5\" (UID: \"e785f8b2-3071-44c5-a171-e3cd9cf831b5\") " Nov 03 03:16:17.209664 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:17.209635 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e785f8b2-3071-44c5-a171-e3cd9cf831b5-istio-data" pod "e785f8b2-3071-44c5-a171-e3cd9cf831b5" (UID: "e785f8b2-3071-44c5-a171-e3cd9cf831b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:16:17.209664 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:17.209648 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e785f8b2-3071-44c5-a171-e3cd9cf831b5-istio-token\" (UniqueName: \"kubernetes.io/projected/e785f8b2-3071-44c5-a171-e3cd9cf831b5-istio-token\") pod \"e785f8b2-3071-44c5-a171-e3cd9cf831b5\" (UID: \"e785f8b2-3071-44c5-a171-e3cd9cf831b5\") " Nov 03 03:16:17.209757 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:17.209678 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e785f8b2-3071-44c5-a171-e3cd9cf831b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e785f8b2-3071-44c5-a171-e3cd9cf831b5-istio-envoy\") pod \"e785f8b2-3071-44c5-a171-e3cd9cf831b5\" (UID: \"e785f8b2-3071-44c5-a171-e3cd9cf831b5\") " Nov 03 03:16:17.209757 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:17.209714 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e785f8b2-3071-44c5-a171-e3cd9cf831b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e785f8b2-3071-44c5-a171-e3cd9cf831b5-istiod-ca-cert\") pod \"e785f8b2-3071-44c5-a171-e3cd9cf831b5\" (UID: \"e785f8b2-3071-44c5-a171-e3cd9cf831b5\") " Nov 03 03:16:17.209867 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:17.209757 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e785f8b2-3071-44c5-a171-e3cd9cf831b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e785f8b2-3071-44c5-a171-e3cd9cf831b5-workload-certs\") pod \"e785f8b2-3071-44c5-a171-e3cd9cf831b5\" (UID: \"e785f8b2-3071-44c5-a171-e3cd9cf831b5\") " Nov 03 03:16:17.209867 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:17.209798 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e785f8b2-3071-44c5-a171-e3cd9cf831b5-credential-socket" pod "e785f8b2-3071-44c5-a171-e3cd9cf831b5" (UID: "e785f8b2-3071-44c5-a171-e3cd9cf831b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:16:17.209867 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:17.209808 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e785f8b2-3071-44c5-a171-e3cd9cf831b5-workload-socket" pod "e785f8b2-3071-44c5-a171-e3cd9cf831b5" (UID: "e785f8b2-3071-44c5-a171-e3cd9cf831b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:16:17.210089 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:17.210068 5605 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e785f8b2-3071-44c5-a171-e3cd9cf831b5-workload-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:16:17.210155 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:17.210094 5605 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e785f8b2-3071-44c5-a171-e3cd9cf831b5-istio-data\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:16:17.210155 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:17.210107 5605 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e785f8b2-3071-44c5-a171-e3cd9cf831b5-credential-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:16:17.210155 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:17.210118 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e785f8b2-3071-44c5-a171-e3cd9cf831b5-workload-certs" pod "e785f8b2-3071-44c5-a171-e3cd9cf831b5" (UID: "e785f8b2-3071-44c5-a171-e3cd9cf831b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:16:17.210155 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:17.210148 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e785f8b2-3071-44c5-a171-e3cd9cf831b5-istiod-ca-cert" pod "e785f8b2-3071-44c5-a171-e3cd9cf831b5" (UID: "e785f8b2-3071-44c5-a171-e3cd9cf831b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:16:17.211730 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:17.211706 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e785f8b2-3071-44c5-a171-e3cd9cf831b5-istio-envoy" pod "e785f8b2-3071-44c5-a171-e3cd9cf831b5" (UID: "e785f8b2-3071-44c5-a171-e3cd9cf831b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:16:17.211835 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:17.211740 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e785f8b2-3071-44c5-a171-e3cd9cf831b5-istio-token" pod "e785f8b2-3071-44c5-a171-e3cd9cf831b5" (UID: "e785f8b2-3071-44c5-a171-e3cd9cf831b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:16:17.211904 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:17.211856 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e785f8b2-3071-44c5-a171-e3cd9cf831b5-istio-podinfo" pod "e785f8b2-3071-44c5-a171-e3cd9cf831b5" (UID: "e785f8b2-3071-44c5-a171-e3cd9cf831b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:16:17.211904 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:17.211879 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e785f8b2-3071-44c5-a171-e3cd9cf831b5-kube-api-access-f8tpd" pod "e785f8b2-3071-44c5-a171-e3cd9cf831b5" (UID: "e785f8b2-3071-44c5-a171-e3cd9cf831b5"). InnerVolumeSpecName "kube-api-access-f8tpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:16:17.311247 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:17.311171 5605 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e785f8b2-3071-44c5-a171-e3cd9cf831b5-istio-podinfo\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:16:17.311247 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:17.311204 5605 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e785f8b2-3071-44c5-a171-e3cd9cf831b5-istio-token\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:16:17.311247 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:17.311216 5605 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e785f8b2-3071-44c5-a171-e3cd9cf831b5-istio-envoy\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:16:17.311247 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:17.311228 5605 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e785f8b2-3071-44c5-a171-e3cd9cf831b5-istiod-ca-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:16:17.311247 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:17.311241 5605 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e785f8b2-3071-44c5-a171-e3cd9cf831b5-workload-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:16:17.311565 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:17.311253 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8tpd\" (UniqueName: \"kubernetes.io/projected/e785f8b2-3071-44c5-a171-e3cd9cf831b5-kube-api-access-f8tpd\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:16:17.882596 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:17.882564 5605 generic.go:356] "Generic (PLEG): container finished" podID="e785f8b2-3071-44c5-a171-e3cd9cf831b5" containerID="5ab280075a28ee53efbedde4e779ff708e8c85495eefee39d482e882c4ec8a81" exitCode=0 Nov 03 03:16:17.882762 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:17.882604 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg" Nov 03 03:16:17.882762 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:17.882605 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg" event={"ID":"e785f8b2-3071-44c5-a171-e3cd9cf831b5","Type":"ContainerDied","Data":"5ab280075a28ee53efbedde4e779ff708e8c85495eefee39d482e882c4ec8a81"} Nov 03 03:16:17.882762 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:17.882728 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg" event={"ID":"e785f8b2-3071-44c5-a171-e3cd9cf831b5","Type":"ContainerDied","Data":"4fe39193acbb002d1628bef15723a4ece59c055705ba06c0c455e87041055c5e"} Nov 03 03:16:17.882762 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:17.882755 5605 scope.go:122] "RemoveContainer" containerID="5ab280075a28ee53efbedde4e779ff708e8c85495eefee39d482e882c4ec8a81" Nov 03 03:16:17.892169 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:17.892151 5605 scope.go:122] "RemoveContainer" containerID="12b618e539b57777e3d9c2cc4d5b5141d66d70c78d9cec021c16ceef0158e0fa" Nov 03 03:16:17.899540 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:17.899522 5605 scope.go:122] "RemoveContainer" containerID="5ab280075a28ee53efbedde4e779ff708e8c85495eefee39d482e882c4ec8a81" Nov 03 03:16:17.899801 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:16:17.899784 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ab280075a28ee53efbedde4e779ff708e8c85495eefee39d482e882c4ec8a81\": container with ID starting with 5ab280075a28ee53efbedde4e779ff708e8c85495eefee39d482e882c4ec8a81 not found: ID does not exist" containerID="5ab280075a28ee53efbedde4e779ff708e8c85495eefee39d482e882c4ec8a81" Nov 03 03:16:17.899880 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:17.899812 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ab280075a28ee53efbedde4e779ff708e8c85495eefee39d482e882c4ec8a81"} err="failed to get container status \"5ab280075a28ee53efbedde4e779ff708e8c85495eefee39d482e882c4ec8a81\": rpc error: code = NotFound desc = could not find container \"5ab280075a28ee53efbedde4e779ff708e8c85495eefee39d482e882c4ec8a81\": container with ID starting with 5ab280075a28ee53efbedde4e779ff708e8c85495eefee39d482e882c4ec8a81 not found: ID does not exist" Nov 03 03:16:17.899880 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:17.899833 5605 scope.go:122] "RemoveContainer" containerID="12b618e539b57777e3d9c2cc4d5b5141d66d70c78d9cec021c16ceef0158e0fa" Nov 03 03:16:17.900069 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:16:17.900049 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12b618e539b57777e3d9c2cc4d5b5141d66d70c78d9cec021c16ceef0158e0fa\": container with ID starting with 12b618e539b57777e3d9c2cc4d5b5141d66d70c78d9cec021c16ceef0158e0fa not found: ID does not exist" containerID="12b618e539b57777e3d9c2cc4d5b5141d66d70c78d9cec021c16ceef0158e0fa" Nov 03 03:16:17.900106 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:17.900075 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12b618e539b57777e3d9c2cc4d5b5141d66d70c78d9cec021c16ceef0158e0fa"} err="failed to get container status \"12b618e539b57777e3d9c2cc4d5b5141d66d70c78d9cec021c16ceef0158e0fa\": rpc error: code = NotFound desc = could not find container \"12b618e539b57777e3d9c2cc4d5b5141d66d70c78d9cec021c16ceef0158e0fa\": container with ID starting with 12b618e539b57777e3d9c2cc4d5b5141d66d70c78d9cec021c16ceef0158e0fa not found: ID does not exist" Nov 03 03:16:18.082184 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:18.082159 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg"] Nov 03 03:16:18.128530 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:18.128509 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-0d474c52-openshift-default-6b6b99d9c9-fgxxg"] Nov 03 03:16:18.526803 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:18.526778 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-594d9845fc-skkgb"] Nov 03 03:16:18.527149 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:18.527042 5605 cpu_manager.go:414] "containerMap: removing container" podUID="d64ce3ec-5732-44f9-9183-20ae46505d60" containerName="payload-pre-processing" Nov 03 03:16:18.527149 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:18.527052 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d64ce3ec-5732-44f9-9183-20ae46505d60" containerName="payload-pre-processing" Nov 03 03:16:18.527149 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:18.527065 5605 cpu_manager.go:414] "containerMap: removing container" podUID="39c8ea8d-397b-471f-8d0f-5a5009292526" containerName="revoke-keys" Nov 03 03:16:18.527149 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:18.527071 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39c8ea8d-397b-471f-8d0f-5a5009292526" containerName="revoke-keys" Nov 03 03:16:18.527149 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:18.527086 5605 cpu_manager.go:414] "containerMap: removing container" podUID="e785f8b2-3071-44c5-a171-e3cd9cf831b5" containerName="istio-proxy" Nov 03 03:16:18.527149 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:18.527091 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e785f8b2-3071-44c5-a171-e3cd9cf831b5" containerName="istio-proxy" Nov 03 03:16:18.527149 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:18.527103 5605 cpu_manager.go:414] "containerMap: removing container" podUID="fa18475a-f0e7-4e76-a046-74648c4092d7" containerName="payload-processing" Nov 03 03:16:18.527149 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:18.527108 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa18475a-f0e7-4e76-a046-74648c4092d7" containerName="payload-processing" Nov 03 03:16:18.527149 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:18.527121 5605 cpu_manager.go:414] "containerMap: removing container" podUID="e785f8b2-3071-44c5-a171-e3cd9cf831b5" containerName="istio-proxy" Nov 03 03:16:18.527149 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:18.527125 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e785f8b2-3071-44c5-a171-e3cd9cf831b5" containerName="istio-proxy" Nov 03 03:16:18.527537 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:18.527174 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="e785f8b2-3071-44c5-a171-e3cd9cf831b5" containerName="istio-proxy" Nov 03 03:16:18.527537 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:18.527182 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="d64ce3ec-5732-44f9-9183-20ae46505d60" containerName="payload-pre-processing" Nov 03 03:16:18.527537 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:18.527190 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="39c8ea8d-397b-471f-8d0f-5a5009292526" containerName="revoke-keys" Nov 03 03:16:18.527537 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:18.527197 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa18475a-f0e7-4e76-a046-74648c4092d7" containerName="payload-processing" Nov 03 03:16:18.531452 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:18.531436 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-594d9845fc-skkgb" Nov 03 03:16:18.583513 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:18.583489 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 03:16:18.583626 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:18.583537 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-whxgk\"" Nov 03 03:16:18.620112 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:18.620089 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mmn4\" (UniqueName: \"kubernetes.io/projected/770a6de2-28ed-4d95-b766-4a49792bdf4b-kube-api-access-5mmn4\") pod \"maas-controller-594d9845fc-skkgb\" (UID: \"770a6de2-28ed-4d95-b766-4a49792bdf4b\") " pod="opendatahub/maas-controller-594d9845fc-skkgb" Nov 03 03:16:18.620238 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:18.620132 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/770a6de2-28ed-4d95-b766-4a49792bdf4b-webhook-cert\") pod \"maas-controller-594d9845fc-skkgb\" (UID: \"770a6de2-28ed-4d95-b766-4a49792bdf4b\") " pod="opendatahub/maas-controller-594d9845fc-skkgb" Nov 03 03:16:18.620306 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:18.620243 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/770a6de2-28ed-4d95-b766-4a49792bdf4b-metrics-certs\") pod \"maas-controller-594d9845fc-skkgb\" (UID: \"770a6de2-28ed-4d95-b766-4a49792bdf4b\") " pod="opendatahub/maas-controller-594d9845fc-skkgb" Nov 03 03:16:18.629330 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:18.629308 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 03:16:18.629458 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:18.629442 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 03:16:18.630804 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:18.630789 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-594d9845fc-skkgb"] Nov 03 03:16:18.720713 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:18.720686 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mmn4\" (UniqueName: \"kubernetes.io/projected/770a6de2-28ed-4d95-b766-4a49792bdf4b-kube-api-access-5mmn4\") pod \"maas-controller-594d9845fc-skkgb\" (UID: \"770a6de2-28ed-4d95-b766-4a49792bdf4b\") " pod="opendatahub/maas-controller-594d9845fc-skkgb" Nov 03 03:16:18.720878 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:18.720723 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/770a6de2-28ed-4d95-b766-4a49792bdf4b-webhook-cert\") pod \"maas-controller-594d9845fc-skkgb\" (UID: \"770a6de2-28ed-4d95-b766-4a49792bdf4b\") " pod="opendatahub/maas-controller-594d9845fc-skkgb" Nov 03 03:16:18.720878 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:18.720778 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/770a6de2-28ed-4d95-b766-4a49792bdf4b-metrics-certs\") pod \"maas-controller-594d9845fc-skkgb\" (UID: \"770a6de2-28ed-4d95-b766-4a49792bdf4b\") " pod="opendatahub/maas-controller-594d9845fc-skkgb" Nov 03 03:16:18.723119 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:18.723097 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/770a6de2-28ed-4d95-b766-4a49792bdf4b-metrics-certs\") pod \"maas-controller-594d9845fc-skkgb\" (UID: \"770a6de2-28ed-4d95-b766-4a49792bdf4b\") " pod="opendatahub/maas-controller-594d9845fc-skkgb" Nov 03 03:16:18.723247 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:18.723228 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/770a6de2-28ed-4d95-b766-4a49792bdf4b-webhook-cert\") pod \"maas-controller-594d9845fc-skkgb\" (UID: \"770a6de2-28ed-4d95-b766-4a49792bdf4b\") " pod="opendatahub/maas-controller-594d9845fc-skkgb" Nov 03 03:16:18.774968 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:18.774947 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mmn4\" (UniqueName: \"kubernetes.io/projected/770a6de2-28ed-4d95-b766-4a49792bdf4b-kube-api-access-5mmn4\") pod \"maas-controller-594d9845fc-skkgb\" (UID: \"770a6de2-28ed-4d95-b766-4a49792bdf4b\") " pod="opendatahub/maas-controller-594d9845fc-skkgb" Nov 03 03:16:18.843459 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:18.843437 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-594d9845fc-skkgb" Nov 03 03:16:18.848938 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:18.848917 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e785f8b2-3071-44c5-a171-e3cd9cf831b5" path="/var/lib/kubelet/pods/e785f8b2-3071-44c5-a171-e3cd9cf831b5/volumes" Nov 03 03:16:19.207480 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:19.207415 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-594d9845fc-skkgb"] Nov 03 03:16:19.213245 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:16:19.213219 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod770a6de2_28ed_4d95_b766_4a49792bdf4b.slice/crio-4e36b830e39d500e8c183877cd30944e4501ed03902f92522a8ca652e63de043 WatchSource:0}: Error finding container 4e36b830e39d500e8c183877cd30944e4501ed03902f92522a8ca652e63de043: Status 404 returned error can't find the container with id 4e36b830e39d500e8c183877cd30944e4501ed03902f92522a8ca652e63de043 Nov 03 03:16:19.214476 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:19.214457 5605 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 03:16:19.895145 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:19.895112 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-594d9845fc-skkgb" event={"ID":"770a6de2-28ed-4d95-b766-4a49792bdf4b","Type":"ContainerStarted","Data":"4e36b830e39d500e8c183877cd30944e4501ed03902f92522a8ca652e63de043"} Nov 03 03:16:20.049730 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:20.049691 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3787f5e3-vd7cs"] Nov 03 03:16:20.096183 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:20.096159 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3787f5e3-vd7cs"] Nov 03 03:16:20.848265 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:20.848224 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02c94438-cf20-439b-bab7-42c2bf20c4a4" path="/var/lib/kubelet/pods/02c94438-cf20-439b-bab7-42c2bf20c4a4/volumes" Nov 03 03:16:23.910838 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:16:23.910803 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:16:25.926028 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:25.925993 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-594d9845fc-skkgb" event={"ID":"770a6de2-28ed-4d95-b766-4a49792bdf4b","Type":"ContainerStarted","Data":"952b147ae587da7149c5f249151e0991ac747baca31b4d56846e15bd700ab4b0"} Nov 03 03:16:25.926443 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:25.926076 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-594d9845fc-skkgb" Nov 03 03:16:26.034232 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:26.034187 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-594d9845fc-skkgb" podStartSLOduration=1.9708493200000001 podStartE2EDuration="8.034174792s" totalImagesPullingTime="6.063325472s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:16:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:16:26.033474145 +0000 UTC m=+5063.292672293" watchObservedRunningTime="2026-11-03 03:16:26.034174792 +0000 UTC m=+5063.293372959" Nov 03 03:16:31.043929 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:31.043894 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb024034-n74dx"] Nov 03 03:16:31.090422 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:31.090386 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb024034-n74dx"] Nov 03 03:16:32.847900 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:32.847865 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="772d8930-c408-44dc-93ce-6f552334959c" path="/var/lib/kubelet/pods/772d8930-c408-44dc-93ce-6f552334959c/volumes" Nov 03 03:16:36.933496 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:36.933464 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-594d9845fc-skkgb" Nov 03 03:16:37.090077 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:37.090048 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-110ef79d-t8x2x"] Nov 03 03:16:37.091123 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:37.091094 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-110ef79d-t8x2x"] Nov 03 03:16:38.847995 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:38.847964 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f0dd36b-6507-4c53-92f9-b0f8d6f0abee" path="/var/lib/kubelet/pods/8f0dd36b-6507-4c53-92f9-b0f8d6f0abee/volumes" Nov 03 03:16:43.852698 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:43.852665 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kz87q"] Nov 03 03:16:43.853073 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:43.853011 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="e785f8b2-3071-44c5-a171-e3cd9cf831b5" containerName="istio-proxy" Nov 03 03:16:43.857253 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:43.857235 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kz87q" Nov 03 03:16:43.917375 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:43.917329 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff914592-f476-4e9d-b25c-3d34963ab378-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-kz87q\" (UID: \"ff914592-f476-4e9d-b25c-3d34963ab378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kz87q" Nov 03 03:16:43.917575 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:43.917551 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff914592-f476-4e9d-b25c-3d34963ab378-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-kz87q\" (UID: \"ff914592-f476-4e9d-b25c-3d34963ab378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kz87q" Nov 03 03:16:43.917727 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:43.917701 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgm4r\" (UniqueName: \"kubernetes.io/projected/ff914592-f476-4e9d-b25c-3d34963ab378-kube-api-access-bgm4r\") pod \"maas-api-revoke-sub-85237e45-kz87q\" (UID: \"ff914592-f476-4e9d-b25c-3d34963ab378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kz87q" Nov 03 03:16:43.954808 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:43.954784 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:16:43.954903 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:43.954845 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:16:43.954903 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:43.954845 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sb4qh\"" Nov 03 03:16:43.954903 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:43.954881 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:16:43.956192 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:43.956177 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kz87q"] Nov 03 03:16:44.018281 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:44.018256 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff914592-f476-4e9d-b25c-3d34963ab378-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-kz87q\" (UID: \"ff914592-f476-4e9d-b25c-3d34963ab378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kz87q" Nov 03 03:16:44.018410 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:44.018283 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff914592-f476-4e9d-b25c-3d34963ab378-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-kz87q\" (UID: \"ff914592-f476-4e9d-b25c-3d34963ab378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kz87q" Nov 03 03:16:44.018410 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:44.018301 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgm4r\" (UniqueName: \"kubernetes.io/projected/ff914592-f476-4e9d-b25c-3d34963ab378-kube-api-access-bgm4r\") pod \"maas-api-revoke-sub-85237e45-kz87q\" (UID: \"ff914592-f476-4e9d-b25c-3d34963ab378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kz87q" Nov 03 03:16:44.018965 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:44.018946 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff914592-f476-4e9d-b25c-3d34963ab378-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-kz87q\" (UID: \"ff914592-f476-4e9d-b25c-3d34963ab378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kz87q" Nov 03 03:16:44.020514 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:44.020495 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff914592-f476-4e9d-b25c-3d34963ab378-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-kz87q\" (UID: \"ff914592-f476-4e9d-b25c-3d34963ab378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kz87q" Nov 03 03:16:44.071550 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:44.071525 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgm4r\" (UniqueName: \"kubernetes.io/projected/ff914592-f476-4e9d-b25c-3d34963ab378-kube-api-access-bgm4r\") pod \"maas-api-revoke-sub-85237e45-kz87q\" (UID: \"ff914592-f476-4e9d-b25c-3d34963ab378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kz87q" Nov 03 03:16:44.169021 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:44.168951 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kz87q" Nov 03 03:16:44.533906 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:44.533832 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kz87q"] Nov 03 03:16:44.540304 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:16:44.540276 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff914592_f476_4e9d_b25c_3d34963ab378.slice/crio-759688d01789e47d7c9b2cb9bec19305991756b523f6c803052f83ecb8b8e36f WatchSource:0}: Error finding container 759688d01789e47d7c9b2cb9bec19305991756b523f6c803052f83ecb8b8e36f: Status 404 returned error can't find the container with id 759688d01789e47d7c9b2cb9bec19305991756b523f6c803052f83ecb8b8e36f Nov 03 03:16:45.020014 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:45.019982 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kz87q" event={"ID":"ff914592-f476-4e9d-b25c-3d34963ab378","Type":"ContainerStarted","Data":"c2fb4c0d1381dcbfa7bf494f92d3675a589174d39de2064c75819022a0f83275"} Nov 03 03:16:45.020509 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:45.020035 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kz87q" event={"ID":"ff914592-f476-4e9d-b25c-3d34963ab378","Type":"ContainerStarted","Data":"759688d01789e47d7c9b2cb9bec19305991756b523f6c803052f83ecb8b8e36f"} Nov 03 03:16:45.131321 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:45.131274 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kz87q" podStartSLOduration=2.13126064 podStartE2EDuration="2.13126064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:16:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:16:45.130777423 +0000 UTC m=+5082.389975575" watchObservedRunningTime="2026-11-03 03:16:45.13126064 +0000 UTC m=+5082.390458792" Nov 03 03:16:46.026258 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:46.026223 5605 generic.go:356] "Generic (PLEG): container finished" podID="ff914592-f476-4e9d-b25c-3d34963ab378" containerID="c2fb4c0d1381dcbfa7bf494f92d3675a589174d39de2064c75819022a0f83275" exitCode=0 Nov 03 03:16:46.026659 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:46.026275 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kz87q" event={"ID":"ff914592-f476-4e9d-b25c-3d34963ab378","Type":"ContainerDied","Data":"c2fb4c0d1381dcbfa7bf494f92d3675a589174d39de2064c75819022a0f83275"} Nov 03 03:16:47.147714 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.147690 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kz87q" Nov 03 03:16:47.241568 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.241542 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff914592-f476-4e9d-b25c-3d34963ab378-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff914592-f476-4e9d-b25c-3d34963ab378-maas-api-openshift-ca\") pod \"ff914592-f476-4e9d-b25c-3d34963ab378\" (UID: \"ff914592-f476-4e9d-b25c-3d34963ab378\") " Nov 03 03:16:47.241726 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.241587 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff914592-f476-4e9d-b25c-3d34963ab378-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff914592-f476-4e9d-b25c-3d34963ab378-maas-api-xks-ca\") pod \"ff914592-f476-4e9d-b25c-3d34963ab378\" (UID: \"ff914592-f476-4e9d-b25c-3d34963ab378\") " Nov 03 03:16:47.241726 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.241638 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff914592-f476-4e9d-b25c-3d34963ab378-kube-api-access-bgm4r\" (UniqueName: \"kubernetes.io/projected/ff914592-f476-4e9d-b25c-3d34963ab378-kube-api-access-bgm4r\") pod \"ff914592-f476-4e9d-b25c-3d34963ab378\" (UID: \"ff914592-f476-4e9d-b25c-3d34963ab378\") " Nov 03 03:16:47.241943 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.241921 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff914592-f476-4e9d-b25c-3d34963ab378-maas-api-openshift-ca" pod "ff914592-f476-4e9d-b25c-3d34963ab378" (UID: "ff914592-f476-4e9d-b25c-3d34963ab378"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:16:47.243679 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.243653 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff914592-f476-4e9d-b25c-3d34963ab378-maas-api-xks-ca" pod "ff914592-f476-4e9d-b25c-3d34963ab378" (UID: "ff914592-f476-4e9d-b25c-3d34963ab378"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:16:47.243873 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.243672 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff914592-f476-4e9d-b25c-3d34963ab378-kube-api-access-bgm4r" pod "ff914592-f476-4e9d-b25c-3d34963ab378" (UID: "ff914592-f476-4e9d-b25c-3d34963ab378"). InnerVolumeSpecName "kube-api-access-bgm4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:16:47.253166 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.253143 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dbd7q"] Nov 03 03:16:47.253531 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.253513 5605 cpu_manager.go:414] "containerMap: removing container" podUID="ff914592-f476-4e9d-b25c-3d34963ab378" containerName="revoke-keys" Nov 03 03:16:47.253615 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.253532 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff914592-f476-4e9d-b25c-3d34963ab378" containerName="revoke-keys" Nov 03 03:16:47.253692 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.253674 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff914592-f476-4e9d-b25c-3d34963ab378" containerName="revoke-keys" Nov 03 03:16:47.256670 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.256652 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dbd7q" Nov 03 03:16:47.300824 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.300803 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p8fqq"] Nov 03 03:16:47.303937 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.303923 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p8fqq" Nov 03 03:16:47.305509 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.305488 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 03:16:47.305509 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.305509 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-fwpk8\"" Nov 03 03:16:47.342528 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.342505 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc490604-2e00-49dd-8423-47fc70709678-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-dbd7q\" (UID: \"dc490604-2e00-49dd-8423-47fc70709678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dbd7q" Nov 03 03:16:47.342616 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.342537 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc490604-2e00-49dd-8423-47fc70709678-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-dbd7q\" (UID: \"dc490604-2e00-49dd-8423-47fc70709678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dbd7q" Nov 03 03:16:47.342616 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.342555 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc490604-2e00-49dd-8423-47fc70709678-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-dbd7q\" (UID: \"dc490604-2e00-49dd-8423-47fc70709678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dbd7q" Nov 03 03:16:47.342616 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.342596 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82phf\" (UniqueName: \"kubernetes.io/projected/dc490604-2e00-49dd-8423-47fc70709678-kube-api-access-82phf\") pod \"maas-default-gateway-openshift-default-5bc884dd87-dbd7q\" (UID: \"dc490604-2e00-49dd-8423-47fc70709678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dbd7q" Nov 03 03:16:47.342727 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.342693 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc490604-2e00-49dd-8423-47fc70709678-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-dbd7q\" (UID: \"dc490604-2e00-49dd-8423-47fc70709678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dbd7q" Nov 03 03:16:47.342727 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.342723 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc490604-2e00-49dd-8423-47fc70709678-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-dbd7q\" (UID: \"dc490604-2e00-49dd-8423-47fc70709678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dbd7q" Nov 03 03:16:47.342796 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.342739 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc490604-2e00-49dd-8423-47fc70709678-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-dbd7q\" (UID: \"dc490604-2e00-49dd-8423-47fc70709678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dbd7q" Nov 03 03:16:47.342796 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.342758 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc490604-2e00-49dd-8423-47fc70709678-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-dbd7q\" (UID: \"dc490604-2e00-49dd-8423-47fc70709678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dbd7q" Nov 03 03:16:47.342860 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.342820 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc490604-2e00-49dd-8423-47fc70709678-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-dbd7q\" (UID: \"dc490604-2e00-49dd-8423-47fc70709678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dbd7q" Nov 03 03:16:47.342907 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.342894 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff914592-f476-4e9d-b25c-3d34963ab378-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:16:47.342948 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.342909 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff914592-f476-4e9d-b25c-3d34963ab378-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:16:47.342948 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.342919 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgm4r\" (UniqueName: \"kubernetes.io/projected/ff914592-f476-4e9d-b25c-3d34963ab378-kube-api-access-bgm4r\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:16:47.352535 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.352505 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dbd7q"] Nov 03 03:16:47.400687 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.400666 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p8fqq"] Nov 03 03:16:47.444204 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.444176 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bedb6e8-5829-4cc2-a912-35131c941e09-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-p8fqq\" (UID: \"0bedb6e8-5829-4cc2-a912-35131c941e09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p8fqq" Nov 03 03:16:47.444332 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.444212 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc490604-2e00-49dd-8423-47fc70709678-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-dbd7q\" (UID: \"dc490604-2e00-49dd-8423-47fc70709678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dbd7q" Nov 03 03:16:47.444332 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.444230 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc490604-2e00-49dd-8423-47fc70709678-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-dbd7q\" (UID: \"dc490604-2e00-49dd-8423-47fc70709678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dbd7q" Nov 03 03:16:47.444332 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.444247 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc490604-2e00-49dd-8423-47fc70709678-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-dbd7q\" (UID: \"dc490604-2e00-49dd-8423-47fc70709678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dbd7q" Nov 03 03:16:47.444332 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.444310 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc490604-2e00-49dd-8423-47fc70709678-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-dbd7q\" (UID: \"dc490604-2e00-49dd-8423-47fc70709678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dbd7q" Nov 03 03:16:47.444597 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.444344 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bedb6e8-5829-4cc2-a912-35131c941e09-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-p8fqq\" (UID: \"0bedb6e8-5829-4cc2-a912-35131c941e09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p8fqq" Nov 03 03:16:47.444597 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.444423 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bedb6e8-5829-4cc2-a912-35131c941e09-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-p8fqq\" (UID: \"0bedb6e8-5829-4cc2-a912-35131c941e09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p8fqq" Nov 03 03:16:47.444597 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.444457 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bedb6e8-5829-4cc2-a912-35131c941e09-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-p8fqq\" (UID: \"0bedb6e8-5829-4cc2-a912-35131c941e09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p8fqq" Nov 03 03:16:47.444769 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.444691 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bedb6e8-5829-4cc2-a912-35131c941e09-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-p8fqq\" (UID: \"0bedb6e8-5829-4cc2-a912-35131c941e09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p8fqq" Nov 03 03:16:47.444769 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.444730 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bedb6e8-5829-4cc2-a912-35131c941e09-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-p8fqq\" (UID: \"0bedb6e8-5829-4cc2-a912-35131c941e09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p8fqq" Nov 03 03:16:47.444896 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.444766 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc490604-2e00-49dd-8423-47fc70709678-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-dbd7q\" (UID: \"dc490604-2e00-49dd-8423-47fc70709678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dbd7q" Nov 03 03:16:47.444896 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.444831 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc490604-2e00-49dd-8423-47fc70709678-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-dbd7q\" (UID: \"dc490604-2e00-49dd-8423-47fc70709678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dbd7q" Nov 03 03:16:47.444896 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.444839 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc490604-2e00-49dd-8423-47fc70709678-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-dbd7q\" (UID: \"dc490604-2e00-49dd-8423-47fc70709678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dbd7q" Nov 03 03:16:47.445077 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.444893 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc490604-2e00-49dd-8423-47fc70709678-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-dbd7q\" (UID: \"dc490604-2e00-49dd-8423-47fc70709678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dbd7q" Nov 03 03:16:47.445077 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.444902 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc490604-2e00-49dd-8423-47fc70709678-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-dbd7q\" (UID: \"dc490604-2e00-49dd-8423-47fc70709678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dbd7q" Nov 03 03:16:47.445077 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.445006 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc490604-2e00-49dd-8423-47fc70709678-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-dbd7q\" (UID: \"dc490604-2e00-49dd-8423-47fc70709678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dbd7q" Nov 03 03:16:47.445077 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.445057 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bedb6e8-5829-4cc2-a912-35131c941e09-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-p8fqq\" (UID: \"0bedb6e8-5829-4cc2-a912-35131c941e09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p8fqq" Nov 03 03:16:47.445334 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.445093 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bedb6e8-5829-4cc2-a912-35131c941e09-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-p8fqq\" (UID: \"0bedb6e8-5829-4cc2-a912-35131c941e09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p8fqq" Nov 03 03:16:47.445334 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.445143 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc490604-2e00-49dd-8423-47fc70709678-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-dbd7q\" (UID: \"dc490604-2e00-49dd-8423-47fc70709678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dbd7q" Nov 03 03:16:47.445334 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.445153 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82phf\" (UniqueName: \"kubernetes.io/projected/dc490604-2e00-49dd-8423-47fc70709678-kube-api-access-82phf\") pod \"maas-default-gateway-openshift-default-5bc884dd87-dbd7q\" (UID: \"dc490604-2e00-49dd-8423-47fc70709678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dbd7q" Nov 03 03:16:47.445334 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.445207 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc490604-2e00-49dd-8423-47fc70709678-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-dbd7q\" (UID: \"dc490604-2e00-49dd-8423-47fc70709678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dbd7q" Nov 03 03:16:47.445334 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.445206 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc490604-2e00-49dd-8423-47fc70709678-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-dbd7q\" (UID: \"dc490604-2e00-49dd-8423-47fc70709678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dbd7q" Nov 03 03:16:47.445334 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.445237 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p89c5\" (UniqueName: \"kubernetes.io/projected/0bedb6e8-5829-4cc2-a912-35131c941e09-kube-api-access-p89c5\") pod \"maas-default-gateway-openshift-default-5bc884dd87-p8fqq\" (UID: \"0bedb6e8-5829-4cc2-a912-35131c941e09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p8fqq" Nov 03 03:16:47.447052 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.447028 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc490604-2e00-49dd-8423-47fc70709678-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-dbd7q\" (UID: \"dc490604-2e00-49dd-8423-47fc70709678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dbd7q" Nov 03 03:16:47.447460 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.447444 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc490604-2e00-49dd-8423-47fc70709678-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-dbd7q\" (UID: \"dc490604-2e00-49dd-8423-47fc70709678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dbd7q" Nov 03 03:16:47.498177 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.498149 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82phf\" (UniqueName: \"kubernetes.io/projected/dc490604-2e00-49dd-8423-47fc70709678-kube-api-access-82phf\") pod \"maas-default-gateway-openshift-default-5bc884dd87-dbd7q\" (UID: \"dc490604-2e00-49dd-8423-47fc70709678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dbd7q" Nov 03 03:16:47.498514 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.498497 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc490604-2e00-49dd-8423-47fc70709678-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-dbd7q\" (UID: \"dc490604-2e00-49dd-8423-47fc70709678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dbd7q" Nov 03 03:16:47.546233 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.546210 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bedb6e8-5829-4cc2-a912-35131c941e09-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-p8fqq\" (UID: \"0bedb6e8-5829-4cc2-a912-35131c941e09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p8fqq" Nov 03 03:16:47.546339 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.546235 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bedb6e8-5829-4cc2-a912-35131c941e09-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-p8fqq\" (UID: \"0bedb6e8-5829-4cc2-a912-35131c941e09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p8fqq" Nov 03 03:16:47.546339 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.546262 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p89c5\" (UniqueName: \"kubernetes.io/projected/0bedb6e8-5829-4cc2-a912-35131c941e09-kube-api-access-p89c5\") pod \"maas-default-gateway-openshift-default-5bc884dd87-p8fqq\" (UID: \"0bedb6e8-5829-4cc2-a912-35131c941e09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p8fqq" Nov 03 03:16:47.546339 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.546288 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bedb6e8-5829-4cc2-a912-35131c941e09-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-p8fqq\" (UID: \"0bedb6e8-5829-4cc2-a912-35131c941e09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p8fqq" Nov 03 03:16:47.546339 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.546330 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bedb6e8-5829-4cc2-a912-35131c941e09-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-p8fqq\" (UID: \"0bedb6e8-5829-4cc2-a912-35131c941e09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p8fqq" Nov 03 03:16:47.546616 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.546381 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bedb6e8-5829-4cc2-a912-35131c941e09-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-p8fqq\" (UID: \"0bedb6e8-5829-4cc2-a912-35131c941e09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p8fqq" Nov 03 03:16:47.546616 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.546415 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bedb6e8-5829-4cc2-a912-35131c941e09-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-p8fqq\" (UID: \"0bedb6e8-5829-4cc2-a912-35131c941e09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p8fqq" Nov 03 03:16:47.546616 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.546451 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bedb6e8-5829-4cc2-a912-35131c941e09-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-p8fqq\" (UID: \"0bedb6e8-5829-4cc2-a912-35131c941e09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p8fqq" Nov 03 03:16:47.546616 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.546476 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bedb6e8-5829-4cc2-a912-35131c941e09-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-p8fqq\" (UID: \"0bedb6e8-5829-4cc2-a912-35131c941e09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p8fqq" Nov 03 03:16:47.546841 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.546737 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bedb6e8-5829-4cc2-a912-35131c941e09-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-p8fqq\" (UID: \"0bedb6e8-5829-4cc2-a912-35131c941e09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p8fqq" Nov 03 03:16:47.546841 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.546811 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bedb6e8-5829-4cc2-a912-35131c941e09-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-p8fqq\" (UID: \"0bedb6e8-5829-4cc2-a912-35131c941e09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p8fqq" Nov 03 03:16:47.546962 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.546862 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bedb6e8-5829-4cc2-a912-35131c941e09-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-p8fqq\" (UID: \"0bedb6e8-5829-4cc2-a912-35131c941e09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p8fqq" Nov 03 03:16:47.547069 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.547050 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bedb6e8-5829-4cc2-a912-35131c941e09-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-p8fqq\" (UID: \"0bedb6e8-5829-4cc2-a912-35131c941e09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p8fqq" Nov 03 03:16:47.547121 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.547051 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bedb6e8-5829-4cc2-a912-35131c941e09-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-p8fqq\" (UID: \"0bedb6e8-5829-4cc2-a912-35131c941e09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p8fqq" Nov 03 03:16:47.548561 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.548545 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bedb6e8-5829-4cc2-a912-35131c941e09-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-p8fqq\" (UID: \"0bedb6e8-5829-4cc2-a912-35131c941e09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p8fqq" Nov 03 03:16:47.548700 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.548684 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bedb6e8-5829-4cc2-a912-35131c941e09-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-p8fqq\" (UID: \"0bedb6e8-5829-4cc2-a912-35131c941e09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p8fqq" Nov 03 03:16:47.568648 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.568603 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dbd7q" Nov 03 03:16:47.599995 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.599970 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bedb6e8-5829-4cc2-a912-35131c941e09-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-p8fqq\" (UID: \"0bedb6e8-5829-4cc2-a912-35131c941e09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p8fqq" Nov 03 03:16:47.600097 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.600053 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p89c5\" (UniqueName: \"kubernetes.io/projected/0bedb6e8-5829-4cc2-a912-35131c941e09-kube-api-access-p89c5\") pod \"maas-default-gateway-openshift-default-5bc884dd87-p8fqq\" (UID: \"0bedb6e8-5829-4cc2-a912-35131c941e09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p8fqq" Nov 03 03:16:47.615892 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.615871 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p8fqq" Nov 03 03:16:47.985753 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.985685 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dbd7q"] Nov 03 03:16:47.988975 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.988949 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p8fqq"] Nov 03 03:16:47.991974 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:16:47.991946 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc490604_2e00_49dd_8423_47fc70709678.slice/crio-d4f2c3b51a13aa8dc2d1365586e3db17d757ab653db8e6c95f7e80b062d51719 WatchSource:0}: Error finding container d4f2c3b51a13aa8dc2d1365586e3db17d757ab653db8e6c95f7e80b062d51719: Status 404 returned error can't find the container with id d4f2c3b51a13aa8dc2d1365586e3db17d757ab653db8e6c95f7e80b062d51719 Nov 03 03:16:47.994018 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.993990 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 03:16:47.994080 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.994048 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 03:16:47.994121 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:47.994076 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 03:16:48.036869 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:48.036850 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kz87q" Nov 03 03:16:48.036955 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:48.036890 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kz87q" event={"ID":"ff914592-f476-4e9d-b25c-3d34963ab378","Type":"ContainerDied","Data":"759688d01789e47d7c9b2cb9bec19305991756b523f6c803052f83ecb8b8e36f"} Nov 03 03:16:48.036955 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:48.036938 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="759688d01789e47d7c9b2cb9bec19305991756b523f6c803052f83ecb8b8e36f" Nov 03 03:16:48.039096 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:48.039069 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dbd7q" event={"ID":"dc490604-2e00-49dd-8423-47fc70709678","Type":"ContainerStarted","Data":"d4f2c3b51a13aa8dc2d1365586e3db17d757ab653db8e6c95f7e80b062d51719"} Nov 03 03:16:48.042648 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:16:48.042625 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bedb6e8_5829_4cc2_a912_35131c941e09.slice/crio-45925e6363a0c9fb33e1995792af9a347112305ed7f4306a26483ea0281900df WatchSource:0}: Error finding container 45925e6363a0c9fb33e1995792af9a347112305ed7f4306a26483ea0281900df: Status 404 returned error can't find the container with id 45925e6363a0c9fb33e1995792af9a347112305ed7f4306a26483ea0281900df Nov 03 03:16:48.045203 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:48.045170 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 03:16:48.045321 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:48.045251 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 03:16:48.045321 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:48.045294 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 03:16:48.046923 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:48.046897 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a497f943-cwt9b"] Nov 03 03:16:48.049534 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:48.049487 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a497f943-cwt9b"] Nov 03 03:16:48.847959 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:48.847921 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1019142d-81ac-42e4-bd92-55e29ae2051a" path="/var/lib/kubelet/pods/1019142d-81ac-42e4-bd92-55e29ae2051a/volumes" Nov 03 03:16:49.045314 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:49.045279 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dbd7q" event={"ID":"dc490604-2e00-49dd-8423-47fc70709678","Type":"ContainerStarted","Data":"90f5a3e069c907b0ba42e145678543ec6490e592ee535885e25767d205a200c8"} Nov 03 03:16:49.046867 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:49.046842 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p8fqq" event={"ID":"0bedb6e8-5829-4cc2-a912-35131c941e09","Type":"ContainerStarted","Data":"7573948a4f09ce30c06a68201051d1d40c5b7f1ccbb54fd90f3a6af2f551df19"} Nov 03 03:16:49.046937 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:49.046895 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p8fqq" event={"ID":"0bedb6e8-5829-4cc2-a912-35131c941e09","Type":"ContainerStarted","Data":"45925e6363a0c9fb33e1995792af9a347112305ed7f4306a26483ea0281900df"} Nov 03 03:16:49.277484 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:49.275184 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p8fqq" podStartSLOduration=2.275164532 podStartE2EDuration="2.275164532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:16:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:16:49.274831108 +0000 UTC m=+5086.534029263" watchObservedRunningTime="2026-11-03 03:16:49.275164532 +0000 UTC m=+5086.534362676" Nov 03 03:16:49.277484 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:49.276665 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dbd7q" podStartSLOduration=2.276655363 podStartE2EDuration="2.276655363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:16:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:16:49.161291779 +0000 UTC m=+5086.420489934" watchObservedRunningTime="2026-11-03 03:16:49.276655363 +0000 UTC m=+5086.535853518" Nov 03 03:16:49.578380 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:49.577708 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dbd7q" Nov 03 03:16:49.581217 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:49.580897 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-dbd7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.35:15021/healthz/ready\": dial tcp 10.134.1.35:15021: connect: connection refused" start-of-body= Nov 03 03:16:49.581217 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:49.580956 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dbd7q" podUID="dc490604-2e00-49dd-8423-47fc70709678" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.35:15021/healthz/ready\": dial tcp 10.134.1.35:15021: connect: connection refused" Nov 03 03:16:49.617136 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:49.617110 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p8fqq" Nov 03 03:16:49.617927 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:49.617905 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-p8fqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.36:15021/healthz/ready\": dial tcp 10.134.1.36:15021: connect: connection refused" start-of-body= Nov 03 03:16:49.617985 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:49.617947 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p8fqq" podUID="0bedb6e8-5829-4cc2-a912-35131c941e09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.36:15021/healthz/ready\": dial tcp 10.134.1.36:15021: connect: connection refused" Nov 03 03:16:50.574381 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:50.573485 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-dbd7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.35:15021/healthz/ready\": dial tcp 10.134.1.35:15021: connect: connection refused" start-of-body= Nov 03 03:16:50.574381 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:50.573545 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dbd7q" podUID="dc490604-2e00-49dd-8423-47fc70709678" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.35:15021/healthz/ready\": dial tcp 10.134.1.35:15021: connect: connection refused" Nov 03 03:16:50.616593 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:50.616562 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-p8fqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.36:15021/healthz/ready\": dial tcp 10.134.1.36:15021: connect: connection refused" start-of-body= Nov 03 03:16:50.616754 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:50.616611 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p8fqq" podUID="0bedb6e8-5829-4cc2-a912-35131c941e09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.36:15021/healthz/ready\": dial tcp 10.134.1.36:15021: connect: connection refused" Nov 03 03:16:51.574224 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:51.573796 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-dbd7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.35:15021/healthz/ready\": dial tcp 10.134.1.35:15021: connect: connection refused" start-of-body= Nov 03 03:16:51.574224 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:51.573854 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dbd7q" podUID="dc490604-2e00-49dd-8423-47fc70709678" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.35:15021/healthz/ready\": dial tcp 10.134.1.35:15021: connect: connection refused" Nov 03 03:16:51.617032 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:51.616994 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-p8fqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.36:15021/healthz/ready\": dial tcp 10.134.1.36:15021: connect: connection refused" start-of-body= Nov 03 03:16:51.617156 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:51.617048 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p8fqq" podUID="0bedb6e8-5829-4cc2-a912-35131c941e09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.36:15021/healthz/ready\": dial tcp 10.134.1.36:15021: connect: connection refused" Nov 03 03:16:52.575381 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:52.574740 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-dbd7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.35:15021/healthz/ready\": dial tcp 10.134.1.35:15021: connect: connection refused" start-of-body= Nov 03 03:16:52.575381 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:52.574802 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dbd7q" podUID="dc490604-2e00-49dd-8423-47fc70709678" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.35:15021/healthz/ready\": dial tcp 10.134.1.35:15021: connect: connection refused" Nov 03 03:16:52.616683 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:52.616652 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-p8fqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.36:15021/healthz/ready\": dial tcp 10.134.1.36:15021: connect: connection refused" start-of-body= Nov 03 03:16:52.616913 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:52.616891 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p8fqq" podUID="0bedb6e8-5829-4cc2-a912-35131c941e09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.36:15021/healthz/ready\": dial tcp 10.134.1.36:15021: connect: connection refused" Nov 03 03:16:53.573383 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:53.571862 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-dbd7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.35:15021/healthz/ready\": dial tcp 10.134.1.35:15021: connect: connection refused" start-of-body= Nov 03 03:16:53.573383 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:53.571951 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dbd7q" podUID="dc490604-2e00-49dd-8423-47fc70709678" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.35:15021/healthz/ready\": dial tcp 10.134.1.35:15021: connect: connection refused" Nov 03 03:16:53.616643 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:53.616610 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-p8fqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.36:15021/healthz/ready\": dial tcp 10.134.1.36:15021: connect: connection refused" start-of-body= Nov 03 03:16:53.617035 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:53.616665 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p8fqq" podUID="0bedb6e8-5829-4cc2-a912-35131c941e09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.36:15021/healthz/ready\": dial tcp 10.134.1.36:15021: connect: connection refused" Nov 03 03:16:53.912832 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:16:53.912743 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:16:54.569702 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:54.569664 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-dbd7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.35:15021/healthz/ready\": dial tcp 10.134.1.35:15021: connect: connection refused" start-of-body= Nov 03 03:16:54.569897 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:54.569719 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dbd7q" podUID="dc490604-2e00-49dd-8423-47fc70709678" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.35:15021/healthz/ready\": dial tcp 10.134.1.35:15021: connect: connection refused" Nov 03 03:16:54.616763 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:54.616726 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-p8fqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.36:15021/healthz/ready\": dial tcp 10.134.1.36:15021: connect: connection refused" start-of-body= Nov 03 03:16:54.617104 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:54.616783 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p8fqq" podUID="0bedb6e8-5829-4cc2-a912-35131c941e09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.36:15021/healthz/ready\": dial tcp 10.134.1.36:15021: connect: connection refused" Nov 03 03:16:56.255332 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:56.255300 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p8fqq" Nov 03 03:16:56.256019 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:56.256001 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p8fqq" Nov 03 03:16:56.256844 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:56.256820 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p8fqq" Nov 03 03:16:56.271138 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:56.271115 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dbd7q" Nov 03 03:16:56.271420 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:56.271398 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dbd7q" Nov 03 03:16:56.272047 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:56.272030 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dbd7q" Nov 03 03:16:56.546963 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:56.546896 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-594d9845fc-skkgb"] Nov 03 03:16:56.547095 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:56.547079 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-594d9845fc-skkgb" podUID="770a6de2-28ed-4d95-b766-4a49792bdf4b" containerName="manager" containerID="cri-o://952b147ae587da7149c5f249151e0991ac747baca31b4d56846e15bd700ab4b0" gracePeriod=30 Nov 03 03:16:56.806281 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:56.806257 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-594d9845fc-skkgb" Nov 03 03:16:56.874001 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:56.873976 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/770a6de2-28ed-4d95-b766-4a49792bdf4b-webhook-cert\" (UniqueName: \"kubernetes.io/secret/770a6de2-28ed-4d95-b766-4a49792bdf4b-webhook-cert\") pod \"770a6de2-28ed-4d95-b766-4a49792bdf4b\" (UID: \"770a6de2-28ed-4d95-b766-4a49792bdf4b\") " Nov 03 03:16:56.874127 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:56.874009 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/770a6de2-28ed-4d95-b766-4a49792bdf4b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/770a6de2-28ed-4d95-b766-4a49792bdf4b-metrics-certs\") pod \"770a6de2-28ed-4d95-b766-4a49792bdf4b\" (UID: \"770a6de2-28ed-4d95-b766-4a49792bdf4b\") " Nov 03 03:16:56.874127 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:56.874030 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/770a6de2-28ed-4d95-b766-4a49792bdf4b-kube-api-access-5mmn4\" (UniqueName: \"kubernetes.io/projected/770a6de2-28ed-4d95-b766-4a49792bdf4b-kube-api-access-5mmn4\") pod \"770a6de2-28ed-4d95-b766-4a49792bdf4b\" (UID: \"770a6de2-28ed-4d95-b766-4a49792bdf4b\") " Nov 03 03:16:56.876060 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:56.876032 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/770a6de2-28ed-4d95-b766-4a49792bdf4b-webhook-cert" pod "770a6de2-28ed-4d95-b766-4a49792bdf4b" (UID: "770a6de2-28ed-4d95-b766-4a49792bdf4b"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:16:56.876060 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:56.876042 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/770a6de2-28ed-4d95-b766-4a49792bdf4b-metrics-certs" pod "770a6de2-28ed-4d95-b766-4a49792bdf4b" (UID: "770a6de2-28ed-4d95-b766-4a49792bdf4b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:16:56.876203 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:56.876099 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/770a6de2-28ed-4d95-b766-4a49792bdf4b-kube-api-access-5mmn4" pod "770a6de2-28ed-4d95-b766-4a49792bdf4b" (UID: "770a6de2-28ed-4d95-b766-4a49792bdf4b"). InnerVolumeSpecName "kube-api-access-5mmn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:16:56.975847 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:56.975811 5605 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/770a6de2-28ed-4d95-b766-4a49792bdf4b-webhook-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:16:56.975847 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:56.975841 5605 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/770a6de2-28ed-4d95-b766-4a49792bdf4b-metrics-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:16:56.975847 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:56.975852 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mmn4\" (UniqueName: \"kubernetes.io/projected/770a6de2-28ed-4d95-b766-4a49792bdf4b-kube-api-access-5mmn4\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:16:57.142677 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:57.142601 5605 generic.go:356] "Generic (PLEG): container finished" podID="770a6de2-28ed-4d95-b766-4a49792bdf4b" containerID="952b147ae587da7149c5f249151e0991ac747baca31b4d56846e15bd700ab4b0" exitCode=0 Nov 03 03:16:57.142677 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:57.142652 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-594d9845fc-skkgb" Nov 03 03:16:57.142864 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:57.142684 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-594d9845fc-skkgb" event={"ID":"770a6de2-28ed-4d95-b766-4a49792bdf4b","Type":"ContainerDied","Data":"952b147ae587da7149c5f249151e0991ac747baca31b4d56846e15bd700ab4b0"} Nov 03 03:16:57.142864 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:57.142741 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-594d9845fc-skkgb" event={"ID":"770a6de2-28ed-4d95-b766-4a49792bdf4b","Type":"ContainerDied","Data":"4e36b830e39d500e8c183877cd30944e4501ed03902f92522a8ca652e63de043"} Nov 03 03:16:57.142864 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:57.142776 5605 scope.go:122] "RemoveContainer" containerID="952b147ae587da7149c5f249151e0991ac747baca31b4d56846e15bd700ab4b0" Nov 03 03:16:57.152081 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:57.152066 5605 scope.go:122] "RemoveContainer" containerID="952b147ae587da7149c5f249151e0991ac747baca31b4d56846e15bd700ab4b0" Nov 03 03:16:57.152328 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:16:57.152309 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"952b147ae587da7149c5f249151e0991ac747baca31b4d56846e15bd700ab4b0\": container with ID starting with 952b147ae587da7149c5f249151e0991ac747baca31b4d56846e15bd700ab4b0 not found: ID does not exist" containerID="952b147ae587da7149c5f249151e0991ac747baca31b4d56846e15bd700ab4b0" Nov 03 03:16:57.152431 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:57.152335 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"952b147ae587da7149c5f249151e0991ac747baca31b4d56846e15bd700ab4b0"} err="failed to get container status \"952b147ae587da7149c5f249151e0991ac747baca31b4d56846e15bd700ab4b0\": rpc error: code = NotFound desc = could not find container \"952b147ae587da7149c5f249151e0991ac747baca31b4d56846e15bd700ab4b0\": container with ID starting with 952b147ae587da7149c5f249151e0991ac747baca31b4d56846e15bd700ab4b0 not found: ID does not exist" Nov 03 03:16:57.296441 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:57.296401 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-594d9845fc-skkgb"] Nov 03 03:16:57.302464 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:57.302445 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-594d9845fc-skkgb"] Nov 03 03:16:58.725770 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:58.725737 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-594d9845fc-7m8kb"] Nov 03 03:16:58.726155 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:58.726031 5605 cpu_manager.go:414] "containerMap: removing container" podUID="770a6de2-28ed-4d95-b766-4a49792bdf4b" containerName="manager" Nov 03 03:16:58.726155 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:58.726041 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="770a6de2-28ed-4d95-b766-4a49792bdf4b" containerName="manager" Nov 03 03:16:58.726155 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:58.726106 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="770a6de2-28ed-4d95-b766-4a49792bdf4b" containerName="manager" Nov 03 03:16:58.730467 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:58.730445 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-594d9845fc-7m8kb" Nov 03 03:16:58.780154 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:58.780127 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 03:16:58.780278 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:58.780264 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-whxgk\"" Nov 03 03:16:58.789584 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:58.789565 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/949a8686-ae14-4c12-9da2-6bb05250692b-metrics-certs\") pod \"maas-controller-594d9845fc-7m8kb\" (UID: \"949a8686-ae14-4c12-9da2-6bb05250692b\") " pod="opendatahub/maas-controller-594d9845fc-7m8kb" Nov 03 03:16:58.789661 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:58.789601 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/949a8686-ae14-4c12-9da2-6bb05250692b-webhook-cert\") pod \"maas-controller-594d9845fc-7m8kb\" (UID: \"949a8686-ae14-4c12-9da2-6bb05250692b\") " pod="opendatahub/maas-controller-594d9845fc-7m8kb" Nov 03 03:16:58.789721 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:58.789708 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffhl2\" (UniqueName: \"kubernetes.io/projected/949a8686-ae14-4c12-9da2-6bb05250692b-kube-api-access-ffhl2\") pod \"maas-controller-594d9845fc-7m8kb\" (UID: \"949a8686-ae14-4c12-9da2-6bb05250692b\") " pod="opendatahub/maas-controller-594d9845fc-7m8kb" Nov 03 03:16:58.826616 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:58.826595 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 03:16:58.826616 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:58.826596 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 03:16:58.828631 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:58.828611 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-594d9845fc-7m8kb"] Nov 03 03:16:58.848029 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:58.848007 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="770a6de2-28ed-4d95-b766-4a49792bdf4b" path="/var/lib/kubelet/pods/770a6de2-28ed-4d95-b766-4a49792bdf4b/volumes" Nov 03 03:16:58.890790 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:58.890768 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffhl2\" (UniqueName: \"kubernetes.io/projected/949a8686-ae14-4c12-9da2-6bb05250692b-kube-api-access-ffhl2\") pod \"maas-controller-594d9845fc-7m8kb\" (UID: \"949a8686-ae14-4c12-9da2-6bb05250692b\") " pod="opendatahub/maas-controller-594d9845fc-7m8kb" Nov 03 03:16:58.890879 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:58.890807 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/949a8686-ae14-4c12-9da2-6bb05250692b-metrics-certs\") pod \"maas-controller-594d9845fc-7m8kb\" (UID: \"949a8686-ae14-4c12-9da2-6bb05250692b\") " pod="opendatahub/maas-controller-594d9845fc-7m8kb" Nov 03 03:16:58.890879 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:58.890840 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/949a8686-ae14-4c12-9da2-6bb05250692b-webhook-cert\") pod \"maas-controller-594d9845fc-7m8kb\" (UID: \"949a8686-ae14-4c12-9da2-6bb05250692b\") " pod="opendatahub/maas-controller-594d9845fc-7m8kb" Nov 03 03:16:58.893274 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:58.893239 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/949a8686-ae14-4c12-9da2-6bb05250692b-metrics-certs\") pod \"maas-controller-594d9845fc-7m8kb\" (UID: \"949a8686-ae14-4c12-9da2-6bb05250692b\") " pod="opendatahub/maas-controller-594d9845fc-7m8kb" Nov 03 03:16:58.893404 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:58.893324 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/949a8686-ae14-4c12-9da2-6bb05250692b-webhook-cert\") pod \"maas-controller-594d9845fc-7m8kb\" (UID: \"949a8686-ae14-4c12-9da2-6bb05250692b\") " pod="opendatahub/maas-controller-594d9845fc-7m8kb" Nov 03 03:16:58.944328 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:58.944305 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffhl2\" (UniqueName: \"kubernetes.io/projected/949a8686-ae14-4c12-9da2-6bb05250692b-kube-api-access-ffhl2\") pod \"maas-controller-594d9845fc-7m8kb\" (UID: \"949a8686-ae14-4c12-9da2-6bb05250692b\") " pod="opendatahub/maas-controller-594d9845fc-7m8kb" Nov 03 03:16:59.042455 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:59.042407 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-594d9845fc-7m8kb" Nov 03 03:16:59.406261 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:16:59.406231 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-594d9845fc-7m8kb"] Nov 03 03:16:59.442875 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:16:59.442853 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod949a8686_ae14_4c12_9da2_6bb05250692b.slice/crio-5875518ca66c4d403c8cd4201bed3ce9ebc9e4ffe0cbd0dc3b8a4e1dca400d28 WatchSource:0}: Error finding container 5875518ca66c4d403c8cd4201bed3ce9ebc9e4ffe0cbd0dc3b8a4e1dca400d28: Status 404 returned error can't find the container with id 5875518ca66c4d403c8cd4201bed3ce9ebc9e4ffe0cbd0dc3b8a4e1dca400d28 Nov 03 03:17:00.043327 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:00.043298 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f36bf64b-nqdn4"] Nov 03 03:17:00.089963 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:00.089935 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f36bf64b-nqdn4"] Nov 03 03:17:00.159597 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:00.159566 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-594d9845fc-7m8kb" event={"ID":"949a8686-ae14-4c12-9da2-6bb05250692b","Type":"ContainerStarted","Data":"5875518ca66c4d403c8cd4201bed3ce9ebc9e4ffe0cbd0dc3b8a4e1dca400d28"} Nov 03 03:17:00.847830 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:00.847790 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b0e9dab-d915-423b-b2ff-20dc33b3a57f" path="/var/lib/kubelet/pods/9b0e9dab-d915-423b-b2ff-20dc33b3a57f/volumes" Nov 03 03:17:01.171923 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:01.171834 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-594d9845fc-7m8kb" event={"ID":"949a8686-ae14-4c12-9da2-6bb05250692b","Type":"ContainerStarted","Data":"a14c72644b06fa5a498374002ddfda9fb30b575ea44f8ded923ba335221dcb45"} Nov 03 03:17:01.172275 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:01.171929 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-594d9845fc-7m8kb" Nov 03 03:17:01.279972 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:01.279916 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-594d9845fc-7m8kb" podStartSLOduration=2.465443856 podStartE2EDuration="3.279903131s" totalImagesPullingTime="814.459275ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:16:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:17:01.278706932 +0000 UTC m=+5098.537905085" watchObservedRunningTime="2026-11-03 03:17:01.279903131 +0000 UTC m=+5098.539101284" Nov 03 03:17:05.493932 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:05.493901 5605 scope.go:122] "RemoveContainer" containerID="a9d39a253eaa219161f514ff022172bd46b281e77f617b59ce6e2525400fb115" Nov 03 03:17:05.502375 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:05.502338 5605 scope.go:122] "RemoveContainer" containerID="0cdeff5a2245ae57533588de7c0872c979e68e282c413e77147eeb5bde3ac5a8" Nov 03 03:17:05.509778 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:05.509759 5605 scope.go:122] "RemoveContainer" containerID="02f57d300f11b69c33fcd1598158c2287407cf9771a4c4289cdfb836488fe2c1" Nov 03 03:17:05.516794 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:05.516771 5605 scope.go:122] "RemoveContainer" containerID="b8b24ea820d90f60906b5e3a647800dee2ff953d260440f90fc233847c0b04f4" Nov 03 03:17:11.042993 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:11.042959 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b81742eb-ggpd6"] Nov 03 03:17:11.089509 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:11.089483 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b81742eb-ggpd6"] Nov 03 03:17:12.179205 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:12.179177 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-594d9845fc-7m8kb" Nov 03 03:17:12.848329 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:12.848296 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75fc2ee9-2682-4feb-b35b-b792d3aad995" path="/var/lib/kubelet/pods/75fc2ee9-2682-4feb-b35b-b792d3aad995/volumes" Nov 03 03:17:23.911022 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:17:23.910984 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:17:26.017254 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:26.017217 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:17:32.162263 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:32.162231 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b7mjf"] Nov 03 03:17:32.166870 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:32.166848 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b7mjf" Nov 03 03:17:32.258372 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:32.258336 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw7t6\" (UniqueName: \"kubernetes.io/projected/cc6733d7-9260-4970-b6fd-9103df988649-kube-api-access-dw7t6\") pod \"maas-api-revoke-sub-f904bff9-b7mjf\" (UID: \"cc6733d7-9260-4970-b6fd-9103df988649\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b7mjf" Nov 03 03:17:32.258536 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:32.258403 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc6733d7-9260-4970-b6fd-9103df988649-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-b7mjf\" (UID: \"cc6733d7-9260-4970-b6fd-9103df988649\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b7mjf" Nov 03 03:17:32.258609 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:32.258528 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc6733d7-9260-4970-b6fd-9103df988649-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-b7mjf\" (UID: \"cc6733d7-9260-4970-b6fd-9103df988649\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b7mjf" Nov 03 03:17:32.263953 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:32.263933 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:17:32.264054 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:32.264025 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sb4qh\"" Nov 03 03:17:32.264054 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:32.264022 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:17:32.264196 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:32.264180 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:17:32.265560 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:32.265540 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b7mjf"] Nov 03 03:17:32.359340 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:32.359314 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc6733d7-9260-4970-b6fd-9103df988649-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-b7mjf\" (UID: \"cc6733d7-9260-4970-b6fd-9103df988649\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b7mjf" Nov 03 03:17:32.359467 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:32.359389 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw7t6\" (UniqueName: \"kubernetes.io/projected/cc6733d7-9260-4970-b6fd-9103df988649-kube-api-access-dw7t6\") pod \"maas-api-revoke-sub-f904bff9-b7mjf\" (UID: \"cc6733d7-9260-4970-b6fd-9103df988649\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b7mjf" Nov 03 03:17:32.359467 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:32.359434 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc6733d7-9260-4970-b6fd-9103df988649-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-b7mjf\" (UID: \"cc6733d7-9260-4970-b6fd-9103df988649\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b7mjf" Nov 03 03:17:32.359922 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:32.359900 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc6733d7-9260-4970-b6fd-9103df988649-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-b7mjf\" (UID: \"cc6733d7-9260-4970-b6fd-9103df988649\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b7mjf" Nov 03 03:17:32.361680 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:32.361661 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc6733d7-9260-4970-b6fd-9103df988649-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-b7mjf\" (UID: \"cc6733d7-9260-4970-b6fd-9103df988649\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b7mjf" Nov 03 03:17:32.412538 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:32.412488 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw7t6\" (UniqueName: \"kubernetes.io/projected/cc6733d7-9260-4970-b6fd-9103df988649-kube-api-access-dw7t6\") pod \"maas-api-revoke-sub-f904bff9-b7mjf\" (UID: \"cc6733d7-9260-4970-b6fd-9103df988649\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b7mjf" Nov 03 03:17:32.481434 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:32.481413 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b7mjf" Nov 03 03:17:32.842822 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:32.842789 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b7mjf"] Nov 03 03:17:32.849990 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:17:32.849958 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc6733d7_9260_4970_b6fd_9103df988649.slice/crio-adb6cb1771591472196b65762ed339da895856ecb9e8a96a47fba4d05c35e28e WatchSource:0}: Error finding container adb6cb1771591472196b65762ed339da895856ecb9e8a96a47fba4d05c35e28e: Status 404 returned error can't find the container with id adb6cb1771591472196b65762ed339da895856ecb9e8a96a47fba4d05c35e28e Nov 03 03:17:33.322511 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:33.322477 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b7mjf" event={"ID":"cc6733d7-9260-4970-b6fd-9103df988649","Type":"ContainerStarted","Data":"a57e2d0eb73e240b721a04f211acabf55e758a11b5457e42045c64d5ef9b3cb2"} Nov 03 03:17:33.322846 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:33.322540 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b7mjf" event={"ID":"cc6733d7-9260-4970-b6fd-9103df988649","Type":"ContainerStarted","Data":"adb6cb1771591472196b65762ed339da895856ecb9e8a96a47fba4d05c35e28e"} Nov 03 03:17:33.431086 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:33.431041 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b7mjf" podStartSLOduration=1.431029243 podStartE2EDuration="1.431029243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:17:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:17:33.428822733 +0000 UTC m=+5130.688020890" watchObservedRunningTime="2026-11-03 03:17:33.431029243 +0000 UTC m=+5130.690227395" Nov 03 03:17:34.328751 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:34.328710 5605 generic.go:356] "Generic (PLEG): container finished" podID="cc6733d7-9260-4970-b6fd-9103df988649" containerID="a57e2d0eb73e240b721a04f211acabf55e758a11b5457e42045c64d5ef9b3cb2" exitCode=0 Nov 03 03:17:34.329157 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:34.328795 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b7mjf" event={"ID":"cc6733d7-9260-4970-b6fd-9103df988649","Type":"ContainerDied","Data":"a57e2d0eb73e240b721a04f211acabf55e758a11b5457e42045c64d5ef9b3cb2"} Nov 03 03:17:35.459070 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:35.459042 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b7mjf" Nov 03 03:17:35.587723 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:35.587626 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc6733d7-9260-4970-b6fd-9103df988649-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc6733d7-9260-4970-b6fd-9103df988649-maas-api-openshift-ca\") pod \"cc6733d7-9260-4970-b6fd-9103df988649\" (UID: \"cc6733d7-9260-4970-b6fd-9103df988649\") " Nov 03 03:17:35.587723 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:35.587702 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc6733d7-9260-4970-b6fd-9103df988649-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc6733d7-9260-4970-b6fd-9103df988649-maas-api-xks-ca\") pod \"cc6733d7-9260-4970-b6fd-9103df988649\" (UID: \"cc6733d7-9260-4970-b6fd-9103df988649\") " Nov 03 03:17:35.587986 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:35.587787 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc6733d7-9260-4970-b6fd-9103df988649-kube-api-access-dw7t6\" (UniqueName: \"kubernetes.io/projected/cc6733d7-9260-4970-b6fd-9103df988649-kube-api-access-dw7t6\") pod \"cc6733d7-9260-4970-b6fd-9103df988649\" (UID: \"cc6733d7-9260-4970-b6fd-9103df988649\") " Nov 03 03:17:35.588079 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:35.588049 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc6733d7-9260-4970-b6fd-9103df988649-maas-api-openshift-ca" pod "cc6733d7-9260-4970-b6fd-9103df988649" (UID: "cc6733d7-9260-4970-b6fd-9103df988649"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:17:35.589928 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:35.589899 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc6733d7-9260-4970-b6fd-9103df988649-maas-api-xks-ca" pod "cc6733d7-9260-4970-b6fd-9103df988649" (UID: "cc6733d7-9260-4970-b6fd-9103df988649"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:17:35.590072 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:35.590023 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc6733d7-9260-4970-b6fd-9103df988649-kube-api-access-dw7t6" pod "cc6733d7-9260-4970-b6fd-9103df988649" (UID: "cc6733d7-9260-4970-b6fd-9103df988649"). InnerVolumeSpecName "kube-api-access-dw7t6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:17:35.689351 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:35.689320 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dw7t6\" (UniqueName: \"kubernetes.io/projected/cc6733d7-9260-4970-b6fd-9103df988649-kube-api-access-dw7t6\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:17:35.689351 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:35.689348 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc6733d7-9260-4970-b6fd-9103df988649-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:17:35.689351 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:35.689377 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc6733d7-9260-4970-b6fd-9103df988649-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:17:36.339953 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:36.339920 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b7mjf" Nov 03 03:17:36.339953 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:36.339942 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b7mjf" event={"ID":"cc6733d7-9260-4970-b6fd-9103df988649","Type":"ContainerDied","Data":"adb6cb1771591472196b65762ed339da895856ecb9e8a96a47fba4d05c35e28e"} Nov 03 03:17:36.340175 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:36.340015 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="adb6cb1771591472196b65762ed339da895856ecb9e8a96a47fba4d05c35e28e" Nov 03 03:17:40.042863 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:40.042828 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-648bc910-dp469"] Nov 03 03:17:40.089632 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:40.089585 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-648bc910-dp469"] Nov 03 03:17:40.849326 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:40.849221 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6899acd9-62fa-46f2-8682-fc9fce7e5d06" path="/var/lib/kubelet/pods/6899acd9-62fa-46f2-8682-fc9fce7e5d06/volumes" Nov 03 03:17:44.847930 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:44.847899 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:17:47.684596 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:47.684565 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bpbq7"] Nov 03 03:17:47.684968 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:47.684882 5605 cpu_manager.go:414] "containerMap: removing container" podUID="cc6733d7-9260-4970-b6fd-9103df988649" containerName="revoke-keys" Nov 03 03:17:47.684968 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:47.684893 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc6733d7-9260-4970-b6fd-9103df988649" containerName="revoke-keys" Nov 03 03:17:47.684968 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:47.684955 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc6733d7-9260-4970-b6fd-9103df988649" containerName="revoke-keys" Nov 03 03:17:47.688326 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:47.688309 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bpbq7" Nov 03 03:17:47.731271 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:47.731248 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k449x"] Nov 03 03:17:47.734569 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:47.734556 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k449x" Nov 03 03:17:47.787499 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:47.787471 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bpbq7"] Nov 03 03:17:47.799415 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:47.799393 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/563c5b47-58ea-4fb8-b9f1-332cf8dae641-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bpbq7\" (UID: \"563c5b47-58ea-4fb8-b9f1-332cf8dae641\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bpbq7" Nov 03 03:17:47.799522 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:47.799428 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/563c5b47-58ea-4fb8-b9f1-332cf8dae641-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bpbq7\" (UID: \"563c5b47-58ea-4fb8-b9f1-332cf8dae641\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bpbq7" Nov 03 03:17:47.799522 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:47.799490 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/563c5b47-58ea-4fb8-b9f1-332cf8dae641-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bpbq7\" (UID: \"563c5b47-58ea-4fb8-b9f1-332cf8dae641\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bpbq7" Nov 03 03:17:47.799618 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:47.799572 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4qcb\" (UniqueName: \"kubernetes.io/projected/563c5b47-58ea-4fb8-b9f1-332cf8dae641-kube-api-access-j4qcb\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bpbq7\" (UID: \"563c5b47-58ea-4fb8-b9f1-332cf8dae641\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bpbq7" Nov 03 03:17:47.799658 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:47.799619 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/563c5b47-58ea-4fb8-b9f1-332cf8dae641-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bpbq7\" (UID: \"563c5b47-58ea-4fb8-b9f1-332cf8dae641\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bpbq7" Nov 03 03:17:47.799658 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:47.799651 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/563c5b47-58ea-4fb8-b9f1-332cf8dae641-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bpbq7\" (UID: \"563c5b47-58ea-4fb8-b9f1-332cf8dae641\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bpbq7" Nov 03 03:17:47.799757 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:47.799727 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/563c5b47-58ea-4fb8-b9f1-332cf8dae641-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bpbq7\" (UID: \"563c5b47-58ea-4fb8-b9f1-332cf8dae641\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bpbq7" Nov 03 03:17:47.799813 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:47.799788 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/563c5b47-58ea-4fb8-b9f1-332cf8dae641-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bpbq7\" (UID: \"563c5b47-58ea-4fb8-b9f1-332cf8dae641\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bpbq7" Nov 03 03:17:47.799856 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:47.799832 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/563c5b47-58ea-4fb8-b9f1-332cf8dae641-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bpbq7\" (UID: \"563c5b47-58ea-4fb8-b9f1-332cf8dae641\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bpbq7" Nov 03 03:17:47.833671 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:47.833651 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k449x"] Nov 03 03:17:47.900701 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:47.900679 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/563c5b47-58ea-4fb8-b9f1-332cf8dae641-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bpbq7\" (UID: \"563c5b47-58ea-4fb8-b9f1-332cf8dae641\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bpbq7" Nov 03 03:17:47.900839 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:47.900707 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/563c5b47-58ea-4fb8-b9f1-332cf8dae641-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bpbq7\" (UID: \"563c5b47-58ea-4fb8-b9f1-332cf8dae641\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bpbq7" Nov 03 03:17:47.900839 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:47.900726 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/563c5b47-58ea-4fb8-b9f1-332cf8dae641-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bpbq7\" (UID: \"563c5b47-58ea-4fb8-b9f1-332cf8dae641\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bpbq7" Nov 03 03:17:47.900839 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:47.900746 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4qcb\" (UniqueName: \"kubernetes.io/projected/563c5b47-58ea-4fb8-b9f1-332cf8dae641-kube-api-access-j4qcb\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bpbq7\" (UID: \"563c5b47-58ea-4fb8-b9f1-332cf8dae641\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bpbq7" Nov 03 03:17:47.900839 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:47.900778 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/563c5b47-58ea-4fb8-b9f1-332cf8dae641-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bpbq7\" (UID: \"563c5b47-58ea-4fb8-b9f1-332cf8dae641\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bpbq7" Nov 03 03:17:47.900839 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:47.900804 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/563c5b47-58ea-4fb8-b9f1-332cf8dae641-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bpbq7\" (UID: \"563c5b47-58ea-4fb8-b9f1-332cf8dae641\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bpbq7" Nov 03 03:17:47.900839 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:47.900839 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/563c5b47-58ea-4fb8-b9f1-332cf8dae641-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bpbq7\" (UID: \"563c5b47-58ea-4fb8-b9f1-332cf8dae641\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bpbq7" Nov 03 03:17:47.901195 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:47.900870 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdnhh\" (UniqueName: \"kubernetes.io/projected/0378fcbc-4027-4275-b274-2c188f3ff442-kube-api-access-kdnhh\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k449x\" (UID: \"0378fcbc-4027-4275-b274-2c188f3ff442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k449x" Nov 03 03:17:47.901195 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:47.900907 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0378fcbc-4027-4275-b274-2c188f3ff442-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k449x\" (UID: \"0378fcbc-4027-4275-b274-2c188f3ff442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k449x" Nov 03 03:17:47.901195 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:47.900946 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0378fcbc-4027-4275-b274-2c188f3ff442-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k449x\" (UID: \"0378fcbc-4027-4275-b274-2c188f3ff442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k449x" Nov 03 03:17:47.901195 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:47.900968 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0378fcbc-4027-4275-b274-2c188f3ff442-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k449x\" (UID: \"0378fcbc-4027-4275-b274-2c188f3ff442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k449x" Nov 03 03:17:47.901195 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:47.900999 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0378fcbc-4027-4275-b274-2c188f3ff442-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k449x\" (UID: \"0378fcbc-4027-4275-b274-2c188f3ff442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k449x" Nov 03 03:17:47.901195 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:47.901026 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0378fcbc-4027-4275-b274-2c188f3ff442-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k449x\" (UID: \"0378fcbc-4027-4275-b274-2c188f3ff442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k449x" Nov 03 03:17:47.901195 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:47.901116 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/563c5b47-58ea-4fb8-b9f1-332cf8dae641-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bpbq7\" (UID: \"563c5b47-58ea-4fb8-b9f1-332cf8dae641\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bpbq7" Nov 03 03:17:47.901195 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:47.901153 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/563c5b47-58ea-4fb8-b9f1-332cf8dae641-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bpbq7\" (UID: \"563c5b47-58ea-4fb8-b9f1-332cf8dae641\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bpbq7" Nov 03 03:17:47.901195 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:47.901164 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0378fcbc-4027-4275-b274-2c188f3ff442-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k449x\" (UID: \"0378fcbc-4027-4275-b274-2c188f3ff442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k449x" Nov 03 03:17:47.901716 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:47.901288 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/563c5b47-58ea-4fb8-b9f1-332cf8dae641-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bpbq7\" (UID: \"563c5b47-58ea-4fb8-b9f1-332cf8dae641\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bpbq7" Nov 03 03:17:47.901716 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:47.901319 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0378fcbc-4027-4275-b274-2c188f3ff442-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k449x\" (UID: \"0378fcbc-4027-4275-b274-2c188f3ff442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k449x" Nov 03 03:17:47.901716 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:47.901373 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0378fcbc-4027-4275-b274-2c188f3ff442-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k449x\" (UID: \"0378fcbc-4027-4275-b274-2c188f3ff442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k449x" Nov 03 03:17:47.901716 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:47.901406 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/563c5b47-58ea-4fb8-b9f1-332cf8dae641-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bpbq7\" (UID: \"563c5b47-58ea-4fb8-b9f1-332cf8dae641\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bpbq7" Nov 03 03:17:47.901716 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:47.901440 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/563c5b47-58ea-4fb8-b9f1-332cf8dae641-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bpbq7\" (UID: \"563c5b47-58ea-4fb8-b9f1-332cf8dae641\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bpbq7" Nov 03 03:17:47.901716 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:47.901483 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/563c5b47-58ea-4fb8-b9f1-332cf8dae641-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bpbq7\" (UID: \"563c5b47-58ea-4fb8-b9f1-332cf8dae641\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bpbq7" Nov 03 03:17:47.902069 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:47.902046 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/563c5b47-58ea-4fb8-b9f1-332cf8dae641-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bpbq7\" (UID: \"563c5b47-58ea-4fb8-b9f1-332cf8dae641\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bpbq7" Nov 03 03:17:47.903198 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:47.903178 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/563c5b47-58ea-4fb8-b9f1-332cf8dae641-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bpbq7\" (UID: \"563c5b47-58ea-4fb8-b9f1-332cf8dae641\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bpbq7" Nov 03 03:17:47.903429 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:47.903407 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/563c5b47-58ea-4fb8-b9f1-332cf8dae641-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bpbq7\" (UID: \"563c5b47-58ea-4fb8-b9f1-332cf8dae641\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bpbq7" Nov 03 03:17:47.955336 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:47.955286 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/563c5b47-58ea-4fb8-b9f1-332cf8dae641-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bpbq7\" (UID: \"563c5b47-58ea-4fb8-b9f1-332cf8dae641\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bpbq7" Nov 03 03:17:47.955460 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:47.955333 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4qcb\" (UniqueName: \"kubernetes.io/projected/563c5b47-58ea-4fb8-b9f1-332cf8dae641-kube-api-access-j4qcb\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bpbq7\" (UID: \"563c5b47-58ea-4fb8-b9f1-332cf8dae641\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bpbq7" Nov 03 03:17:48.000167 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:48.000142 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bpbq7" Nov 03 03:17:48.001750 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:48.001727 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0378fcbc-4027-4275-b274-2c188f3ff442-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k449x\" (UID: \"0378fcbc-4027-4275-b274-2c188f3ff442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k449x" Nov 03 03:17:48.001856 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:48.001772 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0378fcbc-4027-4275-b274-2c188f3ff442-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k449x\" (UID: \"0378fcbc-4027-4275-b274-2c188f3ff442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k449x" Nov 03 03:17:48.001929 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:48.001876 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0378fcbc-4027-4275-b274-2c188f3ff442-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k449x\" (UID: \"0378fcbc-4027-4275-b274-2c188f3ff442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k449x" Nov 03 03:17:48.002006 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:48.001987 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdnhh\" (UniqueName: \"kubernetes.io/projected/0378fcbc-4027-4275-b274-2c188f3ff442-kube-api-access-kdnhh\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k449x\" (UID: \"0378fcbc-4027-4275-b274-2c188f3ff442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k449x" Nov 03 03:17:48.002073 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:48.002032 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0378fcbc-4027-4275-b274-2c188f3ff442-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k449x\" (UID: \"0378fcbc-4027-4275-b274-2c188f3ff442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k449x" Nov 03 03:17:48.002139 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:48.002078 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0378fcbc-4027-4275-b274-2c188f3ff442-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k449x\" (UID: \"0378fcbc-4027-4275-b274-2c188f3ff442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k449x" Nov 03 03:17:48.002139 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:48.002103 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0378fcbc-4027-4275-b274-2c188f3ff442-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k449x\" (UID: \"0378fcbc-4027-4275-b274-2c188f3ff442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k449x" Nov 03 03:17:48.002139 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:48.002109 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0378fcbc-4027-4275-b274-2c188f3ff442-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k449x\" (UID: \"0378fcbc-4027-4275-b274-2c188f3ff442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k449x" Nov 03 03:17:48.002293 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:48.002225 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0378fcbc-4027-4275-b274-2c188f3ff442-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k449x\" (UID: \"0378fcbc-4027-4275-b274-2c188f3ff442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k449x" Nov 03 03:17:48.002293 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:48.002258 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0378fcbc-4027-4275-b274-2c188f3ff442-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k449x\" (UID: \"0378fcbc-4027-4275-b274-2c188f3ff442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k449x" Nov 03 03:17:48.002447 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:48.002390 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0378fcbc-4027-4275-b274-2c188f3ff442-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k449x\" (UID: \"0378fcbc-4027-4275-b274-2c188f3ff442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k449x" Nov 03 03:17:48.002447 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:48.002414 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0378fcbc-4027-4275-b274-2c188f3ff442-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k449x\" (UID: \"0378fcbc-4027-4275-b274-2c188f3ff442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k449x" Nov 03 03:17:48.002592 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:48.002573 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0378fcbc-4027-4275-b274-2c188f3ff442-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k449x\" (UID: \"0378fcbc-4027-4275-b274-2c188f3ff442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k449x" Nov 03 03:17:48.002885 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:48.002863 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0378fcbc-4027-4275-b274-2c188f3ff442-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k449x\" (UID: \"0378fcbc-4027-4275-b274-2c188f3ff442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k449x" Nov 03 03:17:48.004469 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:48.004130 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0378fcbc-4027-4275-b274-2c188f3ff442-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k449x\" (UID: \"0378fcbc-4027-4275-b274-2c188f3ff442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k449x" Nov 03 03:17:48.004551 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:48.004528 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0378fcbc-4027-4275-b274-2c188f3ff442-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k449x\" (UID: \"0378fcbc-4027-4275-b274-2c188f3ff442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k449x" Nov 03 03:17:48.060703 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:48.060673 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0378fcbc-4027-4275-b274-2c188f3ff442-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k449x\" (UID: \"0378fcbc-4027-4275-b274-2c188f3ff442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k449x" Nov 03 03:17:48.060884 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:48.060859 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdnhh\" (UniqueName: \"kubernetes.io/projected/0378fcbc-4027-4275-b274-2c188f3ff442-kube-api-access-kdnhh\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k449x\" (UID: \"0378fcbc-4027-4275-b274-2c188f3ff442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k449x" Nov 03 03:17:48.346440 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:48.346397 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k449x" Nov 03 03:17:48.422248 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:48.422215 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bpbq7"] Nov 03 03:17:48.427501 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:17:48.427460 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod563c5b47_58ea_4fb8_b9f1_332cf8dae641.slice/crio-36ef6af3edb02628bd33b651779d3283962c5c1b89d1a898010adb159df6a4d8 WatchSource:0}: Error finding container 36ef6af3edb02628bd33b651779d3283962c5c1b89d1a898010adb159df6a4d8: Status 404 returned error can't find the container with id 36ef6af3edb02628bd33b651779d3283962c5c1b89d1a898010adb159df6a4d8 Nov 03 03:17:48.431755 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:48.431716 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 03:17:48.431852 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:48.431805 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 03:17:48.431895 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:48.431847 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 03:17:48.721583 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:48.721490 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k449x"] Nov 03 03:17:48.728306 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:17:48.728277 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0378fcbc_4027_4275_b274_2c188f3ff442.slice/crio-6a962595bc418a80d3d824b6b72b63d75e296b57a33011a2bc83d6617cf8c89d WatchSource:0}: Error finding container 6a962595bc418a80d3d824b6b72b63d75e296b57a33011a2bc83d6617cf8c89d: Status 404 returned error can't find the container with id 6a962595bc418a80d3d824b6b72b63d75e296b57a33011a2bc83d6617cf8c89d Nov 03 03:17:48.730854 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:48.730823 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 03:17:48.730964 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:48.730901 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 03:17:48.730964 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:48.730952 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 03:17:49.419388 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:49.417273 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bpbq7" event={"ID":"563c5b47-58ea-4fb8-b9f1-332cf8dae641","Type":"ContainerStarted","Data":"7a45fb84dd46637678ce676d8c84811a4b4e274ba550a6bded20fdf31730f7f9"} Nov 03 03:17:49.419388 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:49.417387 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bpbq7" event={"ID":"563c5b47-58ea-4fb8-b9f1-332cf8dae641","Type":"ContainerStarted","Data":"36ef6af3edb02628bd33b651779d3283962c5c1b89d1a898010adb159df6a4d8"} Nov 03 03:17:49.423131 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:49.423042 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k449x" event={"ID":"0378fcbc-4027-4275-b274-2c188f3ff442","Type":"ContainerStarted","Data":"3b5c482b198ceb232db9cab45cccc8a6b63a6c403fd81b126aa4da1dafd55f90"} Nov 03 03:17:49.423131 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:49.423097 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k449x" event={"ID":"0378fcbc-4027-4275-b274-2c188f3ff442","Type":"ContainerStarted","Data":"6a962595bc418a80d3d824b6b72b63d75e296b57a33011a2bc83d6617cf8c89d"} Nov 03 03:17:49.535756 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:49.535695 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bpbq7" podStartSLOduration=2.535679112 podStartE2EDuration="2.535679112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:17:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:17:49.534822345 +0000 UTC m=+5146.794020498" watchObservedRunningTime="2026-11-03 03:17:49.535679112 +0000 UTC m=+5146.794877266" Nov 03 03:17:50.002914 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:50.002880 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bpbq7" Nov 03 03:17:50.003598 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:50.003571 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-bpbq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.39:15021/healthz/ready\": dial tcp 10.134.1.39:15021: connect: connection refused" start-of-body= Nov 03 03:17:50.003785 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:50.003757 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bpbq7" podUID="563c5b47-58ea-4fb8-b9f1-332cf8dae641" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.39:15021/healthz/ready\": dial tcp 10.134.1.39:15021: connect: connection refused" Nov 03 03:17:50.348570 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:50.348519 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k449x" Nov 03 03:17:50.350771 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:50.350742 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-k449x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused" start-of-body= Nov 03 03:17:50.351007 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:50.350970 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k449x" podUID="0378fcbc-4027-4275-b274-2c188f3ff442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused" Nov 03 03:17:51.009784 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:51.009747 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-bpbq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.39:15021/healthz/ready\": dial tcp 10.134.1.39:15021: connect: connection refused" start-of-body= Nov 03 03:17:51.010423 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:51.010391 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bpbq7" podUID="563c5b47-58ea-4fb8-b9f1-332cf8dae641" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.39:15021/healthz/ready\": dial tcp 10.134.1.39:15021: connect: connection refused" Nov 03 03:17:51.346937 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:51.346897 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-k449x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused" start-of-body= Nov 03 03:17:51.347151 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:51.346953 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k449x" podUID="0378fcbc-4027-4275-b274-2c188f3ff442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused" Nov 03 03:17:52.001406 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:52.001370 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-bpbq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.39:15021/healthz/ready\": dial tcp 10.134.1.39:15021: connect: connection refused" start-of-body= Nov 03 03:17:52.001672 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:52.001645 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bpbq7" podUID="563c5b47-58ea-4fb8-b9f1-332cf8dae641" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.39:15021/healthz/ready\": dial tcp 10.134.1.39:15021: connect: connection refused" Nov 03 03:17:52.346943 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:52.346909 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-k449x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused" start-of-body= Nov 03 03:17:52.347324 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:52.346968 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k449x" podUID="0378fcbc-4027-4275-b274-2c188f3ff442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused" Nov 03 03:17:52.827766 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:52.827712 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k449x" podStartSLOduration=5.827696537 podStartE2EDuration="5.827696537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:17:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:17:49.652817332 +0000 UTC m=+5146.912015499" watchObservedRunningTime="2026-11-03 03:17:52.827696537 +0000 UTC m=+5150.086894690" Nov 03 03:17:52.828340 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:52.828323 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:17:53.003738 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:53.003702 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-bpbq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.39:15021/healthz/ready\": dial tcp 10.134.1.39:15021: connect: connection refused" start-of-body= Nov 03 03:17:53.004085 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:53.004053 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bpbq7" podUID="563c5b47-58ea-4fb8-b9f1-332cf8dae641" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.39:15021/healthz/ready\": dial tcp 10.134.1.39:15021: connect: connection refused" Nov 03 03:17:53.347611 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:53.347577 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-k449x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused" start-of-body= Nov 03 03:17:53.347998 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:53.347627 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k449x" podUID="0378fcbc-4027-4275-b274-2c188f3ff442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused" Nov 03 03:17:53.926464 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:17:53.926425 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:17:54.001230 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:54.000774 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-bpbq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.39:15021/healthz/ready\": dial tcp 10.134.1.39:15021: connect: connection refused" start-of-body= Nov 03 03:17:54.001230 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:54.000855 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bpbq7" podUID="563c5b47-58ea-4fb8-b9f1-332cf8dae641" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.39:15021/healthz/ready\": dial tcp 10.134.1.39:15021: connect: connection refused" Nov 03 03:17:54.347186 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:54.347153 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-k449x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused" start-of-body= Nov 03 03:17:54.347465 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:54.347431 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k449x" podUID="0378fcbc-4027-4275-b274-2c188f3ff442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused" Nov 03 03:17:55.001472 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:55.001420 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-bpbq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.39:15021/healthz/ready\": dial tcp 10.134.1.39:15021: connect: connection refused" start-of-body= Nov 03 03:17:55.001884 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:55.001496 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bpbq7" podUID="563c5b47-58ea-4fb8-b9f1-332cf8dae641" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.39:15021/healthz/ready\": dial tcp 10.134.1.39:15021: connect: connection refused" Nov 03 03:17:55.042422 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:55.042393 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5517bac8-nhgwv"] Nov 03 03:17:55.045089 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:55.045068 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5517bac8-nhgwv"] Nov 03 03:17:55.347038 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:55.346997 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-k449x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused" start-of-body= Nov 03 03:17:55.347194 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:55.347067 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k449x" podUID="0378fcbc-4027-4275-b274-2c188f3ff442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused" Nov 03 03:17:56.789397 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:56.789338 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bpbq7" Nov 03 03:17:56.790019 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:56.789994 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bpbq7" Nov 03 03:17:56.790889 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:56.790869 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bpbq7" Nov 03 03:17:56.849881 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:56.849852 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad0e4039-aeec-4828-80de-2e3490319eec" path="/var/lib/kubelet/pods/ad0e4039-aeec-4828-80de-2e3490319eec/volumes" Nov 03 03:17:57.119400 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:57.119343 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k449x" Nov 03 03:17:57.119745 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:57.119719 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k449x" Nov 03 03:17:57.120655 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:17:57.120637 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k449x" Nov 03 03:18:01.758915 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:18:01.758884 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:18:02.042025 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:18:02.041942 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdfaf0b9-6n8kr"] Nov 03 03:18:02.088482 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:18:02.088454 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdfaf0b9-6n8kr"] Nov 03 03:18:02.848792 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:18:02.848762 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b106f86-5a55-4614-9373-6a3f898d369f" path="/var/lib/kubelet/pods/8b106f86-5a55-4614-9373-6a3f898d369f/volumes" Nov 03 03:18:05.558790 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:18:05.558760 5605 scope.go:122] "RemoveContainer" containerID="e5586df9e1e3691598fa8898dad032b83f5fd113d5cb79a8c385aef364f17eb4" Nov 03 03:18:05.567104 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:18:05.567088 5605 scope.go:122] "RemoveContainer" containerID="d6a07f504b06c70dea2fe70aaf4bb7e1521f32d641fdf61aff0476a1c652e315" Nov 03 03:18:05.574241 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:18:05.574219 5605 scope.go:122] "RemoveContainer" containerID="02542ff862cba3a697859422870697aeb4b7a8af4efd62964bc776b1e1bf0f50" Nov 03 03:18:05.581116 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:18:05.581093 5605 scope.go:122] "RemoveContainer" containerID="3d4d906f825118ddbcf563b2edc336199c6eeebf3d5f82d22adf89e31d8ae4f7" Nov 03 03:18:05.588374 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:18:05.588344 5605 scope.go:122] "RemoveContainer" containerID="99d966d108a92cf368e79254ba7358cce6a52cd5b7cec7ac5f9e534c887f7b8d" Nov 03 03:18:09.953843 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:18:09.953809 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:18:17.049455 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:18:17.049420 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6406d0c-gv425"] Nov 03 03:18:17.096040 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:18:17.096010 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6406d0c-gv425"] Nov 03 03:18:18.851014 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:18:18.850979 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55eeee4d-8327-4e10-836e-d6609ad111a3" path="/var/lib/kubelet/pods/55eeee4d-8327-4e10-836e-d6609ad111a3/volumes" Nov 03 03:18:23.911441 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:18:23.911401 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:18:47.946601 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:18:47.946563 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:18:51.883654 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:18:51.883622 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-e934465c-kserve-64cd646f4d-c2nms"] Nov 03 03:18:51.887187 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:18:51.887166 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-e934465c-kserve-64cd646f4d-c2nms" Nov 03 03:18:51.936579 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:18:51.936552 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 03 03:18:51.983179 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:18:51.983144 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 03 03:18:51.983377 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:18:51.983191 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-nlcr8\"" Nov 03 03:18:51.983377 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:18:51.983269 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-e934465c-kserve-self-signed-certs\"" Nov 03 03:18:51.984568 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:18:51.984548 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-e934465c-kserve-64cd646f4d-c2nms"] Nov 03 03:18:52.032542 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:18:52.032507 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4320be89-aa2d-4145-ad6f-07ab28aedb77-model-cache\") pod \"e2e-prio-pool-e934465c-kserve-64cd646f4d-c2nms\" (UID: \"4320be89-aa2d-4145-ad6f-07ab28aedb77\") " pod="llm/e2e-prio-pool-e934465c-kserve-64cd646f4d-c2nms" Nov 03 03:18:52.032542 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:18:52.032547 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4320be89-aa2d-4145-ad6f-07ab28aedb77-dshm\") pod \"e2e-prio-pool-e934465c-kserve-64cd646f4d-c2nms\" (UID: \"4320be89-aa2d-4145-ad6f-07ab28aedb77\") " pod="llm/e2e-prio-pool-e934465c-kserve-64cd646f4d-c2nms" Nov 03 03:18:52.032765 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:18:52.032708 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vbpq\" (UniqueName: \"kubernetes.io/projected/4320be89-aa2d-4145-ad6f-07ab28aedb77-kube-api-access-2vbpq\") pod \"e2e-prio-pool-e934465c-kserve-64cd646f4d-c2nms\" (UID: \"4320be89-aa2d-4145-ad6f-07ab28aedb77\") " pod="llm/e2e-prio-pool-e934465c-kserve-64cd646f4d-c2nms" Nov 03 03:18:52.032765 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:18:52.032751 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4320be89-aa2d-4145-ad6f-07ab28aedb77-home\") pod \"e2e-prio-pool-e934465c-kserve-64cd646f4d-c2nms\" (UID: \"4320be89-aa2d-4145-ad6f-07ab28aedb77\") " pod="llm/e2e-prio-pool-e934465c-kserve-64cd646f4d-c2nms" Nov 03 03:18:52.032898 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:18:52.032869 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4320be89-aa2d-4145-ad6f-07ab28aedb77-tmp-dir\") pod \"e2e-prio-pool-e934465c-kserve-64cd646f4d-c2nms\" (UID: \"4320be89-aa2d-4145-ad6f-07ab28aedb77\") " pod="llm/e2e-prio-pool-e934465c-kserve-64cd646f4d-c2nms" Nov 03 03:18:52.032946 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:18:52.032907 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4320be89-aa2d-4145-ad6f-07ab28aedb77-tls-certs\") pod \"e2e-prio-pool-e934465c-kserve-64cd646f4d-c2nms\" (UID: \"4320be89-aa2d-4145-ad6f-07ab28aedb77\") " pod="llm/e2e-prio-pool-e934465c-kserve-64cd646f4d-c2nms" Nov 03 03:18:52.133565 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:18:52.133530 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4320be89-aa2d-4145-ad6f-07ab28aedb77-model-cache\") pod \"e2e-prio-pool-e934465c-kserve-64cd646f4d-c2nms\" (UID: \"4320be89-aa2d-4145-ad6f-07ab28aedb77\") " pod="llm/e2e-prio-pool-e934465c-kserve-64cd646f4d-c2nms" Nov 03 03:18:52.133565 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:18:52.133567 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4320be89-aa2d-4145-ad6f-07ab28aedb77-dshm\") pod \"e2e-prio-pool-e934465c-kserve-64cd646f4d-c2nms\" (UID: \"4320be89-aa2d-4145-ad6f-07ab28aedb77\") " pod="llm/e2e-prio-pool-e934465c-kserve-64cd646f4d-c2nms" Nov 03 03:18:52.133872 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:18:52.133754 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vbpq\" (UniqueName: \"kubernetes.io/projected/4320be89-aa2d-4145-ad6f-07ab28aedb77-kube-api-access-2vbpq\") pod \"e2e-prio-pool-e934465c-kserve-64cd646f4d-c2nms\" (UID: \"4320be89-aa2d-4145-ad6f-07ab28aedb77\") " pod="llm/e2e-prio-pool-e934465c-kserve-64cd646f4d-c2nms" Nov 03 03:18:52.133872 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:18:52.133809 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4320be89-aa2d-4145-ad6f-07ab28aedb77-home\") pod \"e2e-prio-pool-e934465c-kserve-64cd646f4d-c2nms\" (UID: \"4320be89-aa2d-4145-ad6f-07ab28aedb77\") " pod="llm/e2e-prio-pool-e934465c-kserve-64cd646f4d-c2nms" Nov 03 03:18:52.134145 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:18:52.133879 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4320be89-aa2d-4145-ad6f-07ab28aedb77-tmp-dir\") pod \"e2e-prio-pool-e934465c-kserve-64cd646f4d-c2nms\" (UID: \"4320be89-aa2d-4145-ad6f-07ab28aedb77\") " pod="llm/e2e-prio-pool-e934465c-kserve-64cd646f4d-c2nms" Nov 03 03:18:52.134145 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:18:52.133919 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4320be89-aa2d-4145-ad6f-07ab28aedb77-tls-certs\") pod \"e2e-prio-pool-e934465c-kserve-64cd646f4d-c2nms\" (UID: \"4320be89-aa2d-4145-ad6f-07ab28aedb77\") " pod="llm/e2e-prio-pool-e934465c-kserve-64cd646f4d-c2nms" Nov 03 03:18:52.134145 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:18:52.133988 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4320be89-aa2d-4145-ad6f-07ab28aedb77-model-cache\") pod \"e2e-prio-pool-e934465c-kserve-64cd646f4d-c2nms\" (UID: \"4320be89-aa2d-4145-ad6f-07ab28aedb77\") " pod="llm/e2e-prio-pool-e934465c-kserve-64cd646f4d-c2nms" Nov 03 03:18:52.134494 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:18:52.134218 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4320be89-aa2d-4145-ad6f-07ab28aedb77-home\") pod \"e2e-prio-pool-e934465c-kserve-64cd646f4d-c2nms\" (UID: \"4320be89-aa2d-4145-ad6f-07ab28aedb77\") " pod="llm/e2e-prio-pool-e934465c-kserve-64cd646f4d-c2nms" Nov 03 03:18:52.134494 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:18:52.134244 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4320be89-aa2d-4145-ad6f-07ab28aedb77-tmp-dir\") pod \"e2e-prio-pool-e934465c-kserve-64cd646f4d-c2nms\" (UID: \"4320be89-aa2d-4145-ad6f-07ab28aedb77\") " pod="llm/e2e-prio-pool-e934465c-kserve-64cd646f4d-c2nms" Nov 03 03:18:52.135973 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:18:52.135955 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4320be89-aa2d-4145-ad6f-07ab28aedb77-dshm\") pod \"e2e-prio-pool-e934465c-kserve-64cd646f4d-c2nms\" (UID: \"4320be89-aa2d-4145-ad6f-07ab28aedb77\") " pod="llm/e2e-prio-pool-e934465c-kserve-64cd646f4d-c2nms" Nov 03 03:18:52.136411 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:18:52.136390 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4320be89-aa2d-4145-ad6f-07ab28aedb77-tls-certs\") pod \"e2e-prio-pool-e934465c-kserve-64cd646f4d-c2nms\" (UID: \"4320be89-aa2d-4145-ad6f-07ab28aedb77\") " pod="llm/e2e-prio-pool-e934465c-kserve-64cd646f4d-c2nms" Nov 03 03:18:52.187200 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:18:52.187171 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vbpq\" (UniqueName: \"kubernetes.io/projected/4320be89-aa2d-4145-ad6f-07ab28aedb77-kube-api-access-2vbpq\") pod \"e2e-prio-pool-e934465c-kserve-64cd646f4d-c2nms\" (UID: \"4320be89-aa2d-4145-ad6f-07ab28aedb77\") " pod="llm/e2e-prio-pool-e934465c-kserve-64cd646f4d-c2nms" Nov 03 03:18:52.199063 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:18:52.199035 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-e934465c-kserve-64cd646f4d-c2nms" Nov 03 03:18:52.564465 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:18:52.564434 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-e934465c-kserve-64cd646f4d-c2nms"] Nov 03 03:18:52.571297 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:18:52.571267 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4320be89_aa2d_4145_ad6f_07ab28aedb77.slice/crio-aa30763ba54c2be3eb49304a91e61a20fd1ec624031e41d261a8c1ee2b0d9236 WatchSource:0}: Error finding container aa30763ba54c2be3eb49304a91e61a20fd1ec624031e41d261a8c1ee2b0d9236: Status 404 returned error can't find the container with id aa30763ba54c2be3eb49304a91e61a20fd1ec624031e41d261a8c1ee2b0d9236 Nov 03 03:18:52.782694 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:18:52.782659 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-e934465c-kserve-64cd646f4d-c2nms" event={"ID":"4320be89-aa2d-4145-ad6f-07ab28aedb77","Type":"ContainerStarted","Data":"45a5bb2eaee03c47ce62ebcaefcee220c73fabf79f5199a21de867b533c9e7f8"} Nov 03 03:18:52.782694 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:18:52.782697 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-e934465c-kserve-64cd646f4d-c2nms" event={"ID":"4320be89-aa2d-4145-ad6f-07ab28aedb77","Type":"ContainerStarted","Data":"aa30763ba54c2be3eb49304a91e61a20fd1ec624031e41d261a8c1ee2b0d9236"} Nov 03 03:18:52.889859 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:18:52.889761 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-e934465c-kserve-64cd646f4d-c2nms" podStartSLOduration=1.8897476850000001 podStartE2EDuration="1.889747685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:18:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:18:52.888579348 +0000 UTC m=+5210.147777503" watchObservedRunningTime="2026-11-03 03:18:52.889747685 +0000 UTC m=+5210.148945839" Nov 03 03:18:53.200262 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:18:53.200187 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-e934465c-kserve-64cd646f4d-c2nms" Nov 03 03:18:53.911731 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:18:53.911693 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:18:56.945604 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:18:56.945574 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:19:02.199312 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:19:02.199277 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-prio-pool-e934465c-kserve-64cd646f4d-c2nms" Nov 03 03:19:02.211323 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:19:02.211300 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-prio-pool-e934465c-kserve-64cd646f4d-c2nms" Nov 03 03:19:02.876864 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:19:02.876833 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-e934465c-kserve-64cd646f4d-c2nms" Nov 03 03:19:05.618570 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:19:05.618540 5605 scope.go:122] "RemoveContainer" containerID="6855d79955c48bb4ec56c1a705da9676ead08e637c085ad5075177b35aa4eb5d" Nov 03 03:19:05.626781 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:19:05.626757 5605 scope.go:122] "RemoveContainer" containerID="eb9795ec9e0abd928d836f52258bccb337c9caee3e1839e2bc25ea997f931035" Nov 03 03:19:05.634137 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:19:05.634119 5605 scope.go:122] "RemoveContainer" containerID="ce21f876def0f1608d7b67a44826dc7200a28faf6a659f0eecf8d29d2e365bdd" Nov 03 03:19:05.641166 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:19:05.641148 5605 scope.go:122] "RemoveContainer" containerID="3f6d81044c13d278bdae1835818c1f8a2ec4479e9216af5e736222282aea37f4" Nov 03 03:19:23.911243 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:19:23.911159 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:19:24.055747 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:19:24.055710 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6aba08c1-hndzn"] Nov 03 03:19:24.102312 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:19:24.102284 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6aba08c1-hndzn"] Nov 03 03:19:24.849231 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:19:24.849199 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09fa4230-1064-47be-9d05-34232e5cb1e3" path="/var/lib/kubelet/pods/09fa4230-1064-47be-9d05-34232e5cb1e3/volumes" Nov 03 03:19:36.046554 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:19:36.046522 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-796c034d-4lksf"] Nov 03 03:19:36.093230 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:19:36.093201 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-796c034d-4lksf"] Nov 03 03:19:36.848140 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:19:36.848109 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e5078d7-c363-438d-891e-e03666ee4fb4" path="/var/lib/kubelet/pods/3e5078d7-c363-438d-891e-e03666ee4fb4/volumes" Nov 03 03:19:39.469140 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:19:39.469104 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:19:42.300390 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:19:42.299445 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-p8fqq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 03:19:42.300390 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:19:42.299523 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p8fqq" podUID="0bedb6e8-5829-4cc2-a912-35131c941e09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 03:19:42.328508 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:19:42.328444 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-dbd7q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 03:19:42.328797 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:19:42.328751 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dbd7q" podUID="dc490604-2e00-49dd-8423-47fc70709678" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 03:19:42.837503 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:19:42.837451 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-bpbq7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 03:19:42.837820 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:19:42.837774 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bpbq7" podUID="563c5b47-58ea-4fb8-b9f1-332cf8dae641" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 03:19:43.126645 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:19:43.126548 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-k449x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 03:19:43.126969 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:19:43.126929 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k449x" podUID="0378fcbc-4027-4275-b274-2c188f3ff442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 03:19:51.860459 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:19:51.860424 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f24d4143-zl2b6"] Nov 03 03:19:51.863827 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:19:51.863805 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f24d4143-zl2b6" Nov 03 03:19:51.959400 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:19:51.959351 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:19:51.959548 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:19:51.959424 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sb4qh\"" Nov 03 03:19:51.959548 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:19:51.959437 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:19:51.959548 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:19:51.959437 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:19:51.960805 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:19:51.960786 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f24d4143-zl2b6"] Nov 03 03:19:52.017404 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:19:52.017379 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8730b0b5-e304-4b23-89a1-73af5ccd0e60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f24d4143-zl2b6\" (UID: \"8730b0b5-e304-4b23-89a1-73af5ccd0e60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f24d4143-zl2b6" Nov 03 03:19:52.017595 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:19:52.017433 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhq9v\" (UniqueName: \"kubernetes.io/projected/8730b0b5-e304-4b23-89a1-73af5ccd0e60-kube-api-access-mhq9v\") pod \"maas-api-revoke-sub-f24d4143-zl2b6\" (UID: \"8730b0b5-e304-4b23-89a1-73af5ccd0e60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f24d4143-zl2b6" Nov 03 03:19:52.017595 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:19:52.017481 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8730b0b5-e304-4b23-89a1-73af5ccd0e60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f24d4143-zl2b6\" (UID: \"8730b0b5-e304-4b23-89a1-73af5ccd0e60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f24d4143-zl2b6" Nov 03 03:19:52.118769 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:19:52.118701 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8730b0b5-e304-4b23-89a1-73af5ccd0e60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f24d4143-zl2b6\" (UID: \"8730b0b5-e304-4b23-89a1-73af5ccd0e60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f24d4143-zl2b6" Nov 03 03:19:52.118769 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:19:52.118756 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhq9v\" (UniqueName: \"kubernetes.io/projected/8730b0b5-e304-4b23-89a1-73af5ccd0e60-kube-api-access-mhq9v\") pod \"maas-api-revoke-sub-f24d4143-zl2b6\" (UID: \"8730b0b5-e304-4b23-89a1-73af5ccd0e60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f24d4143-zl2b6" Nov 03 03:19:52.118970 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:19:52.118778 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8730b0b5-e304-4b23-89a1-73af5ccd0e60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f24d4143-zl2b6\" (UID: \"8730b0b5-e304-4b23-89a1-73af5ccd0e60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f24d4143-zl2b6" Nov 03 03:19:52.119416 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:19:52.119393 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8730b0b5-e304-4b23-89a1-73af5ccd0e60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f24d4143-zl2b6\" (UID: \"8730b0b5-e304-4b23-89a1-73af5ccd0e60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f24d4143-zl2b6" Nov 03 03:19:52.121050 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:19:52.121026 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8730b0b5-e304-4b23-89a1-73af5ccd0e60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f24d4143-zl2b6\" (UID: \"8730b0b5-e304-4b23-89a1-73af5ccd0e60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f24d4143-zl2b6" Nov 03 03:19:52.171893 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:19:52.171867 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhq9v\" (UniqueName: \"kubernetes.io/projected/8730b0b5-e304-4b23-89a1-73af5ccd0e60-kube-api-access-mhq9v\") pod \"maas-api-revoke-sub-f24d4143-zl2b6\" (UID: \"8730b0b5-e304-4b23-89a1-73af5ccd0e60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f24d4143-zl2b6" Nov 03 03:19:52.176647 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:19:52.176630 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f24d4143-zl2b6" Nov 03 03:19:52.542877 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:19:52.542799 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f24d4143-zl2b6"] Nov 03 03:19:52.548841 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:19:52.548815 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8730b0b5_e304_4b23_89a1_73af5ccd0e60.slice/crio-0d18add0fbbeecf9d63c71e7bc972254187cfc92055ef1f93635eee580d4e3e0 WatchSource:0}: Error finding container 0d18add0fbbeecf9d63c71e7bc972254187cfc92055ef1f93635eee580d4e3e0: Status 404 returned error can't find the container with id 0d18add0fbbeecf9d63c71e7bc972254187cfc92055ef1f93635eee580d4e3e0 Nov 03 03:19:53.110294 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:19:53.110262 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f24d4143-zl2b6" event={"ID":"8730b0b5-e304-4b23-89a1-73af5ccd0e60","Type":"ContainerStarted","Data":"a94ca6801a8efca54dfb6ee1792d3bedc70c59860430a54259d10e7d8eeef586"} Nov 03 03:19:53.110658 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:19:53.110313 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f24d4143-zl2b6" event={"ID":"8730b0b5-e304-4b23-89a1-73af5ccd0e60","Type":"ContainerStarted","Data":"0d18add0fbbeecf9d63c71e7bc972254187cfc92055ef1f93635eee580d4e3e0"} Nov 03 03:19:53.217709 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:19:53.217614 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f24d4143-zl2b6" podStartSLOduration=2.2175986659999998 podStartE2EDuration="2.217598666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:19:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:19:53.21721358 +0000 UTC m=+5270.476411735" watchObservedRunningTime="2026-11-03 03:19:53.217598666 +0000 UTC m=+5270.476796821" Nov 03 03:19:53.911654 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:19:53.911620 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:19:54.116581 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:19:54.116550 5605 generic.go:356] "Generic (PLEG): container finished" podID="8730b0b5-e304-4b23-89a1-73af5ccd0e60" containerID="a94ca6801a8efca54dfb6ee1792d3bedc70c59860430a54259d10e7d8eeef586" exitCode=0 Nov 03 03:19:54.116973 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:19:54.116638 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f24d4143-zl2b6" event={"ID":"8730b0b5-e304-4b23-89a1-73af5ccd0e60","Type":"ContainerDied","Data":"a94ca6801a8efca54dfb6ee1792d3bedc70c59860430a54259d10e7d8eeef586"} Nov 03 03:19:55.248572 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:19:55.248546 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f24d4143-zl2b6" Nov 03 03:19:55.344677 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:19:55.344635 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8730b0b5-e304-4b23-89a1-73af5ccd0e60-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8730b0b5-e304-4b23-89a1-73af5ccd0e60-maas-api-openshift-ca\") pod \"8730b0b5-e304-4b23-89a1-73af5ccd0e60\" (UID: \"8730b0b5-e304-4b23-89a1-73af5ccd0e60\") " Nov 03 03:19:55.344858 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:19:55.344735 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8730b0b5-e304-4b23-89a1-73af5ccd0e60-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8730b0b5-e304-4b23-89a1-73af5ccd0e60-maas-api-xks-ca\") pod \"8730b0b5-e304-4b23-89a1-73af5ccd0e60\" (UID: \"8730b0b5-e304-4b23-89a1-73af5ccd0e60\") " Nov 03 03:19:55.344858 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:19:55.344837 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8730b0b5-e304-4b23-89a1-73af5ccd0e60-kube-api-access-mhq9v\" (UniqueName: \"kubernetes.io/projected/8730b0b5-e304-4b23-89a1-73af5ccd0e60-kube-api-access-mhq9v\") pod \"8730b0b5-e304-4b23-89a1-73af5ccd0e60\" (UID: \"8730b0b5-e304-4b23-89a1-73af5ccd0e60\") " Nov 03 03:19:55.345109 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:19:55.345075 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8730b0b5-e304-4b23-89a1-73af5ccd0e60-maas-api-openshift-ca" pod "8730b0b5-e304-4b23-89a1-73af5ccd0e60" (UID: "8730b0b5-e304-4b23-89a1-73af5ccd0e60"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:19:55.346912 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:19:55.346883 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8730b0b5-e304-4b23-89a1-73af5ccd0e60-kube-api-access-mhq9v" pod "8730b0b5-e304-4b23-89a1-73af5ccd0e60" (UID: "8730b0b5-e304-4b23-89a1-73af5ccd0e60"). InnerVolumeSpecName "kube-api-access-mhq9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:19:55.346912 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:19:55.346904 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8730b0b5-e304-4b23-89a1-73af5ccd0e60-maas-api-xks-ca" pod "8730b0b5-e304-4b23-89a1-73af5ccd0e60" (UID: "8730b0b5-e304-4b23-89a1-73af5ccd0e60"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:19:55.446617 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:19:55.446526 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8730b0b5-e304-4b23-89a1-73af5ccd0e60-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:19:55.446617 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:19:55.446560 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8730b0b5-e304-4b23-89a1-73af5ccd0e60-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:19:55.446617 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:19:55.446570 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhq9v\" (UniqueName: \"kubernetes.io/projected/8730b0b5-e304-4b23-89a1-73af5ccd0e60-kube-api-access-mhq9v\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:19:56.126612 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:19:56.126582 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f24d4143-zl2b6" Nov 03 03:19:56.126784 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:19:56.126583 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f24d4143-zl2b6" event={"ID":"8730b0b5-e304-4b23-89a1-73af5ccd0e60","Type":"ContainerDied","Data":"0d18add0fbbeecf9d63c71e7bc972254187cfc92055ef1f93635eee580d4e3e0"} Nov 03 03:19:56.126784 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:19:56.126704 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d18add0fbbeecf9d63c71e7bc972254187cfc92055ef1f93635eee580d4e3e0" Nov 03 03:20:04.666271 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:04.666237 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:20:04.934499 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:04.934418 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-e934465c-kserve-64cd646f4d-c2nms"] Nov 03 03:20:05.668041 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:05.668010 5605 scope.go:122] "RemoveContainer" containerID="8190cd1b0cfee93a5c3d708ca74e0325b844b6f74ba256463212bdc25e072524" Nov 03 03:20:10.689165 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:10.689128 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-622d7413-6lpd7"] Nov 03 03:20:10.689576 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:10.689465 5605 cpu_manager.go:414] "containerMap: removing container" podUID="8730b0b5-e304-4b23-89a1-73af5ccd0e60" containerName="revoke-keys" Nov 03 03:20:10.689576 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:10.689475 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8730b0b5-e304-4b23-89a1-73af5ccd0e60" containerName="revoke-keys" Nov 03 03:20:10.689576 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:10.689559 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="8730b0b5-e304-4b23-89a1-73af5ccd0e60" containerName="revoke-keys" Nov 03 03:20:10.693922 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:10.693897 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-622d7413-6lpd7" Nov 03 03:20:10.782666 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:10.782632 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04712f50-7574-4123-8009-984532ac7a65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-622d7413-6lpd7\" (UID: \"04712f50-7574-4123-8009-984532ac7a65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-622d7413-6lpd7" Nov 03 03:20:10.782810 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:10.782694 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04712f50-7574-4123-8009-984532ac7a65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-622d7413-6lpd7\" (UID: \"04712f50-7574-4123-8009-984532ac7a65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-622d7413-6lpd7" Nov 03 03:20:10.782869 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:10.782799 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cpfd\" (UniqueName: \"kubernetes.io/projected/04712f50-7574-4123-8009-984532ac7a65-kube-api-access-9cpfd\") pod \"maas-api-revoke-sub-622d7413-6lpd7\" (UID: \"04712f50-7574-4123-8009-984532ac7a65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-622d7413-6lpd7" Nov 03 03:20:10.790672 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:10.790645 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:20:10.790672 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:10.790658 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:20:10.790672 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:10.790660 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:20:10.790884 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:10.790734 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sb4qh\"" Nov 03 03:20:10.792105 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:10.792086 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-622d7413-6lpd7"] Nov 03 03:20:10.883559 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:10.883530 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cpfd\" (UniqueName: \"kubernetes.io/projected/04712f50-7574-4123-8009-984532ac7a65-kube-api-access-9cpfd\") pod \"maas-api-revoke-sub-622d7413-6lpd7\" (UID: \"04712f50-7574-4123-8009-984532ac7a65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-622d7413-6lpd7" Nov 03 03:20:10.883752 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:10.883578 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04712f50-7574-4123-8009-984532ac7a65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-622d7413-6lpd7\" (UID: \"04712f50-7574-4123-8009-984532ac7a65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-622d7413-6lpd7" Nov 03 03:20:10.883752 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:10.883620 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04712f50-7574-4123-8009-984532ac7a65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-622d7413-6lpd7\" (UID: \"04712f50-7574-4123-8009-984532ac7a65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-622d7413-6lpd7" Nov 03 03:20:10.884251 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:10.884224 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04712f50-7574-4123-8009-984532ac7a65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-622d7413-6lpd7\" (UID: \"04712f50-7574-4123-8009-984532ac7a65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-622d7413-6lpd7" Nov 03 03:20:10.886063 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:10.886047 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04712f50-7574-4123-8009-984532ac7a65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-622d7413-6lpd7\" (UID: \"04712f50-7574-4123-8009-984532ac7a65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-622d7413-6lpd7" Nov 03 03:20:10.936720 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:10.936695 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cpfd\" (UniqueName: \"kubernetes.io/projected/04712f50-7574-4123-8009-984532ac7a65-kube-api-access-9cpfd\") pod \"maas-api-revoke-sub-622d7413-6lpd7\" (UID: \"04712f50-7574-4123-8009-984532ac7a65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-622d7413-6lpd7" Nov 03 03:20:11.006556 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:11.006498 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-622d7413-6lpd7" Nov 03 03:20:11.370576 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:11.370532 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-622d7413-6lpd7"] Nov 03 03:20:11.379888 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:20:11.378989 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04712f50_7574_4123_8009_984532ac7a65.slice/crio-c3c068e38aebf061da7821f8378f1075333e4b604d8ca4553add63cc47068692 WatchSource:0}: Error finding container c3c068e38aebf061da7821f8378f1075333e4b604d8ca4553add63cc47068692: Status 404 returned error can't find the container with id c3c068e38aebf061da7821f8378f1075333e4b604d8ca4553add63cc47068692 Nov 03 03:20:12.209259 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:12.209221 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-622d7413-6lpd7" event={"ID":"04712f50-7574-4123-8009-984532ac7a65","Type":"ContainerStarted","Data":"4942433aecc2bc9027069a3f7a80234698e9996071ab549809fba3e30c013014"} Nov 03 03:20:12.214008 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:12.213968 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-622d7413-6lpd7" event={"ID":"04712f50-7574-4123-8009-984532ac7a65","Type":"ContainerStarted","Data":"c3c068e38aebf061da7821f8378f1075333e4b604d8ca4553add63cc47068692"} Nov 03 03:20:13.243397 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:13.239497 5605 generic.go:356] "Generic (PLEG): container finished" podID="04712f50-7574-4123-8009-984532ac7a65" containerID="4942433aecc2bc9027069a3f7a80234698e9996071ab549809fba3e30c013014" exitCode=0 Nov 03 03:20:13.244554 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:13.244525 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-622d7413-6lpd7" event={"ID":"04712f50-7574-4123-8009-984532ac7a65","Type":"ContainerDied","Data":"4942433aecc2bc9027069a3f7a80234698e9996071ab549809fba3e30c013014"} Nov 03 03:20:14.373728 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:14.373702 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-622d7413-6lpd7" Nov 03 03:20:14.458098 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:14.458064 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04712f50-7574-4123-8009-984532ac7a65-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04712f50-7574-4123-8009-984532ac7a65-maas-api-xks-ca\") pod \"04712f50-7574-4123-8009-984532ac7a65\" (UID: \"04712f50-7574-4123-8009-984532ac7a65\") " Nov 03 03:20:14.458276 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:14.458123 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04712f50-7574-4123-8009-984532ac7a65-kube-api-access-9cpfd\" (UniqueName: \"kubernetes.io/projected/04712f50-7574-4123-8009-984532ac7a65-kube-api-access-9cpfd\") pod \"04712f50-7574-4123-8009-984532ac7a65\" (UID: \"04712f50-7574-4123-8009-984532ac7a65\") " Nov 03 03:20:14.458276 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:14.458154 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04712f50-7574-4123-8009-984532ac7a65-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04712f50-7574-4123-8009-984532ac7a65-maas-api-openshift-ca\") pod \"04712f50-7574-4123-8009-984532ac7a65\" (UID: \"04712f50-7574-4123-8009-984532ac7a65\") " Nov 03 03:20:14.458632 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:14.458602 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04712f50-7574-4123-8009-984532ac7a65-maas-api-openshift-ca" pod "04712f50-7574-4123-8009-984532ac7a65" (UID: "04712f50-7574-4123-8009-984532ac7a65"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:20:14.460189 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:14.460164 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04712f50-7574-4123-8009-984532ac7a65-maas-api-xks-ca" pod "04712f50-7574-4123-8009-984532ac7a65" (UID: "04712f50-7574-4123-8009-984532ac7a65"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:20:14.460262 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:14.460223 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04712f50-7574-4123-8009-984532ac7a65-kube-api-access-9cpfd" pod "04712f50-7574-4123-8009-984532ac7a65" (UID: "04712f50-7574-4123-8009-984532ac7a65"). InnerVolumeSpecName "kube-api-access-9cpfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:20:14.559726 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:14.559694 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04712f50-7574-4123-8009-984532ac7a65-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:20:14.559888 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:14.559732 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cpfd\" (UniqueName: \"kubernetes.io/projected/04712f50-7574-4123-8009-984532ac7a65-kube-api-access-9cpfd\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:20:14.559888 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:14.559749 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04712f50-7574-4123-8009-984532ac7a65-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:20:15.253614 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:15.253541 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-622d7413-6lpd7" Nov 03 03:20:15.253614 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:15.253551 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-622d7413-6lpd7" event={"ID":"04712f50-7574-4123-8009-984532ac7a65","Type":"ContainerDied","Data":"c3c068e38aebf061da7821f8378f1075333e4b604d8ca4553add63cc47068692"} Nov 03 03:20:15.253614 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:15.253594 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3c068e38aebf061da7821f8378f1075333e4b604d8ca4553add63cc47068692" Nov 03 03:20:16.949201 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:16.949162 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:20:19.955587 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:19.955541 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-e934465c-kserve-64cd646f4d-c2nms" podUID="4320be89-aa2d-4145-ad6f-07ab28aedb77" containerName="main" containerID="cri-o://45a5bb2eaee03c47ce62ebcaefcee220c73fabf79f5199a21de867b533c9e7f8" gracePeriod=45 Nov 03 03:20:20.193945 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:20.193913 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-e934465c-kserve-64cd646f4d-c2nms" Nov 03 03:20:20.278982 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:20.278897 5605 generic.go:356] "Generic (PLEG): container finished" podID="4320be89-aa2d-4145-ad6f-07ab28aedb77" containerID="45a5bb2eaee03c47ce62ebcaefcee220c73fabf79f5199a21de867b533c9e7f8" exitCode=0 Nov 03 03:20:20.278982 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:20.278936 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-e934465c-kserve-64cd646f4d-c2nms" event={"ID":"4320be89-aa2d-4145-ad6f-07ab28aedb77","Type":"ContainerDied","Data":"45a5bb2eaee03c47ce62ebcaefcee220c73fabf79f5199a21de867b533c9e7f8"} Nov 03 03:20:20.278982 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:20.278940 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-e934465c-kserve-64cd646f4d-c2nms" Nov 03 03:20:20.278982 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:20.278976 5605 scope.go:122] "RemoveContainer" containerID="45a5bb2eaee03c47ce62ebcaefcee220c73fabf79f5199a21de867b533c9e7f8" Nov 03 03:20:20.279270 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:20.278965 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-e934465c-kserve-64cd646f4d-c2nms" event={"ID":"4320be89-aa2d-4145-ad6f-07ab28aedb77","Type":"ContainerDied","Data":"aa30763ba54c2be3eb49304a91e61a20fd1ec624031e41d261a8c1ee2b0d9236"} Nov 03 03:20:20.286857 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:20.286838 5605 scope.go:122] "RemoveContainer" containerID="45a5bb2eaee03c47ce62ebcaefcee220c73fabf79f5199a21de867b533c9e7f8" Nov 03 03:20:20.287132 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:20:20.287114 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45a5bb2eaee03c47ce62ebcaefcee220c73fabf79f5199a21de867b533c9e7f8\": container with ID starting with 45a5bb2eaee03c47ce62ebcaefcee220c73fabf79f5199a21de867b533c9e7f8 not found: ID does not exist" containerID="45a5bb2eaee03c47ce62ebcaefcee220c73fabf79f5199a21de867b533c9e7f8" Nov 03 03:20:20.287179 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:20.287142 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45a5bb2eaee03c47ce62ebcaefcee220c73fabf79f5199a21de867b533c9e7f8"} err="failed to get container status \"45a5bb2eaee03c47ce62ebcaefcee220c73fabf79f5199a21de867b533c9e7f8\": rpc error: code = NotFound desc = could not find container \"45a5bb2eaee03c47ce62ebcaefcee220c73fabf79f5199a21de867b533c9e7f8\": container with ID starting with 45a5bb2eaee03c47ce62ebcaefcee220c73fabf79f5199a21de867b533c9e7f8 not found: ID does not exist" Nov 03 03:20:20.305567 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:20.305538 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4320be89-aa2d-4145-ad6f-07ab28aedb77-home\" (UniqueName: \"kubernetes.io/empty-dir/4320be89-aa2d-4145-ad6f-07ab28aedb77-home\") pod \"4320be89-aa2d-4145-ad6f-07ab28aedb77\" (UID: \"4320be89-aa2d-4145-ad6f-07ab28aedb77\") " Nov 03 03:20:20.305687 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:20.305667 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4320be89-aa2d-4145-ad6f-07ab28aedb77-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4320be89-aa2d-4145-ad6f-07ab28aedb77-tmp-dir\") pod \"4320be89-aa2d-4145-ad6f-07ab28aedb77\" (UID: \"4320be89-aa2d-4145-ad6f-07ab28aedb77\") " Nov 03 03:20:20.305755 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:20.305737 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4320be89-aa2d-4145-ad6f-07ab28aedb77-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4320be89-aa2d-4145-ad6f-07ab28aedb77-model-cache\") pod \"4320be89-aa2d-4145-ad6f-07ab28aedb77\" (UID: \"4320be89-aa2d-4145-ad6f-07ab28aedb77\") " Nov 03 03:20:20.305818 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:20.305791 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4320be89-aa2d-4145-ad6f-07ab28aedb77-kube-api-access-2vbpq\" (UniqueName: \"kubernetes.io/projected/4320be89-aa2d-4145-ad6f-07ab28aedb77-kube-api-access-2vbpq\") pod \"4320be89-aa2d-4145-ad6f-07ab28aedb77\" (UID: \"4320be89-aa2d-4145-ad6f-07ab28aedb77\") " Nov 03 03:20:20.305893 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:20.305876 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4320be89-aa2d-4145-ad6f-07ab28aedb77-tls-certs\" (UniqueName: \"kubernetes.io/secret/4320be89-aa2d-4145-ad6f-07ab28aedb77-tls-certs\") pod \"4320be89-aa2d-4145-ad6f-07ab28aedb77\" (UID: \"4320be89-aa2d-4145-ad6f-07ab28aedb77\") " Nov 03 03:20:20.305963 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:20.305918 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4320be89-aa2d-4145-ad6f-07ab28aedb77-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4320be89-aa2d-4145-ad6f-07ab28aedb77-dshm\") pod \"4320be89-aa2d-4145-ad6f-07ab28aedb77\" (UID: \"4320be89-aa2d-4145-ad6f-07ab28aedb77\") " Nov 03 03:20:20.306017 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:20.305945 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4320be89-aa2d-4145-ad6f-07ab28aedb77-tmp-dir" pod "4320be89-aa2d-4145-ad6f-07ab28aedb77" (UID: "4320be89-aa2d-4145-ad6f-07ab28aedb77"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:20:20.306072 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:20.306013 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4320be89-aa2d-4145-ad6f-07ab28aedb77-model-cache" pod "4320be89-aa2d-4145-ad6f-07ab28aedb77" (UID: "4320be89-aa2d-4145-ad6f-07ab28aedb77"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:20:20.306072 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:20.306039 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4320be89-aa2d-4145-ad6f-07ab28aedb77-home" pod "4320be89-aa2d-4145-ad6f-07ab28aedb77" (UID: "4320be89-aa2d-4145-ad6f-07ab28aedb77"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:20:20.306271 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:20.306251 5605 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4320be89-aa2d-4145-ad6f-07ab28aedb77-home\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:20:20.306348 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:20.306277 5605 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4320be89-aa2d-4145-ad6f-07ab28aedb77-tmp-dir\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:20:20.306348 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:20.306292 5605 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4320be89-aa2d-4145-ad6f-07ab28aedb77-model-cache\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:20:20.308790 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:20.308765 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4320be89-aa2d-4145-ad6f-07ab28aedb77-kube-api-access-2vbpq" pod "4320be89-aa2d-4145-ad6f-07ab28aedb77" (UID: "4320be89-aa2d-4145-ad6f-07ab28aedb77"). InnerVolumeSpecName "kube-api-access-2vbpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:20:20.308902 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:20.308827 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4320be89-aa2d-4145-ad6f-07ab28aedb77-dshm" pod "4320be89-aa2d-4145-ad6f-07ab28aedb77" (UID: "4320be89-aa2d-4145-ad6f-07ab28aedb77"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:20:20.309484 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:20.309456 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4320be89-aa2d-4145-ad6f-07ab28aedb77-tls-certs" pod "4320be89-aa2d-4145-ad6f-07ab28aedb77" (UID: "4320be89-aa2d-4145-ad6f-07ab28aedb77"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:20:20.407459 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:20.407431 5605 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4320be89-aa2d-4145-ad6f-07ab28aedb77-tls-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:20:20.407459 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:20.407453 5605 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4320be89-aa2d-4145-ad6f-07ab28aedb77-dshm\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:20:20.407459 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:20.407463 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vbpq\" (UniqueName: \"kubernetes.io/projected/4320be89-aa2d-4145-ad6f-07ab28aedb77-kube-api-access-2vbpq\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:20:20.691139 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:20.691109 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-e934465c-kserve-64cd646f4d-c2nms"] Nov 03 03:20:20.694622 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:20.694600 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-e934465c-kserve-64cd646f4d-c2nms"] Nov 03 03:20:20.851598 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:20.849926 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4320be89-aa2d-4145-ad6f-07ab28aedb77" path="/var/lib/kubelet/pods/4320be89-aa2d-4145-ad6f-07ab28aedb77/volumes" Nov 03 03:20:23.913796 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:20:23.913759 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:20:23.924437 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:23.924407 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl"] Nov 03 03:20:23.924755 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:23.924741 5605 cpu_manager.go:414] "containerMap: removing container" podUID="4320be89-aa2d-4145-ad6f-07ab28aedb77" containerName="main" Nov 03 03:20:23.924840 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:23.924755 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4320be89-aa2d-4145-ad6f-07ab28aedb77" containerName="main" Nov 03 03:20:23.924840 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:23.924778 5605 cpu_manager.go:414] "containerMap: removing container" podUID="04712f50-7574-4123-8009-984532ac7a65" containerName="revoke-keys" Nov 03 03:20:23.924840 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:23.924783 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04712f50-7574-4123-8009-984532ac7a65" containerName="revoke-keys" Nov 03 03:20:23.925016 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:23.924844 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="4320be89-aa2d-4145-ad6f-07ab28aedb77" containerName="main" Nov 03 03:20:23.925016 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:23.924856 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="04712f50-7574-4123-8009-984532ac7a65" containerName="revoke-keys" Nov 03 03:20:23.929484 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:23.929462 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl" Nov 03 03:20:24.024687 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:24.024652 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl"] Nov 03 03:20:24.026538 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:24.026510 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-2232ad-openshift-default-dockercfg-mhk6d\"" Nov 03 03:20:24.039826 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:24.039797 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1437932f-1521-4f4d-a1d5-6d4be6fca6ce-istio-token\") pod \"e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl\" (UID: \"1437932f-1521-4f4d-a1d5-6d4be6fca6ce\") " pod="openshift-ingress/e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl" Nov 03 03:20:24.039985 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:24.039853 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1437932f-1521-4f4d-a1d5-6d4be6fca6ce-workload-socket\") pod \"e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl\" (UID: \"1437932f-1521-4f4d-a1d5-6d4be6fca6ce\") " pod="openshift-ingress/e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl" Nov 03 03:20:24.039985 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:24.039941 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1437932f-1521-4f4d-a1d5-6d4be6fca6ce-workload-certs\") pod \"e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl\" (UID: \"1437932f-1521-4f4d-a1d5-6d4be6fca6ce\") " pod="openshift-ingress/e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl" Nov 03 03:20:24.040117 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:24.040012 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1437932f-1521-4f4d-a1d5-6d4be6fca6ce-credential-socket\") pod \"e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl\" (UID: \"1437932f-1521-4f4d-a1d5-6d4be6fca6ce\") " pod="openshift-ingress/e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl" Nov 03 03:20:24.040117 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:24.040069 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtv5x\" (UniqueName: \"kubernetes.io/projected/1437932f-1521-4f4d-a1d5-6d4be6fca6ce-kube-api-access-qtv5x\") pod \"e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl\" (UID: \"1437932f-1521-4f4d-a1d5-6d4be6fca6ce\") " pod="openshift-ingress/e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl" Nov 03 03:20:24.040231 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:24.040116 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1437932f-1521-4f4d-a1d5-6d4be6fca6ce-istio-data\") pod \"e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl\" (UID: \"1437932f-1521-4f4d-a1d5-6d4be6fca6ce\") " pod="openshift-ingress/e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl" Nov 03 03:20:24.040231 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:24.040157 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1437932f-1521-4f4d-a1d5-6d4be6fca6ce-istio-envoy\") pod \"e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl\" (UID: \"1437932f-1521-4f4d-a1d5-6d4be6fca6ce\") " pod="openshift-ingress/e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl" Nov 03 03:20:24.040349 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:24.040259 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1437932f-1521-4f4d-a1d5-6d4be6fca6ce-istio-podinfo\") pod \"e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl\" (UID: \"1437932f-1521-4f4d-a1d5-6d4be6fca6ce\") " pod="openshift-ingress/e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl" Nov 03 03:20:24.040349 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:24.040296 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1437932f-1521-4f4d-a1d5-6d4be6fca6ce-istiod-ca-cert\") pod \"e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl\" (UID: \"1437932f-1521-4f4d-a1d5-6d4be6fca6ce\") " pod="openshift-ingress/e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl" Nov 03 03:20:24.141317 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:24.141281 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1437932f-1521-4f4d-a1d5-6d4be6fca6ce-workload-socket\") pod \"e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl\" (UID: \"1437932f-1521-4f4d-a1d5-6d4be6fca6ce\") " pod="openshift-ingress/e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl" Nov 03 03:20:24.141317 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:24.141319 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1437932f-1521-4f4d-a1d5-6d4be6fca6ce-workload-certs\") pod \"e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl\" (UID: \"1437932f-1521-4f4d-a1d5-6d4be6fca6ce\") " pod="openshift-ingress/e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl" Nov 03 03:20:24.141623 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:24.141337 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1437932f-1521-4f4d-a1d5-6d4be6fca6ce-credential-socket\") pod \"e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl\" (UID: \"1437932f-1521-4f4d-a1d5-6d4be6fca6ce\") " pod="openshift-ingress/e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl" Nov 03 03:20:24.141623 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:24.141394 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtv5x\" (UniqueName: \"kubernetes.io/projected/1437932f-1521-4f4d-a1d5-6d4be6fca6ce-kube-api-access-qtv5x\") pod \"e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl\" (UID: \"1437932f-1521-4f4d-a1d5-6d4be6fca6ce\") " pod="openshift-ingress/e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl" Nov 03 03:20:24.141623 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:24.141425 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1437932f-1521-4f4d-a1d5-6d4be6fca6ce-istio-data\") pod \"e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl\" (UID: \"1437932f-1521-4f4d-a1d5-6d4be6fca6ce\") " pod="openshift-ingress/e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl" Nov 03 03:20:24.141623 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:24.141444 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1437932f-1521-4f4d-a1d5-6d4be6fca6ce-istio-envoy\") pod \"e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl\" (UID: \"1437932f-1521-4f4d-a1d5-6d4be6fca6ce\") " pod="openshift-ingress/e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl" Nov 03 03:20:24.141623 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:24.141471 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1437932f-1521-4f4d-a1d5-6d4be6fca6ce-istio-podinfo\") pod \"e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl\" (UID: \"1437932f-1521-4f4d-a1d5-6d4be6fca6ce\") " pod="openshift-ingress/e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl" Nov 03 03:20:24.141623 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:24.141497 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1437932f-1521-4f4d-a1d5-6d4be6fca6ce-istiod-ca-cert\") pod \"e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl\" (UID: \"1437932f-1521-4f4d-a1d5-6d4be6fca6ce\") " pod="openshift-ingress/e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl" Nov 03 03:20:24.141623 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:24.141534 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1437932f-1521-4f4d-a1d5-6d4be6fca6ce-istio-token\") pod \"e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl\" (UID: \"1437932f-1521-4f4d-a1d5-6d4be6fca6ce\") " pod="openshift-ingress/e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl" Nov 03 03:20:24.142011 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:24.141813 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1437932f-1521-4f4d-a1d5-6d4be6fca6ce-workload-certs\") pod \"e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl\" (UID: \"1437932f-1521-4f4d-a1d5-6d4be6fca6ce\") " pod="openshift-ingress/e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl" Nov 03 03:20:24.142011 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:24.141977 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1437932f-1521-4f4d-a1d5-6d4be6fca6ce-workload-socket\") pod \"e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl\" (UID: \"1437932f-1521-4f4d-a1d5-6d4be6fca6ce\") " pod="openshift-ingress/e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl" Nov 03 03:20:24.142128 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:24.142083 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1437932f-1521-4f4d-a1d5-6d4be6fca6ce-istio-data\") pod \"e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl\" (UID: \"1437932f-1521-4f4d-a1d5-6d4be6fca6ce\") " pod="openshift-ingress/e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl" Nov 03 03:20:24.142197 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:24.142174 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1437932f-1521-4f4d-a1d5-6d4be6fca6ce-credential-socket\") pod \"e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl\" (UID: \"1437932f-1521-4f4d-a1d5-6d4be6fca6ce\") " pod="openshift-ingress/e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl" Nov 03 03:20:24.142325 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:24.142304 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1437932f-1521-4f4d-a1d5-6d4be6fca6ce-istiod-ca-cert\") pod \"e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl\" (UID: \"1437932f-1521-4f4d-a1d5-6d4be6fca6ce\") " pod="openshift-ingress/e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl" Nov 03 03:20:24.144181 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:24.144154 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1437932f-1521-4f4d-a1d5-6d4be6fca6ce-istio-envoy\") pod \"e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl\" (UID: \"1437932f-1521-4f4d-a1d5-6d4be6fca6ce\") " pod="openshift-ingress/e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl" Nov 03 03:20:24.144294 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:24.144256 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1437932f-1521-4f4d-a1d5-6d4be6fca6ce-istio-podinfo\") pod \"e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl\" (UID: \"1437932f-1521-4f4d-a1d5-6d4be6fca6ce\") " pod="openshift-ingress/e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl" Nov 03 03:20:24.195308 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:24.195248 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1437932f-1521-4f4d-a1d5-6d4be6fca6ce-istio-token\") pod \"e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl\" (UID: \"1437932f-1521-4f4d-a1d5-6d4be6fca6ce\") " pod="openshift-ingress/e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl" Nov 03 03:20:24.195463 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:24.195442 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtv5x\" (UniqueName: \"kubernetes.io/projected/1437932f-1521-4f4d-a1d5-6d4be6fca6ce-kube-api-access-qtv5x\") pod \"e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl\" (UID: \"1437932f-1521-4f4d-a1d5-6d4be6fca6ce\") " pod="openshift-ingress/e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl" Nov 03 03:20:24.244505 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:24.244478 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl" Nov 03 03:20:24.661831 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:24.661796 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl"] Nov 03 03:20:24.668298 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:20:24.668267 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1437932f_1521_4f4d_a1d5_6d4be6fca6ce.slice/crio-4d6a753493ff504f6e3d52dd6d346261e774f79856ea419046bff04533232ec5 WatchSource:0}: Error finding container 4d6a753493ff504f6e3d52dd6d346261e774f79856ea419046bff04533232ec5: Status 404 returned error can't find the container with id 4d6a753493ff504f6e3d52dd6d346261e774f79856ea419046bff04533232ec5 Nov 03 03:20:24.670348 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:24.670318 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 03:20:24.670442 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:24.670396 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 03:20:24.670442 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:24.670426 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 03:20:25.329887 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:25.329849 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl" event={"ID":"1437932f-1521-4f4d-a1d5-6d4be6fca6ce","Type":"ContainerStarted","Data":"64e4f4e4a137d9333213b5d16cfea7bbc4d1c206cc20196e861d7a318be8a54d"} Nov 03 03:20:25.330283 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:25.329909 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl" event={"ID":"1437932f-1521-4f4d-a1d5-6d4be6fca6ce","Type":"ContainerStarted","Data":"4d6a753493ff504f6e3d52dd6d346261e774f79856ea419046bff04533232ec5"} Nov 03 03:20:25.440584 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:25.440518 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl" podStartSLOduration=2.440499463 podStartE2EDuration="2.440499463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:20:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:20:25.438568549 +0000 UTC m=+5302.697766715" watchObservedRunningTime="2026-11-03 03:20:25.440499463 +0000 UTC m=+5302.699697619" Nov 03 03:20:26.244849 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:26.244817 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl" Nov 03 03:20:26.249571 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:26.249547 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl" Nov 03 03:20:26.334912 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:26.334879 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl" Nov 03 03:20:26.335804 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:26.335782 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl" Nov 03 03:20:28.171490 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:28.171451 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-k449x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 03:20:28.172087 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:28.172055 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k449x" podUID="0378fcbc-4027-4275-b274-2c188f3ff442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 03:20:32.068881 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:32.068844 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2232ad-659f4pmxc6"] Nov 03 03:20:32.072535 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:32.072512 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2232ad-659f4pmxc6" Nov 03 03:20:32.121214 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:32.121189 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-2232ad\"" Nov 03 03:20:32.150142 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:32.150101 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd8wb\" (UniqueName: \"kubernetes.io/projected/98211640-77e1-43d9-adf4-143ba87481a8-kube-api-access-rd8wb\") pod \"payload-pre-processing-e2e-worker-single-main-2232ad-659f4pmxc6\" (UID: \"98211640-77e1-43d9-adf4-143ba87481a8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2232ad-659f4pmxc6" Nov 03 03:20:32.150351 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:32.150177 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98211640-77e1-43d9-adf4-143ba87481a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-2232ad-659f4pmxc6\" (UID: \"98211640-77e1-43d9-adf4-143ba87481a8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2232ad-659f4pmxc6" Nov 03 03:20:32.169286 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:32.169240 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-4b649ca5-dockercfg-tqx6f\"" Nov 03 03:20:32.173370 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:32.171899 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2232ad-659f4pmxc6"] Nov 03 03:20:32.231795 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:32.231762 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2232ad-57b9b8b5flmq5p"] Nov 03 03:20:32.235446 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:32.235428 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2232ad-57b9b8b5flmq5p" Nov 03 03:20:32.251257 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:32.251231 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd8wb\" (UniqueName: \"kubernetes.io/projected/98211640-77e1-43d9-adf4-143ba87481a8-kube-api-access-rd8wb\") pod \"payload-pre-processing-e2e-worker-single-main-2232ad-659f4pmxc6\" (UID: \"98211640-77e1-43d9-adf4-143ba87481a8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2232ad-659f4pmxc6" Nov 03 03:20:32.251350 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:32.251295 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98211640-77e1-43d9-adf4-143ba87481a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-2232ad-659f4pmxc6\" (UID: \"98211640-77e1-43d9-adf4-143ba87481a8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2232ad-659f4pmxc6" Nov 03 03:20:32.252025 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:32.252005 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98211640-77e1-43d9-adf4-143ba87481a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-2232ad-659f4pmxc6\" (UID: \"98211640-77e1-43d9-adf4-143ba87481a8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2232ad-659f4pmxc6" Nov 03 03:20:32.289546 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:32.289517 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2232ad-57b9b8b5flmq5p"] Nov 03 03:20:32.307570 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:32.307549 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd8wb\" (UniqueName: \"kubernetes.io/projected/98211640-77e1-43d9-adf4-143ba87481a8-kube-api-access-rd8wb\") pod \"payload-pre-processing-e2e-worker-single-main-2232ad-659f4pmxc6\" (UID: \"98211640-77e1-43d9-adf4-143ba87481a8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2232ad-659f4pmxc6" Nov 03 03:20:32.352447 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:32.352386 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96f7t\" (UniqueName: \"kubernetes.io/projected/80b9fd99-012a-4b15-88d0-34075301288f-kube-api-access-96f7t\") pod \"payload-processing-e2e-worker-single-main-2232ad-57b9b8b5flmq5p\" (UID: \"80b9fd99-012a-4b15-88d0-34075301288f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2232ad-57b9b8b5flmq5p" Nov 03 03:20:32.352545 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:32.352509 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80b9fd99-012a-4b15-88d0-34075301288f-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-2232ad-57b9b8b5flmq5p\" (UID: \"80b9fd99-012a-4b15-88d0-34075301288f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2232ad-57b9b8b5flmq5p" Nov 03 03:20:32.385889 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:32.385868 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2232ad-659f4pmxc6" Nov 03 03:20:32.453800 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:32.453744 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80b9fd99-012a-4b15-88d0-34075301288f-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-2232ad-57b9b8b5flmq5p\" (UID: \"80b9fd99-012a-4b15-88d0-34075301288f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2232ad-57b9b8b5flmq5p" Nov 03 03:20:32.454666 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:32.454515 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96f7t\" (UniqueName: \"kubernetes.io/projected/80b9fd99-012a-4b15-88d0-34075301288f-kube-api-access-96f7t\") pod \"payload-processing-e2e-worker-single-main-2232ad-57b9b8b5flmq5p\" (UID: \"80b9fd99-012a-4b15-88d0-34075301288f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2232ad-57b9b8b5flmq5p" Nov 03 03:20:32.454666 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:32.454631 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80b9fd99-012a-4b15-88d0-34075301288f-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-2232ad-57b9b8b5flmq5p\" (UID: \"80b9fd99-012a-4b15-88d0-34075301288f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2232ad-57b9b8b5flmq5p" Nov 03 03:20:32.509764 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:32.509735 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96f7t\" (UniqueName: \"kubernetes.io/projected/80b9fd99-012a-4b15-88d0-34075301288f-kube-api-access-96f7t\") pod \"payload-processing-e2e-worker-single-main-2232ad-57b9b8b5flmq5p\" (UID: \"80b9fd99-012a-4b15-88d0-34075301288f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2232ad-57b9b8b5flmq5p" Nov 03 03:20:32.548565 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:32.548542 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2232ad-57b9b8b5flmq5p" Nov 03 03:20:32.754265 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:32.754195 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2232ad-659f4pmxc6"] Nov 03 03:20:32.759994 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:20:32.759967 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98211640_77e1_43d9_adf4_143ba87481a8.slice/crio-23163f0b5547c9a788c49b680053eaf0b6e4f6c50c6f076a84cfbf8df25dda06 WatchSource:0}: Error finding container 23163f0b5547c9a788c49b680053eaf0b6e4f6c50c6f076a84cfbf8df25dda06: Status 404 returned error can't find the container with id 23163f0b5547c9a788c49b680053eaf0b6e4f6c50c6f076a84cfbf8df25dda06 Nov 03 03:20:32.916270 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:32.916237 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2232ad-57b9b8b5flmq5p"] Nov 03 03:20:32.922480 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:20:32.922455 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80b9fd99_012a_4b15_88d0_34075301288f.slice/crio-6865a4eab84d87967f01b6215a82c051ef2082e634f316cce37ccbc5a3418709 WatchSource:0}: Error finding container 6865a4eab84d87967f01b6215a82c051ef2082e634f316cce37ccbc5a3418709: Status 404 returned error can't find the container with id 6865a4eab84d87967f01b6215a82c051ef2082e634f316cce37ccbc5a3418709 Nov 03 03:20:33.379618 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:33.379582 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2232ad-659f4pmxc6" event={"ID":"98211640-77e1-43d9-adf4-143ba87481a8","Type":"ContainerStarted","Data":"1732c1f3935df0b378593eb7771fd771c312728bb90b68a4f90618ded1366ef5"} Nov 03 03:20:33.380016 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:33.379645 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2232ad-659f4pmxc6" event={"ID":"98211640-77e1-43d9-adf4-143ba87481a8","Type":"ContainerStarted","Data":"23163f0b5547c9a788c49b680053eaf0b6e4f6c50c6f076a84cfbf8df25dda06"} Nov 03 03:20:33.380016 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:33.379672 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2232ad-659f4pmxc6" Nov 03 03:20:33.381149 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:33.381115 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2232ad-57b9b8b5flmq5p" event={"ID":"80b9fd99-012a-4b15-88d0-34075301288f","Type":"ContainerStarted","Data":"f0ffd4632340df0fc924806cb022069435b18a8520c54e6af446531cf3585ff1"} Nov 03 03:20:33.381283 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:33.381162 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2232ad-57b9b8b5flmq5p" event={"ID":"80b9fd99-012a-4b15-88d0-34075301288f","Type":"ContainerStarted","Data":"6865a4eab84d87967f01b6215a82c051ef2082e634f316cce37ccbc5a3418709"} Nov 03 03:20:33.381283 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:33.381200 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2232ad-57b9b8b5flmq5p" Nov 03 03:20:33.488247 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:33.488202 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2232ad-659f4pmxc6" podStartSLOduration=1.488192915 podStartE2EDuration="1.488192915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:20:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:20:33.487285184 +0000 UTC m=+5310.746483350" watchObservedRunningTime="2026-11-03 03:20:33.488192915 +0000 UTC m=+5310.747391068" Nov 03 03:20:33.595161 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:33.595119 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2232ad-57b9b8b5flmq5p" podStartSLOduration=1.595109244 podStartE2EDuration="1.595109244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:20:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:20:33.593827796 +0000 UTC m=+5310.853025952" watchObservedRunningTime="2026-11-03 03:20:33.595109244 +0000 UTC m=+5310.854307399" Nov 03 03:20:42.823201 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:42.823124 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:20:44.389142 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:44.389107 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2232ad-659f4pmxc6" Nov 03 03:20:44.390313 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:44.390286 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2232ad-57b9b8b5flmq5p" Nov 03 03:20:52.471636 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:52.471606 5605 generic.go:356] "Generic (PLEG): container finished" podID="1437932f-1521-4f4d-a1d5-6d4be6fca6ce" containerID="64e4f4e4a137d9333213b5d16cfea7bbc4d1c206cc20196e861d7a318be8a54d" exitCode=0 Nov 03 03:20:52.472017 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:52.471681 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl" event={"ID":"1437932f-1521-4f4d-a1d5-6d4be6fca6ce","Type":"ContainerDied","Data":"64e4f4e4a137d9333213b5d16cfea7bbc4d1c206cc20196e861d7a318be8a54d"} Nov 03 03:20:52.472071 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:52.472016 5605 scope.go:122] "RemoveContainer" containerID="64e4f4e4a137d9333213b5d16cfea7bbc4d1c206cc20196e861d7a318be8a54d" Nov 03 03:20:52.473193 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:52.473160 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 03:20:52.473274 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:52.473242 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 03:20:52.473312 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:52.473278 5605 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 03:20:53.483129 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:53.483093 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl" event={"ID":"1437932f-1521-4f4d-a1d5-6d4be6fca6ce","Type":"ContainerStarted","Data":"82bd16825388638234e36b7d9484b7dd58bf0be30d3560eda24c51dedcfde855"} Nov 03 03:20:53.912776 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:20:53.912734 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:20:54.245498 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:54.245409 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl" Nov 03 03:20:54.246699 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:54.246668 5605 patch_prober.go:29] interesting pod/e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.44:15021/healthz/ready\": dial tcp 10.134.1.44:15021: connect: connection refused" start-of-body= Nov 03 03:20:54.246834 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:54.246723 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl" podUID="1437932f-1521-4f4d-a1d5-6d4be6fca6ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.44:15021/healthz/ready\": dial tcp 10.134.1.44:15021: connect: connection refused" Nov 03 03:20:55.245804 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:55.245769 5605 patch_prober.go:29] interesting pod/e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.44:15021/healthz/ready\": dial tcp 10.134.1.44:15021: connect: connection refused" start-of-body= Nov 03 03:20:55.246555 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:55.246509 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl" podUID="1437932f-1521-4f4d-a1d5-6d4be6fca6ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.44:15021/healthz/ready\": dial tcp 10.134.1.44:15021: connect: connection refused" Nov 03 03:20:56.245211 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:56.245170 5605 patch_prober.go:29] interesting pod/e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.44:15021/healthz/ready\": dial tcp 10.134.1.44:15021: connect: connection refused" start-of-body= Nov 03 03:20:56.245458 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:56.245223 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl" podUID="1437932f-1521-4f4d-a1d5-6d4be6fca6ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.44:15021/healthz/ready\": dial tcp 10.134.1.44:15021: connect: connection refused" Nov 03 03:20:56.337386 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:56.336792 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl" Nov 03 03:20:56.505335 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:56.505251 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-2232ad/simulator-single-main-2232ad-kserve-d5f6df57d-5vd6s"] Nov 03 03:20:56.508881 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:56.508860 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-2232ad/simulator-single-main-2232ad-kserve-d5f6df57d-5vd6s" Nov 03 03:20:56.599185 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:56.599151 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c563a06d-f795-42e2-bb43-726fc4fd2a0b-tmp-dir\") pod \"simulator-single-main-2232ad-kserve-d5f6df57d-5vd6s\" (UID: \"c563a06d-f795-42e2-bb43-726fc4fd2a0b\") " pod="e2e-models-e2e-worker-single-main-2232ad/simulator-single-main-2232ad-kserve-d5f6df57d-5vd6s" Nov 03 03:20:56.599185 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:56.599187 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdg2p\" (UniqueName: \"kubernetes.io/projected/c563a06d-f795-42e2-bb43-726fc4fd2a0b-kube-api-access-vdg2p\") pod \"simulator-single-main-2232ad-kserve-d5f6df57d-5vd6s\" (UID: \"c563a06d-f795-42e2-bb43-726fc4fd2a0b\") " pod="e2e-models-e2e-worker-single-main-2232ad/simulator-single-main-2232ad-kserve-d5f6df57d-5vd6s" Nov 03 03:20:56.599438 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:56.599226 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c563a06d-f795-42e2-bb43-726fc4fd2a0b-model-cache\") pod \"simulator-single-main-2232ad-kserve-d5f6df57d-5vd6s\" (UID: \"c563a06d-f795-42e2-bb43-726fc4fd2a0b\") " pod="e2e-models-e2e-worker-single-main-2232ad/simulator-single-main-2232ad-kserve-d5f6df57d-5vd6s" Nov 03 03:20:56.599438 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:56.599270 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c563a06d-f795-42e2-bb43-726fc4fd2a0b-tls-certs\") pod \"simulator-single-main-2232ad-kserve-d5f6df57d-5vd6s\" (UID: \"c563a06d-f795-42e2-bb43-726fc4fd2a0b\") " pod="e2e-models-e2e-worker-single-main-2232ad/simulator-single-main-2232ad-kserve-d5f6df57d-5vd6s" Nov 03 03:20:56.599438 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:56.599285 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c563a06d-f795-42e2-bb43-726fc4fd2a0b-home\") pod \"simulator-single-main-2232ad-kserve-d5f6df57d-5vd6s\" (UID: \"c563a06d-f795-42e2-bb43-726fc4fd2a0b\") " pod="e2e-models-e2e-worker-single-main-2232ad/simulator-single-main-2232ad-kserve-d5f6df57d-5vd6s" Nov 03 03:20:56.599438 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:56.599332 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c563a06d-f795-42e2-bb43-726fc4fd2a0b-dshm\") pod \"simulator-single-main-2232ad-kserve-d5f6df57d-5vd6s\" (UID: \"c563a06d-f795-42e2-bb43-726fc4fd2a0b\") " pod="e2e-models-e2e-worker-single-main-2232ad/simulator-single-main-2232ad-kserve-d5f6df57d-5vd6s" Nov 03 03:20:56.628601 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:56.628562 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-2232ad\"/\"openshift-service-ca.crt\"" Nov 03 03:20:56.628791 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:56.628695 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-2232ad\"/\"simulator-single-main-2232ad-kserve-self-signed-certs\"" Nov 03 03:20:56.628791 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:56.628763 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-2232ad\"/\"kube-root-ca.crt\"" Nov 03 03:20:56.629371 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:56.629342 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-2232ad\"/\"default-dockercfg-dpjlh\"" Nov 03 03:20:56.630650 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:56.630628 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-2232ad/simulator-single-main-2232ad-kserve-d5f6df57d-5vd6s"] Nov 03 03:20:56.700258 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:56.700206 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c563a06d-f795-42e2-bb43-726fc4fd2a0b-tmp-dir\") pod \"simulator-single-main-2232ad-kserve-d5f6df57d-5vd6s\" (UID: \"c563a06d-f795-42e2-bb43-726fc4fd2a0b\") " pod="e2e-models-e2e-worker-single-main-2232ad/simulator-single-main-2232ad-kserve-d5f6df57d-5vd6s" Nov 03 03:20:56.700258 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:56.700261 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdg2p\" (UniqueName: \"kubernetes.io/projected/c563a06d-f795-42e2-bb43-726fc4fd2a0b-kube-api-access-vdg2p\") pod \"simulator-single-main-2232ad-kserve-d5f6df57d-5vd6s\" (UID: \"c563a06d-f795-42e2-bb43-726fc4fd2a0b\") " pod="e2e-models-e2e-worker-single-main-2232ad/simulator-single-main-2232ad-kserve-d5f6df57d-5vd6s" Nov 03 03:20:56.700537 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:56.700295 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c563a06d-f795-42e2-bb43-726fc4fd2a0b-model-cache\") pod \"simulator-single-main-2232ad-kserve-d5f6df57d-5vd6s\" (UID: \"c563a06d-f795-42e2-bb43-726fc4fd2a0b\") " pod="e2e-models-e2e-worker-single-main-2232ad/simulator-single-main-2232ad-kserve-d5f6df57d-5vd6s" Nov 03 03:20:56.700537 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:56.700350 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c563a06d-f795-42e2-bb43-726fc4fd2a0b-tls-certs\") pod \"simulator-single-main-2232ad-kserve-d5f6df57d-5vd6s\" (UID: \"c563a06d-f795-42e2-bb43-726fc4fd2a0b\") " pod="e2e-models-e2e-worker-single-main-2232ad/simulator-single-main-2232ad-kserve-d5f6df57d-5vd6s" Nov 03 03:20:56.700537 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:56.700401 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c563a06d-f795-42e2-bb43-726fc4fd2a0b-home\") pod \"simulator-single-main-2232ad-kserve-d5f6df57d-5vd6s\" (UID: \"c563a06d-f795-42e2-bb43-726fc4fd2a0b\") " pod="e2e-models-e2e-worker-single-main-2232ad/simulator-single-main-2232ad-kserve-d5f6df57d-5vd6s" Nov 03 03:20:56.700537 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:56.700452 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c563a06d-f795-42e2-bb43-726fc4fd2a0b-dshm\") pod \"simulator-single-main-2232ad-kserve-d5f6df57d-5vd6s\" (UID: \"c563a06d-f795-42e2-bb43-726fc4fd2a0b\") " pod="e2e-models-e2e-worker-single-main-2232ad/simulator-single-main-2232ad-kserve-d5f6df57d-5vd6s" Nov 03 03:20:56.700716 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:56.700618 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c563a06d-f795-42e2-bb43-726fc4fd2a0b-tmp-dir\") pod \"simulator-single-main-2232ad-kserve-d5f6df57d-5vd6s\" (UID: \"c563a06d-f795-42e2-bb43-726fc4fd2a0b\") " pod="e2e-models-e2e-worker-single-main-2232ad/simulator-single-main-2232ad-kserve-d5f6df57d-5vd6s" Nov 03 03:20:56.700817 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:56.700801 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c563a06d-f795-42e2-bb43-726fc4fd2a0b-model-cache\") pod \"simulator-single-main-2232ad-kserve-d5f6df57d-5vd6s\" (UID: \"c563a06d-f795-42e2-bb43-726fc4fd2a0b\") " pod="e2e-models-e2e-worker-single-main-2232ad/simulator-single-main-2232ad-kserve-d5f6df57d-5vd6s" Nov 03 03:20:56.701154 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:56.701134 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c563a06d-f795-42e2-bb43-726fc4fd2a0b-home\") pod \"simulator-single-main-2232ad-kserve-d5f6df57d-5vd6s\" (UID: \"c563a06d-f795-42e2-bb43-726fc4fd2a0b\") " pod="e2e-models-e2e-worker-single-main-2232ad/simulator-single-main-2232ad-kserve-d5f6df57d-5vd6s" Nov 03 03:20:56.702694 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:56.702637 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c563a06d-f795-42e2-bb43-726fc4fd2a0b-dshm\") pod \"simulator-single-main-2232ad-kserve-d5f6df57d-5vd6s\" (UID: \"c563a06d-f795-42e2-bb43-726fc4fd2a0b\") " pod="e2e-models-e2e-worker-single-main-2232ad/simulator-single-main-2232ad-kserve-d5f6df57d-5vd6s" Nov 03 03:20:56.703068 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:56.703050 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c563a06d-f795-42e2-bb43-726fc4fd2a0b-tls-certs\") pod \"simulator-single-main-2232ad-kserve-d5f6df57d-5vd6s\" (UID: \"c563a06d-f795-42e2-bb43-726fc4fd2a0b\") " pod="e2e-models-e2e-worker-single-main-2232ad/simulator-single-main-2232ad-kserve-d5f6df57d-5vd6s" Nov 03 03:20:56.759223 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:56.759091 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdg2p\" (UniqueName: \"kubernetes.io/projected/c563a06d-f795-42e2-bb43-726fc4fd2a0b-kube-api-access-vdg2p\") pod \"simulator-single-main-2232ad-kserve-d5f6df57d-5vd6s\" (UID: \"c563a06d-f795-42e2-bb43-726fc4fd2a0b\") " pod="e2e-models-e2e-worker-single-main-2232ad/simulator-single-main-2232ad-kserve-d5f6df57d-5vd6s" Nov 03 03:20:56.820552 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:56.820510 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-2232ad/simulator-single-main-2232ad-kserve-d5f6df57d-5vd6s" Nov 03 03:20:57.222886 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:57.222845 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-2232ad/simulator-single-main-2232ad-kserve-d5f6df57d-5vd6s"] Nov 03 03:20:57.229138 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:20:57.229104 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc563a06d_f795_42e2_bb43_726fc4fd2a0b.slice/crio-ea5d80dcf66599ab14e9cc29295f33d4f47e4fc69d46b256ceec02678b02719e WatchSource:0}: Error finding container ea5d80dcf66599ab14e9cc29295f33d4f47e4fc69d46b256ceec02678b02719e: Status 404 returned error can't find the container with id ea5d80dcf66599ab14e9cc29295f33d4f47e4fc69d46b256ceec02678b02719e Nov 03 03:20:57.245306 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:57.245280 5605 patch_prober.go:29] interesting pod/e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.44:15021/healthz/ready\": dial tcp 10.134.1.44:15021: connect: connection refused" start-of-body= Nov 03 03:20:57.245411 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:57.245344 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl" podUID="1437932f-1521-4f4d-a1d5-6d4be6fca6ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.44:15021/healthz/ready\": dial tcp 10.134.1.44:15021: connect: connection refused" Nov 03 03:20:57.506236 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:57.506192 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-2232ad/simulator-single-main-2232ad-kserve-d5f6df57d-5vd6s" event={"ID":"c563a06d-f795-42e2-bb43-726fc4fd2a0b","Type":"ContainerStarted","Data":"cf928545923acff67f0f76a9856f4e178b5618847494415348f8de34fd512c8c"} Nov 03 03:20:57.506236 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:57.506251 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-2232ad/simulator-single-main-2232ad-kserve-d5f6df57d-5vd6s" event={"ID":"c563a06d-f795-42e2-bb43-726fc4fd2a0b","Type":"ContainerStarted","Data":"ea5d80dcf66599ab14e9cc29295f33d4f47e4fc69d46b256ceec02678b02719e"} Nov 03 03:20:57.627972 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:57.627916 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-2232ad/simulator-single-main-2232ad-kserve-d5f6df57d-5vd6s" podStartSLOduration=1.627887124 podStartE2EDuration="1.627887124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:20:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:20:57.627738498 +0000 UTC m=+5334.886936652" watchObservedRunningTime="2026-11-03 03:20:57.627887124 +0000 UTC m=+5334.887085260" Nov 03 03:20:57.821172 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:57.821135 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-2232ad/simulator-single-main-2232ad-kserve-d5f6df57d-5vd6s" Nov 03 03:20:58.244975 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:58.244933 5605 patch_prober.go:29] interesting pod/e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.44:15021/healthz/ready\": dial tcp 10.134.1.44:15021: connect: connection refused" start-of-body= Nov 03 03:20:58.245170 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:58.245003 5605 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl" podUID="1437932f-1521-4f4d-a1d5-6d4be6fca6ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.44:15021/healthz/ready\": dial tcp 10.134.1.44:15021: connect: connection refused" Nov 03 03:20:59.249633 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:59.249601 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl" Nov 03 03:20:59.250588 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:20:59.250567 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl" Nov 03 03:21:00.956488 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:21:00.956430 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:21:05.688169 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:21:05.688132 5605 scope.go:122] "RemoveContainer" containerID="c376e2e6dbebc0888b539f008362a003d55c5e433fd5ed9390ced812014a9a73" Nov 03 03:21:05.696191 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:21:05.696168 5605 scope.go:122] "RemoveContainer" containerID="89c8710d210d554210b63762285e262e0c59494c7b556b68b3e1031ebad8d3eb" Nov 03 03:21:06.820784 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:21:06.820748 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-2232ad/simulator-single-main-2232ad-kserve-d5f6df57d-5vd6s" Nov 03 03:21:06.833279 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:21:06.833251 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-2232ad/simulator-single-main-2232ad-kserve-d5f6df57d-5vd6s" Nov 03 03:21:07.567946 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:21:07.567914 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-2232ad/simulator-single-main-2232ad-kserve-d5f6df57d-5vd6s" Nov 03 03:21:23.913285 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:21:23.913240 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:21:27.031481 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:21:27.031440 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:21:48.042330 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:21:48.042295 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kz87q"] Nov 03 03:21:48.088941 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:21:48.088914 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kz87q"] Nov 03 03:21:48.848575 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:21:48.848540 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff914592-f476-4e9d-b25c-3d34963ab378" path="/var/lib/kubelet/pods/ff914592-f476-4e9d-b25c-3d34963ab378/volumes" Nov 03 03:21:53.911445 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:21:53.911405 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:21:55.162709 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:21:55.162677 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:21:58.377793 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:21:58.377753 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-2232ad/simulator-single-main-2232ad-kserve-d5f6df57d-5vd6s"] Nov 03 03:22:00.471929 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:00.471894 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2232ad-57b9b8b5flmq5p"] Nov 03 03:22:00.576544 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:00.576507 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2232ad-659f4pmxc6"] Nov 03 03:22:05.472469 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:05.472391 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2232ad-57b9b8b5flmq5p" podUID="80b9fd99-012a-4b15-88d0-34075301288f" containerName="payload-processing" containerID="cri-o://f0ffd4632340df0fc924806cb022069435b18a8520c54e6af446531cf3585ff1" gracePeriod=25 Nov 03 03:22:05.576774 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:05.576732 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2232ad-659f4pmxc6" podUID="98211640-77e1-43d9-adf4-143ba87481a8" containerName="payload-pre-processing" containerID="cri-o://1732c1f3935df0b378593eb7771fd771c312728bb90b68a4f90618ded1366ef5" gracePeriod=25 Nov 03 03:22:05.722169 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:05.722131 5605 scope.go:122] "RemoveContainer" containerID="1a4ba13e508e0dbec61425dbbf7ccbd920a0ad12cb8df09a1b3bed2be6c59308" Nov 03 03:22:07.872910 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:07.872874 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:22:13.397190 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:13.397097 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-2232ad/simulator-single-main-2232ad-kserve-d5f6df57d-5vd6s" podUID="c563a06d-f795-42e2-bb43-726fc4fd2a0b" containerName="main" containerID="cri-o://cf928545923acff67f0f76a9856f4e178b5618847494415348f8de34fd512c8c" gracePeriod=45 Nov 03 03:22:13.557685 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:13.557641 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-2232ad/simulator-single-main-2232ad-kserve-d5f6df57d-5vd6s" podUID="c563a06d-f795-42e2-bb43-726fc4fd2a0b" containerName="main" probeResult="failure" output="Get \"https://10.134.1.47:8000/ready\": dial tcp 10.134.1.47:8000: connect: connection refused" Nov 03 03:22:13.642946 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:13.642920 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-2232ad/simulator-single-main-2232ad-kserve-d5f6df57d-5vd6s" Nov 03 03:22:13.691682 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:13.691631 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c563a06d-f795-42e2-bb43-726fc4fd2a0b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c563a06d-f795-42e2-bb43-726fc4fd2a0b-tmp-dir\") pod \"c563a06d-f795-42e2-bb43-726fc4fd2a0b\" (UID: \"c563a06d-f795-42e2-bb43-726fc4fd2a0b\") " Nov 03 03:22:13.691777 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:13.691736 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c563a06d-f795-42e2-bb43-726fc4fd2a0b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c563a06d-f795-42e2-bb43-726fc4fd2a0b-dshm\") pod \"c563a06d-f795-42e2-bb43-726fc4fd2a0b\" (UID: \"c563a06d-f795-42e2-bb43-726fc4fd2a0b\") " Nov 03 03:22:13.691777 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:13.691760 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c563a06d-f795-42e2-bb43-726fc4fd2a0b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c563a06d-f795-42e2-bb43-726fc4fd2a0b-model-cache\") pod \"c563a06d-f795-42e2-bb43-726fc4fd2a0b\" (UID: \"c563a06d-f795-42e2-bb43-726fc4fd2a0b\") " Nov 03 03:22:13.691880 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:13.691786 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c563a06d-f795-42e2-bb43-726fc4fd2a0b-tls-certs\" (UniqueName: \"kubernetes.io/secret/c563a06d-f795-42e2-bb43-726fc4fd2a0b-tls-certs\") pod \"c563a06d-f795-42e2-bb43-726fc4fd2a0b\" (UID: \"c563a06d-f795-42e2-bb43-726fc4fd2a0b\") " Nov 03 03:22:13.691880 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:13.691845 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c563a06d-f795-42e2-bb43-726fc4fd2a0b-home\" (UniqueName: \"kubernetes.io/empty-dir/c563a06d-f795-42e2-bb43-726fc4fd2a0b-home\") pod \"c563a06d-f795-42e2-bb43-726fc4fd2a0b\" (UID: \"c563a06d-f795-42e2-bb43-726fc4fd2a0b\") " Nov 03 03:22:13.691880 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:13.691849 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c563a06d-f795-42e2-bb43-726fc4fd2a0b-tmp-dir" pod "c563a06d-f795-42e2-bb43-726fc4fd2a0b" (UID: "c563a06d-f795-42e2-bb43-726fc4fd2a0b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:22:13.692053 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:13.691879 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c563a06d-f795-42e2-bb43-726fc4fd2a0b-kube-api-access-vdg2p\" (UniqueName: \"kubernetes.io/projected/c563a06d-f795-42e2-bb43-726fc4fd2a0b-kube-api-access-vdg2p\") pod \"c563a06d-f795-42e2-bb43-726fc4fd2a0b\" (UID: \"c563a06d-f795-42e2-bb43-726fc4fd2a0b\") " Nov 03 03:22:13.692115 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:13.692052 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c563a06d-f795-42e2-bb43-726fc4fd2a0b-home" pod "c563a06d-f795-42e2-bb43-726fc4fd2a0b" (UID: "c563a06d-f795-42e2-bb43-726fc4fd2a0b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:22:13.692115 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:13.692067 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c563a06d-f795-42e2-bb43-726fc4fd2a0b-model-cache" pod "c563a06d-f795-42e2-bb43-726fc4fd2a0b" (UID: "c563a06d-f795-42e2-bb43-726fc4fd2a0b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:22:13.692221 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:13.692190 5605 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c563a06d-f795-42e2-bb43-726fc4fd2a0b-model-cache\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:22:13.692221 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:13.692210 5605 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c563a06d-f795-42e2-bb43-726fc4fd2a0b-home\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:22:13.692292 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:13.692225 5605 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c563a06d-f795-42e2-bb43-726fc4fd2a0b-tmp-dir\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:22:13.693934 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:13.693914 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c563a06d-f795-42e2-bb43-726fc4fd2a0b-tls-certs" pod "c563a06d-f795-42e2-bb43-726fc4fd2a0b" (UID: "c563a06d-f795-42e2-bb43-726fc4fd2a0b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:22:13.694006 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:13.693936 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c563a06d-f795-42e2-bb43-726fc4fd2a0b-dshm" pod "c563a06d-f795-42e2-bb43-726fc4fd2a0b" (UID: "c563a06d-f795-42e2-bb43-726fc4fd2a0b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:22:13.694056 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:13.694040 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c563a06d-f795-42e2-bb43-726fc4fd2a0b-kube-api-access-vdg2p" pod "c563a06d-f795-42e2-bb43-726fc4fd2a0b" (UID: "c563a06d-f795-42e2-bb43-726fc4fd2a0b"). InnerVolumeSpecName "kube-api-access-vdg2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:22:13.792784 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:13.792756 5605 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c563a06d-f795-42e2-bb43-726fc4fd2a0b-dshm\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:22:13.792784 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:13.792781 5605 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c563a06d-f795-42e2-bb43-726fc4fd2a0b-tls-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:22:13.792952 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:13.792791 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdg2p\" (UniqueName: \"kubernetes.io/projected/c563a06d-f795-42e2-bb43-726fc4fd2a0b-kube-api-access-vdg2p\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:22:13.854692 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:13.854655 5605 generic.go:356] "Generic (PLEG): container finished" podID="c563a06d-f795-42e2-bb43-726fc4fd2a0b" containerID="cf928545923acff67f0f76a9856f4e178b5618847494415348f8de34fd512c8c" exitCode=0 Nov 03 03:22:13.854845 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:13.854705 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-2232ad/simulator-single-main-2232ad-kserve-d5f6df57d-5vd6s" Nov 03 03:22:13.854845 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:13.854743 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-2232ad/simulator-single-main-2232ad-kserve-d5f6df57d-5vd6s" event={"ID":"c563a06d-f795-42e2-bb43-726fc4fd2a0b","Type":"ContainerDied","Data":"cf928545923acff67f0f76a9856f4e178b5618847494415348f8de34fd512c8c"} Nov 03 03:22:13.854845 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:13.854794 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-2232ad/simulator-single-main-2232ad-kserve-d5f6df57d-5vd6s" event={"ID":"c563a06d-f795-42e2-bb43-726fc4fd2a0b","Type":"ContainerDied","Data":"ea5d80dcf66599ab14e9cc29295f33d4f47e4fc69d46b256ceec02678b02719e"} Nov 03 03:22:13.854845 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:13.854817 5605 scope.go:122] "RemoveContainer" containerID="cf928545923acff67f0f76a9856f4e178b5618847494415348f8de34fd512c8c" Nov 03 03:22:13.864732 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:13.864715 5605 scope.go:122] "RemoveContainer" containerID="cf928545923acff67f0f76a9856f4e178b5618847494415348f8de34fd512c8c" Nov 03 03:22:13.864999 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:22:13.864978 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf928545923acff67f0f76a9856f4e178b5618847494415348f8de34fd512c8c\": container with ID starting with cf928545923acff67f0f76a9856f4e178b5618847494415348f8de34fd512c8c not found: ID does not exist" containerID="cf928545923acff67f0f76a9856f4e178b5618847494415348f8de34fd512c8c" Nov 03 03:22:13.865061 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:13.865011 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf928545923acff67f0f76a9856f4e178b5618847494415348f8de34fd512c8c"} err="failed to get container status \"cf928545923acff67f0f76a9856f4e178b5618847494415348f8de34fd512c8c\": rpc error: code = NotFound desc = could not find container \"cf928545923acff67f0f76a9856f4e178b5618847494415348f8de34fd512c8c\": container with ID starting with cf928545923acff67f0f76a9856f4e178b5618847494415348f8de34fd512c8c not found: ID does not exist" Nov 03 03:22:14.008051 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:14.007981 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-2232ad/simulator-single-main-2232ad-kserve-d5f6df57d-5vd6s"] Nov 03 03:22:14.010790 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:14.010768 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-2232ad/simulator-single-main-2232ad-kserve-d5f6df57d-5vd6s"] Nov 03 03:22:14.387006 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:14.386976 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2232ad-57b9b8b5flmq5p" podUID="80b9fd99-012a-4b15-88d0-34075301288f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:22:14.387167 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:14.387092 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2232ad-659f4pmxc6" podUID="98211640-77e1-43d9-adf4-143ba87481a8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:22:14.848890 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:14.848862 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c563a06d-f795-42e2-bb43-726fc4fd2a0b" path="/var/lib/kubelet/pods/c563a06d-f795-42e2-bb43-726fc4fd2a0b/volumes" Nov 03 03:22:23.911153 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:22:23.911113 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:22:24.386560 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:24.386517 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2232ad-659f4pmxc6" podUID="98211640-77e1-43d9-adf4-143ba87481a8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:22:24.386714 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:24.386517 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2232ad-57b9b8b5flmq5p" podUID="80b9fd99-012a-4b15-88d0-34075301288f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:22:25.713411 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:25.713380 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2232ad-57b9b8b5flmq5p" Nov 03 03:22:25.793847 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:25.793812 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80b9fd99-012a-4b15-88d0-34075301288f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80b9fd99-012a-4b15-88d0-34075301288f-plugins-config-volume\") pod \"80b9fd99-012a-4b15-88d0-34075301288f\" (UID: \"80b9fd99-012a-4b15-88d0-34075301288f\") " Nov 03 03:22:25.794021 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:25.793894 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80b9fd99-012a-4b15-88d0-34075301288f-kube-api-access-96f7t\" (UniqueName: \"kubernetes.io/projected/80b9fd99-012a-4b15-88d0-34075301288f-kube-api-access-96f7t\") pod \"80b9fd99-012a-4b15-88d0-34075301288f\" (UID: \"80b9fd99-012a-4b15-88d0-34075301288f\") " Nov 03 03:22:25.794202 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:25.794182 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80b9fd99-012a-4b15-88d0-34075301288f-plugins-config-volume" pod "80b9fd99-012a-4b15-88d0-34075301288f" (UID: "80b9fd99-012a-4b15-88d0-34075301288f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:22:25.795983 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:25.795954 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80b9fd99-012a-4b15-88d0-34075301288f-kube-api-access-96f7t" pod "80b9fd99-012a-4b15-88d0-34075301288f" (UID: "80b9fd99-012a-4b15-88d0-34075301288f"). InnerVolumeSpecName "kube-api-access-96f7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:22:25.811570 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:25.811549 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2232ad-659f4pmxc6" Nov 03 03:22:25.895201 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:25.895165 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98211640-77e1-43d9-adf4-143ba87481a8-kube-api-access-rd8wb\" (UniqueName: \"kubernetes.io/projected/98211640-77e1-43d9-adf4-143ba87481a8-kube-api-access-rd8wb\") pod \"98211640-77e1-43d9-adf4-143ba87481a8\" (UID: \"98211640-77e1-43d9-adf4-143ba87481a8\") " Nov 03 03:22:25.895408 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:25.895213 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98211640-77e1-43d9-adf4-143ba87481a8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98211640-77e1-43d9-adf4-143ba87481a8-plugins-config-volume\") pod \"98211640-77e1-43d9-adf4-143ba87481a8\" (UID: \"98211640-77e1-43d9-adf4-143ba87481a8\") " Nov 03 03:22:25.895514 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:25.895496 5605 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80b9fd99-012a-4b15-88d0-34075301288f-plugins-config-volume\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:22:25.895590 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:25.895515 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96f7t\" (UniqueName: \"kubernetes.io/projected/80b9fd99-012a-4b15-88d0-34075301288f-kube-api-access-96f7t\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:22:25.895674 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:25.895648 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98211640-77e1-43d9-adf4-143ba87481a8-plugins-config-volume" pod "98211640-77e1-43d9-adf4-143ba87481a8" (UID: "98211640-77e1-43d9-adf4-143ba87481a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:22:25.897313 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:25.897295 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98211640-77e1-43d9-adf4-143ba87481a8-kube-api-access-rd8wb" pod "98211640-77e1-43d9-adf4-143ba87481a8" (UID: "98211640-77e1-43d9-adf4-143ba87481a8"). InnerVolumeSpecName "kube-api-access-rd8wb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:22:25.905217 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:25.905193 5605 generic.go:356] "Generic (PLEG): container finished" podID="98211640-77e1-43d9-adf4-143ba87481a8" containerID="1732c1f3935df0b378593eb7771fd771c312728bb90b68a4f90618ded1366ef5" exitCode=0 Nov 03 03:22:25.905332 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:25.905234 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2232ad-659f4pmxc6" Nov 03 03:22:25.905332 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:25.905268 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2232ad-659f4pmxc6" event={"ID":"98211640-77e1-43d9-adf4-143ba87481a8","Type":"ContainerDied","Data":"1732c1f3935df0b378593eb7771fd771c312728bb90b68a4f90618ded1366ef5"} Nov 03 03:22:25.905332 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:25.905319 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2232ad-659f4pmxc6" event={"ID":"98211640-77e1-43d9-adf4-143ba87481a8","Type":"ContainerDied","Data":"23163f0b5547c9a788c49b680053eaf0b6e4f6c50c6f076a84cfbf8df25dda06"} Nov 03 03:22:25.905495 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:25.905345 5605 scope.go:122] "RemoveContainer" containerID="1732c1f3935df0b378593eb7771fd771c312728bb90b68a4f90618ded1366ef5" Nov 03 03:22:25.906588 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:25.906568 5605 generic.go:356] "Generic (PLEG): container finished" podID="80b9fd99-012a-4b15-88d0-34075301288f" containerID="f0ffd4632340df0fc924806cb022069435b18a8520c54e6af446531cf3585ff1" exitCode=0 Nov 03 03:22:25.906692 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:25.906601 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2232ad-57b9b8b5flmq5p" event={"ID":"80b9fd99-012a-4b15-88d0-34075301288f","Type":"ContainerDied","Data":"f0ffd4632340df0fc924806cb022069435b18a8520c54e6af446531cf3585ff1"} Nov 03 03:22:25.906692 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:25.906615 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2232ad-57b9b8b5flmq5p" Nov 03 03:22:25.906692 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:25.906649 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2232ad-57b9b8b5flmq5p" event={"ID":"80b9fd99-012a-4b15-88d0-34075301288f","Type":"ContainerDied","Data":"6865a4eab84d87967f01b6215a82c051ef2082e634f316cce37ccbc5a3418709"} Nov 03 03:22:25.914596 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:25.914578 5605 scope.go:122] "RemoveContainer" containerID="1732c1f3935df0b378593eb7771fd771c312728bb90b68a4f90618ded1366ef5" Nov 03 03:22:25.914913 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:22:25.914891 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1732c1f3935df0b378593eb7771fd771c312728bb90b68a4f90618ded1366ef5\": container with ID starting with 1732c1f3935df0b378593eb7771fd771c312728bb90b68a4f90618ded1366ef5 not found: ID does not exist" containerID="1732c1f3935df0b378593eb7771fd771c312728bb90b68a4f90618ded1366ef5" Nov 03 03:22:25.914980 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:25.914920 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1732c1f3935df0b378593eb7771fd771c312728bb90b68a4f90618ded1366ef5"} err="failed to get container status \"1732c1f3935df0b378593eb7771fd771c312728bb90b68a4f90618ded1366ef5\": rpc error: code = NotFound desc = could not find container \"1732c1f3935df0b378593eb7771fd771c312728bb90b68a4f90618ded1366ef5\": container with ID starting with 1732c1f3935df0b378593eb7771fd771c312728bb90b68a4f90618ded1366ef5 not found: ID does not exist" Nov 03 03:22:25.914980 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:25.914937 5605 scope.go:122] "RemoveContainer" containerID="f0ffd4632340df0fc924806cb022069435b18a8520c54e6af446531cf3585ff1" Nov 03 03:22:25.922505 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:25.922487 5605 scope.go:122] "RemoveContainer" containerID="f0ffd4632340df0fc924806cb022069435b18a8520c54e6af446531cf3585ff1" Nov 03 03:22:25.922746 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:22:25.922727 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0ffd4632340df0fc924806cb022069435b18a8520c54e6af446531cf3585ff1\": container with ID starting with f0ffd4632340df0fc924806cb022069435b18a8520c54e6af446531cf3585ff1 not found: ID does not exist" containerID="f0ffd4632340df0fc924806cb022069435b18a8520c54e6af446531cf3585ff1" Nov 03 03:22:25.922793 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:25.922758 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0ffd4632340df0fc924806cb022069435b18a8520c54e6af446531cf3585ff1"} err="failed to get container status \"f0ffd4632340df0fc924806cb022069435b18a8520c54e6af446531cf3585ff1\": rpc error: code = NotFound desc = could not find container \"f0ffd4632340df0fc924806cb022069435b18a8520c54e6af446531cf3585ff1\": container with ID starting with f0ffd4632340df0fc924806cb022069435b18a8520c54e6af446531cf3585ff1 not found: ID does not exist" Nov 03 03:22:25.996448 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:25.996349 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rd8wb\" (UniqueName: \"kubernetes.io/projected/98211640-77e1-43d9-adf4-143ba87481a8-kube-api-access-rd8wb\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:22:25.996448 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:25.996394 5605 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98211640-77e1-43d9-adf4-143ba87481a8-plugins-config-volume\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:22:26.046704 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:26.046671 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2232ad-57b9b8b5flmq5p"] Nov 03 03:22:26.050593 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:26.050561 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2232ad-57b9b8b5flmq5p"] Nov 03 03:22:26.152775 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:26.152745 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2232ad-659f4pmxc6"] Nov 03 03:22:26.156921 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:26.156894 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2232ad-659f4pmxc6"] Nov 03 03:22:26.850016 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:26.849975 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80b9fd99-012a-4b15-88d0-34075301288f" path="/var/lib/kubelet/pods/80b9fd99-012a-4b15-88d0-34075301288f/volumes" Nov 03 03:22:26.850535 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:26.850506 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98211640-77e1-43d9-adf4-143ba87481a8" path="/var/lib/kubelet/pods/98211640-77e1-43d9-adf4-143ba87481a8/volumes" Nov 03 03:22:36.046108 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:36.046071 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b7mjf"] Nov 03 03:22:36.092670 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:36.092620 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b7mjf"] Nov 03 03:22:36.850437 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:36.850396 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc6733d7-9260-4970-b6fd-9103df988649" path="/var/lib/kubelet/pods/cc6733d7-9260-4970-b6fd-9103df988649/volumes" Nov 03 03:22:45.650050 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:45.650018 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:22:48.629405 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:48.629376 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2232ad-g4stc"] Nov 03 03:22:48.629793 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:48.629696 5605 cpu_manager.go:414] "containerMap: removing container" podUID="80b9fd99-012a-4b15-88d0-34075301288f" containerName="payload-processing" Nov 03 03:22:48.629793 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:48.629709 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80b9fd99-012a-4b15-88d0-34075301288f" containerName="payload-processing" Nov 03 03:22:48.629793 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:48.629745 5605 cpu_manager.go:414] "containerMap: removing container" podUID="98211640-77e1-43d9-adf4-143ba87481a8" containerName="payload-pre-processing" Nov 03 03:22:48.629793 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:48.629750 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98211640-77e1-43d9-adf4-143ba87481a8" containerName="payload-pre-processing" Nov 03 03:22:48.629793 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:48.629759 5605 cpu_manager.go:414] "containerMap: removing container" podUID="c563a06d-f795-42e2-bb43-726fc4fd2a0b" containerName="main" Nov 03 03:22:48.629793 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:48.629764 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c563a06d-f795-42e2-bb43-726fc4fd2a0b" containerName="main" Nov 03 03:22:48.630001 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:48.629810 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="98211640-77e1-43d9-adf4-143ba87481a8" containerName="payload-pre-processing" Nov 03 03:22:48.630001 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:48.629817 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="c563a06d-f795-42e2-bb43-726fc4fd2a0b" containerName="main" Nov 03 03:22:48.630001 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:48.629823 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="80b9fd99-012a-4b15-88d0-34075301288f" containerName="payload-processing" Nov 03 03:22:48.632767 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:48.632747 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2232ad-g4stc" Nov 03 03:22:48.681496 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:48.681470 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:22:48.681637 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:48.681514 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:22:48.687026 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:48.686998 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c786e073-3944-4748-b180-f291d3616f68-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2232ad-g4stc\" (UID: \"c786e073-3944-4748-b180-f291d3616f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2232ad-g4stc" Nov 03 03:22:48.687143 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:48.687042 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7b2x\" (UniqueName: \"kubernetes.io/projected/c786e073-3944-4748-b180-f291d3616f68-kube-api-access-f7b2x\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2232ad-g4stc\" (UID: \"c786e073-3944-4748-b180-f291d3616f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2232ad-g4stc" Nov 03 03:22:48.687143 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:48.687067 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c786e073-3944-4748-b180-f291d3616f68-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2232ad-g4stc\" (UID: \"c786e073-3944-4748-b180-f291d3616f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2232ad-g4stc" Nov 03 03:22:48.728172 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:48.728147 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:22:48.728172 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:48.728168 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sb4qh\"" Nov 03 03:22:48.729636 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:48.729615 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2232ad-g4stc"] Nov 03 03:22:48.788518 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:48.788494 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c786e073-3944-4748-b180-f291d3616f68-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2232ad-g4stc\" (UID: \"c786e073-3944-4748-b180-f291d3616f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2232ad-g4stc" Nov 03 03:22:48.788623 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:48.788521 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7b2x\" (UniqueName: \"kubernetes.io/projected/c786e073-3944-4748-b180-f291d3616f68-kube-api-access-f7b2x\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2232ad-g4stc\" (UID: \"c786e073-3944-4748-b180-f291d3616f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2232ad-g4stc" Nov 03 03:22:48.788623 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:48.788540 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c786e073-3944-4748-b180-f291d3616f68-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2232ad-g4stc\" (UID: \"c786e073-3944-4748-b180-f291d3616f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2232ad-g4stc" Nov 03 03:22:48.789204 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:48.789183 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c786e073-3944-4748-b180-f291d3616f68-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2232ad-g4stc\" (UID: \"c786e073-3944-4748-b180-f291d3616f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2232ad-g4stc" Nov 03 03:22:48.790748 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:48.790727 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c786e073-3944-4748-b180-f291d3616f68-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2232ad-g4stc\" (UID: \"c786e073-3944-4748-b180-f291d3616f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2232ad-g4stc" Nov 03 03:22:48.842634 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:48.842606 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7b2x\" (UniqueName: \"kubernetes.io/projected/c786e073-3944-4748-b180-f291d3616f68-kube-api-access-f7b2x\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2232ad-g4stc\" (UID: \"c786e073-3944-4748-b180-f291d3616f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2232ad-g4stc" Nov 03 03:22:48.950476 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:48.950424 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2232ad-g4stc" Nov 03 03:22:49.312681 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:49.312652 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2232ad-g4stc"] Nov 03 03:22:49.320098 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:22:49.320070 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc786e073_3944_4748_b180_f291d3616f68.slice/crio-dbc04824c396844f54fc060773492ea96d4f38d55490757652ba8b48105455f3 WatchSource:0}: Error finding container dbc04824c396844f54fc060773492ea96d4f38d55490757652ba8b48105455f3: Status 404 returned error can't find the container with id dbc04824c396844f54fc060773492ea96d4f38d55490757652ba8b48105455f3 Nov 03 03:22:50.014052 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:50.013972 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2232ad-g4stc" event={"ID":"c786e073-3944-4748-b180-f291d3616f68","Type":"ContainerStarted","Data":"a686a7b270d8c81965607502a89e4b1f908dcfa9da3758a6312b71bd3c88edc9"} Nov 03 03:22:50.014052 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:50.014034 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2232ad-g4stc" event={"ID":"c786e073-3944-4748-b180-f291d3616f68","Type":"ContainerStarted","Data":"dbc04824c396844f54fc060773492ea96d4f38d55490757652ba8b48105455f3"} Nov 03 03:22:50.122292 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:50.122243 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2232ad-g4stc" podStartSLOduration=2.122228584 podStartE2EDuration="2.122228584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:22:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:22:50.120540511 +0000 UTC m=+5447.379738666" watchObservedRunningTime="2026-11-03 03:22:50.122228584 +0000 UTC m=+5447.381426782" Nov 03 03:22:51.019661 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:51.019628 5605 generic.go:356] "Generic (PLEG): container finished" podID="c786e073-3944-4748-b180-f291d3616f68" containerID="a686a7b270d8c81965607502a89e4b1f908dcfa9da3758a6312b71bd3c88edc9" exitCode=0 Nov 03 03:22:51.020057 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:51.019721 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2232ad-g4stc" event={"ID":"c786e073-3944-4748-b180-f291d3616f68","Type":"ContainerDied","Data":"a686a7b270d8c81965607502a89e4b1f908dcfa9da3758a6312b71bd3c88edc9"} Nov 03 03:22:52.151393 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:52.151353 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2232ad-g4stc" Nov 03 03:22:52.220374 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:52.220334 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c786e073-3944-4748-b180-f291d3616f68-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c786e073-3944-4748-b180-f291d3616f68-maas-api-xks-ca\") pod \"c786e073-3944-4748-b180-f291d3616f68\" (UID: \"c786e073-3944-4748-b180-f291d3616f68\") " Nov 03 03:22:52.220573 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:52.220438 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c786e073-3944-4748-b180-f291d3616f68-kube-api-access-f7b2x\" (UniqueName: \"kubernetes.io/projected/c786e073-3944-4748-b180-f291d3616f68-kube-api-access-f7b2x\") pod \"c786e073-3944-4748-b180-f291d3616f68\" (UID: \"c786e073-3944-4748-b180-f291d3616f68\") " Nov 03 03:22:52.220573 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:52.220564 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c786e073-3944-4748-b180-f291d3616f68-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c786e073-3944-4748-b180-f291d3616f68-maas-api-openshift-ca\") pod \"c786e073-3944-4748-b180-f291d3616f68\" (UID: \"c786e073-3944-4748-b180-f291d3616f68\") " Nov 03 03:22:52.221059 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:52.221034 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c786e073-3944-4748-b180-f291d3616f68-maas-api-openshift-ca" pod "c786e073-3944-4748-b180-f291d3616f68" (UID: "c786e073-3944-4748-b180-f291d3616f68"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:22:52.222777 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:52.222749 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c786e073-3944-4748-b180-f291d3616f68-kube-api-access-f7b2x" pod "c786e073-3944-4748-b180-f291d3616f68" (UID: "c786e073-3944-4748-b180-f291d3616f68"). InnerVolumeSpecName "kube-api-access-f7b2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:22:52.222996 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:52.222975 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c786e073-3944-4748-b180-f291d3616f68-maas-api-xks-ca" pod "c786e073-3944-4748-b180-f291d3616f68" (UID: "c786e073-3944-4748-b180-f291d3616f68"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:22:52.322315 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:52.322280 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7b2x\" (UniqueName: \"kubernetes.io/projected/c786e073-3944-4748-b180-f291d3616f68-kube-api-access-f7b2x\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:22:52.322315 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:52.322311 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c786e073-3944-4748-b180-f291d3616f68-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:22:52.322526 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:52.322326 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c786e073-3944-4748-b180-f291d3616f68-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:22:53.029456 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:53.029431 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2232ad-g4stc" Nov 03 03:22:53.029456 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:53.029452 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2232ad-g4stc" event={"ID":"c786e073-3944-4748-b180-f291d3616f68","Type":"ContainerDied","Data":"dbc04824c396844f54fc060773492ea96d4f38d55490757652ba8b48105455f3"} Nov 03 03:22:53.029670 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:22:53.029493 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dbc04824c396844f54fc060773492ea96d4f38d55490757652ba8b48105455f3" Nov 03 03:22:53.912211 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:22:53.912175 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:23:05.746379 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:05.746339 5605 scope.go:122] "RemoveContainer" containerID="c2fb4c0d1381dcbfa7bf494f92d3675a589174d39de2064c75819022a0f83275" Nov 03 03:23:07.943708 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:07.943669 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2232ad-g4stc"] Nov 03 03:23:07.990083 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:07.990047 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2232ad-g4stc"] Nov 03 03:23:08.224147 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:08.224051 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl"] Nov 03 03:23:08.224411 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:08.224340 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl" podUID="1437932f-1521-4f4d-a1d5-6d4be6fca6ce" containerName="istio-proxy" containerID="cri-o://82bd16825388638234e36b7d9484b7dd58bf0be30d3560eda24c51dedcfde855" gracePeriod=30 Nov 03 03:23:08.848107 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:08.848070 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c786e073-3944-4748-b180-f291d3616f68" path="/var/lib/kubelet/pods/c786e073-3944-4748-b180-f291d3616f68/volumes" Nov 03 03:23:10.716842 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:10.716807 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:23:13.470607 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:13.470575 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl" Nov 03 03:23:13.611154 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:13.611123 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1437932f-1521-4f4d-a1d5-6d4be6fca6ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1437932f-1521-4f4d-a1d5-6d4be6fca6ce-workload-certs\") pod \"1437932f-1521-4f4d-a1d5-6d4be6fca6ce\" (UID: \"1437932f-1521-4f4d-a1d5-6d4be6fca6ce\") " Nov 03 03:23:13.611406 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:13.611162 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1437932f-1521-4f4d-a1d5-6d4be6fca6ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1437932f-1521-4f4d-a1d5-6d4be6fca6ce-istio-data\") pod \"1437932f-1521-4f4d-a1d5-6d4be6fca6ce\" (UID: \"1437932f-1521-4f4d-a1d5-6d4be6fca6ce\") " Nov 03 03:23:13.611406 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:13.611183 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1437932f-1521-4f4d-a1d5-6d4be6fca6ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1437932f-1521-4f4d-a1d5-6d4be6fca6ce-istiod-ca-cert\") pod \"1437932f-1521-4f4d-a1d5-6d4be6fca6ce\" (UID: \"1437932f-1521-4f4d-a1d5-6d4be6fca6ce\") " Nov 03 03:23:13.611406 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:13.611228 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1437932f-1521-4f4d-a1d5-6d4be6fca6ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1437932f-1521-4f4d-a1d5-6d4be6fca6ce-istio-podinfo\") pod \"1437932f-1521-4f4d-a1d5-6d4be6fca6ce\" (UID: \"1437932f-1521-4f4d-a1d5-6d4be6fca6ce\") " Nov 03 03:23:13.611406 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:13.611276 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1437932f-1521-4f4d-a1d5-6d4be6fca6ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1437932f-1521-4f4d-a1d5-6d4be6fca6ce-workload-socket\") pod \"1437932f-1521-4f4d-a1d5-6d4be6fca6ce\" (UID: \"1437932f-1521-4f4d-a1d5-6d4be6fca6ce\") " Nov 03 03:23:13.611406 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:13.611331 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1437932f-1521-4f4d-a1d5-6d4be6fca6ce-kube-api-access-qtv5x\" (UniqueName: \"kubernetes.io/projected/1437932f-1521-4f4d-a1d5-6d4be6fca6ce-kube-api-access-qtv5x\") pod \"1437932f-1521-4f4d-a1d5-6d4be6fca6ce\" (UID: \"1437932f-1521-4f4d-a1d5-6d4be6fca6ce\") " Nov 03 03:23:13.611714 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:13.611412 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1437932f-1521-4f4d-a1d5-6d4be6fca6ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1437932f-1521-4f4d-a1d5-6d4be6fca6ce-istio-envoy\") pod \"1437932f-1521-4f4d-a1d5-6d4be6fca6ce\" (UID: \"1437932f-1521-4f4d-a1d5-6d4be6fca6ce\") " Nov 03 03:23:13.611714 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:13.611461 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1437932f-1521-4f4d-a1d5-6d4be6fca6ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1437932f-1521-4f4d-a1d5-6d4be6fca6ce-credential-socket\") pod \"1437932f-1521-4f4d-a1d5-6d4be6fca6ce\" (UID: \"1437932f-1521-4f4d-a1d5-6d4be6fca6ce\") " Nov 03 03:23:13.611714 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:13.611519 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1437932f-1521-4f4d-a1d5-6d4be6fca6ce-workload-certs" pod "1437932f-1521-4f4d-a1d5-6d4be6fca6ce" (UID: "1437932f-1521-4f4d-a1d5-6d4be6fca6ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:23:13.611714 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:13.611524 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1437932f-1521-4f4d-a1d5-6d4be6fca6ce-istio-token\" (UniqueName: \"kubernetes.io/projected/1437932f-1521-4f4d-a1d5-6d4be6fca6ce-istio-token\") pod \"1437932f-1521-4f4d-a1d5-6d4be6fca6ce\" (UID: \"1437932f-1521-4f4d-a1d5-6d4be6fca6ce\") " Nov 03 03:23:13.611714 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:13.611534 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1437932f-1521-4f4d-a1d5-6d4be6fca6ce-istio-data" pod "1437932f-1521-4f4d-a1d5-6d4be6fca6ce" (UID: "1437932f-1521-4f4d-a1d5-6d4be6fca6ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:23:13.611714 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:13.611650 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1437932f-1521-4f4d-a1d5-6d4be6fca6ce-istiod-ca-cert" pod "1437932f-1521-4f4d-a1d5-6d4be6fca6ce" (UID: "1437932f-1521-4f4d-a1d5-6d4be6fca6ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:23:13.612037 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:13.611804 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1437932f-1521-4f4d-a1d5-6d4be6fca6ce-credential-socket" pod "1437932f-1521-4f4d-a1d5-6d4be6fca6ce" (UID: "1437932f-1521-4f4d-a1d5-6d4be6fca6ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:23:13.612037 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:13.611897 5605 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1437932f-1521-4f4d-a1d5-6d4be6fca6ce-workload-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:23:13.612037 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:13.611917 5605 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1437932f-1521-4f4d-a1d5-6d4be6fca6ce-istio-data\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:23:13.612037 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:13.611926 5605 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1437932f-1521-4f4d-a1d5-6d4be6fca6ce-istiod-ca-cert\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:23:13.612037 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:13.611936 5605 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1437932f-1521-4f4d-a1d5-6d4be6fca6ce-credential-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:23:13.612037 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:13.611969 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1437932f-1521-4f4d-a1d5-6d4be6fca6ce-workload-socket" pod "1437932f-1521-4f4d-a1d5-6d4be6fca6ce" (UID: "1437932f-1521-4f4d-a1d5-6d4be6fca6ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:23:13.613702 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:13.613674 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1437932f-1521-4f4d-a1d5-6d4be6fca6ce-istio-envoy" pod "1437932f-1521-4f4d-a1d5-6d4be6fca6ce" (UID: "1437932f-1521-4f4d-a1d5-6d4be6fca6ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:23:13.613810 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:13.613709 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1437932f-1521-4f4d-a1d5-6d4be6fca6ce-istio-podinfo" pod "1437932f-1521-4f4d-a1d5-6d4be6fca6ce" (UID: "1437932f-1521-4f4d-a1d5-6d4be6fca6ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:23:13.613978 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:13.613964 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1437932f-1521-4f4d-a1d5-6d4be6fca6ce-istio-token" pod "1437932f-1521-4f4d-a1d5-6d4be6fca6ce" (UID: "1437932f-1521-4f4d-a1d5-6d4be6fca6ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:23:13.614073 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:13.614050 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1437932f-1521-4f4d-a1d5-6d4be6fca6ce-kube-api-access-qtv5x" pod "1437932f-1521-4f4d-a1d5-6d4be6fca6ce" (UID: "1437932f-1521-4f4d-a1d5-6d4be6fca6ce"). InnerVolumeSpecName "kube-api-access-qtv5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:23:13.712687 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:13.712652 5605 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1437932f-1521-4f4d-a1d5-6d4be6fca6ce-istio-token\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:23:13.712687 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:13.712681 5605 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1437932f-1521-4f4d-a1d5-6d4be6fca6ce-istio-podinfo\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:23:13.712687 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:13.712692 5605 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1437932f-1521-4f4d-a1d5-6d4be6fca6ce-workload-socket\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:23:13.712940 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:13.712703 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtv5x\" (UniqueName: \"kubernetes.io/projected/1437932f-1521-4f4d-a1d5-6d4be6fca6ce-kube-api-access-qtv5x\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:23:13.712940 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:13.712716 5605 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1437932f-1521-4f4d-a1d5-6d4be6fca6ce-istio-envoy\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:23:14.126565 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:14.126526 5605 generic.go:356] "Generic (PLEG): container finished" podID="1437932f-1521-4f4d-a1d5-6d4be6fca6ce" containerID="82bd16825388638234e36b7d9484b7dd58bf0be30d3560eda24c51dedcfde855" exitCode=0 Nov 03 03:23:14.126743 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:14.126586 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl" Nov 03 03:23:14.126743 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:14.126599 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl" event={"ID":"1437932f-1521-4f4d-a1d5-6d4be6fca6ce","Type":"ContainerDied","Data":"82bd16825388638234e36b7d9484b7dd58bf0be30d3560eda24c51dedcfde855"} Nov 03 03:23:14.126743 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:14.126652 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl" event={"ID":"1437932f-1521-4f4d-a1d5-6d4be6fca6ce","Type":"ContainerDied","Data":"4d6a753493ff504f6e3d52dd6d346261e774f79856ea419046bff04533232ec5"} Nov 03 03:23:14.126743 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:14.126682 5605 scope.go:122] "RemoveContainer" containerID="82bd16825388638234e36b7d9484b7dd58bf0be30d3560eda24c51dedcfde855" Nov 03 03:23:14.136390 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:14.136350 5605 scope.go:122] "RemoveContainer" containerID="64e4f4e4a137d9333213b5d16cfea7bbc4d1c206cc20196e861d7a318be8a54d" Nov 03 03:23:14.143986 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:14.143966 5605 scope.go:122] "RemoveContainer" containerID="82bd16825388638234e36b7d9484b7dd58bf0be30d3560eda24c51dedcfde855" Nov 03 03:23:14.144239 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:23:14.144215 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82bd16825388638234e36b7d9484b7dd58bf0be30d3560eda24c51dedcfde855\": container with ID starting with 82bd16825388638234e36b7d9484b7dd58bf0be30d3560eda24c51dedcfde855 not found: ID does not exist" containerID="82bd16825388638234e36b7d9484b7dd58bf0be30d3560eda24c51dedcfde855" Nov 03 03:23:14.144304 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:14.144254 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82bd16825388638234e36b7d9484b7dd58bf0be30d3560eda24c51dedcfde855"} err="failed to get container status \"82bd16825388638234e36b7d9484b7dd58bf0be30d3560eda24c51dedcfde855\": rpc error: code = NotFound desc = could not find container \"82bd16825388638234e36b7d9484b7dd58bf0be30d3560eda24c51dedcfde855\": container with ID starting with 82bd16825388638234e36b7d9484b7dd58bf0be30d3560eda24c51dedcfde855 not found: ID does not exist" Nov 03 03:23:14.144304 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:14.144278 5605 scope.go:122] "RemoveContainer" containerID="64e4f4e4a137d9333213b5d16cfea7bbc4d1c206cc20196e861d7a318be8a54d" Nov 03 03:23:14.144923 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:23:14.144526 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64e4f4e4a137d9333213b5d16cfea7bbc4d1c206cc20196e861d7a318be8a54d\": container with ID starting with 64e4f4e4a137d9333213b5d16cfea7bbc4d1c206cc20196e861d7a318be8a54d not found: ID does not exist" containerID="64e4f4e4a137d9333213b5d16cfea7bbc4d1c206cc20196e861d7a318be8a54d" Nov 03 03:23:14.144923 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:14.144545 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64e4f4e4a137d9333213b5d16cfea7bbc4d1c206cc20196e861d7a318be8a54d"} err="failed to get container status \"64e4f4e4a137d9333213b5d16cfea7bbc4d1c206cc20196e861d7a318be8a54d\": rpc error: code = NotFound desc = could not find container \"64e4f4e4a137d9333213b5d16cfea7bbc4d1c206cc20196e861d7a318be8a54d\": container with ID starting with 64e4f4e4a137d9333213b5d16cfea7bbc4d1c206cc20196e861d7a318be8a54d not found: ID does not exist" Nov 03 03:23:14.280366 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:14.280322 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl"] Nov 03 03:23:14.326727 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:14.326700 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2232ad-openshift-default-6c84bbdbf-fxfxl"] Nov 03 03:23:14.848843 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:14.848811 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1437932f-1521-4f4d-a1d5-6d4be6fca6ce" path="/var/lib/kubelet/pods/1437932f-1521-4f4d-a1d5-6d4be6fca6ce/volumes" Nov 03 03:23:16.495883 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:16.495841 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-e1520875-kserve-757cbb86b9-7gq6c"] Nov 03 03:23:16.496489 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:16.496327 5605 cpu_manager.go:414] "containerMap: removing container" podUID="1437932f-1521-4f4d-a1d5-6d4be6fca6ce" containerName="istio-proxy" Nov 03 03:23:16.496489 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:16.496347 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1437932f-1521-4f4d-a1d5-6d4be6fca6ce" containerName="istio-proxy" Nov 03 03:23:16.496489 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:16.496384 5605 cpu_manager.go:414] "containerMap: removing container" podUID="c786e073-3944-4748-b180-f291d3616f68" containerName="revoke-keys" Nov 03 03:23:16.496489 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:16.496393 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c786e073-3944-4748-b180-f291d3616f68" containerName="revoke-keys" Nov 03 03:23:16.496728 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:16.496523 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="1437932f-1521-4f4d-a1d5-6d4be6fca6ce" containerName="istio-proxy" Nov 03 03:23:16.496728 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:16.496537 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="c786e073-3944-4748-b180-f291d3616f68" containerName="revoke-keys" Nov 03 03:23:16.496728 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:16.496552 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="1437932f-1521-4f4d-a1d5-6d4be6fca6ce" containerName="istio-proxy" Nov 03 03:23:16.501203 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:16.501180 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e1520875-kserve-757cbb86b9-7gq6c" Nov 03 03:23:16.598999 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:16.598964 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-e1520875-kserve-self-signed-certs\"" Nov 03 03:23:16.598999 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:16.598975 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-nlcr8\"" Nov 03 03:23:16.599249 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:16.599058 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 03 03:23:16.599249 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:16.599088 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 03 03:23:16.600515 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:16.600491 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-e1520875-kserve-757cbb86b9-7gq6c"] Nov 03 03:23:16.637708 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:16.637668 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6-model-cache\") pod \"e2e-same-id-a-e1520875-kserve-757cbb86b9-7gq6c\" (UID: \"9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6\") " pod="llm/e2e-same-id-a-e1520875-kserve-757cbb86b9-7gq6c" Nov 03 03:23:16.638205 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:16.637762 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6-tls-certs\") pod \"e2e-same-id-a-e1520875-kserve-757cbb86b9-7gq6c\" (UID: \"9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6\") " pod="llm/e2e-same-id-a-e1520875-kserve-757cbb86b9-7gq6c" Nov 03 03:23:16.638205 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:16.637793 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6-dshm\") pod \"e2e-same-id-a-e1520875-kserve-757cbb86b9-7gq6c\" (UID: \"9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6\") " pod="llm/e2e-same-id-a-e1520875-kserve-757cbb86b9-7gq6c" Nov 03 03:23:16.638205 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:16.637845 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4w9n\" (UniqueName: \"kubernetes.io/projected/9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6-kube-api-access-b4w9n\") pod \"e2e-same-id-a-e1520875-kserve-757cbb86b9-7gq6c\" (UID: \"9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6\") " pod="llm/e2e-same-id-a-e1520875-kserve-757cbb86b9-7gq6c" Nov 03 03:23:16.638205 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:16.637902 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6-tmp-dir\") pod \"e2e-same-id-a-e1520875-kserve-757cbb86b9-7gq6c\" (UID: \"9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6\") " pod="llm/e2e-same-id-a-e1520875-kserve-757cbb86b9-7gq6c" Nov 03 03:23:16.638205 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:16.637932 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6-home\") pod \"e2e-same-id-a-e1520875-kserve-757cbb86b9-7gq6c\" (UID: \"9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6\") " pod="llm/e2e-same-id-a-e1520875-kserve-757cbb86b9-7gq6c" Nov 03 03:23:16.738968 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:16.738935 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6-tmp-dir\") pod \"e2e-same-id-a-e1520875-kserve-757cbb86b9-7gq6c\" (UID: \"9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6\") " pod="llm/e2e-same-id-a-e1520875-kserve-757cbb86b9-7gq6c" Nov 03 03:23:16.739193 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:16.738977 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6-home\") pod \"e2e-same-id-a-e1520875-kserve-757cbb86b9-7gq6c\" (UID: \"9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6\") " pod="llm/e2e-same-id-a-e1520875-kserve-757cbb86b9-7gq6c" Nov 03 03:23:16.739193 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:16.739021 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6-model-cache\") pod \"e2e-same-id-a-e1520875-kserve-757cbb86b9-7gq6c\" (UID: \"9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6\") " pod="llm/e2e-same-id-a-e1520875-kserve-757cbb86b9-7gq6c" Nov 03 03:23:16.739193 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:16.739086 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6-tls-certs\") pod \"e2e-same-id-a-e1520875-kserve-757cbb86b9-7gq6c\" (UID: \"9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6\") " pod="llm/e2e-same-id-a-e1520875-kserve-757cbb86b9-7gq6c" Nov 03 03:23:16.739193 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:16.739113 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6-dshm\") pod \"e2e-same-id-a-e1520875-kserve-757cbb86b9-7gq6c\" (UID: \"9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6\") " pod="llm/e2e-same-id-a-e1520875-kserve-757cbb86b9-7gq6c" Nov 03 03:23:16.739465 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:16.739217 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4w9n\" (UniqueName: \"kubernetes.io/projected/9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6-kube-api-access-b4w9n\") pod \"e2e-same-id-a-e1520875-kserve-757cbb86b9-7gq6c\" (UID: \"9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6\") " pod="llm/e2e-same-id-a-e1520875-kserve-757cbb86b9-7gq6c" Nov 03 03:23:16.739465 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:16.739440 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6-tmp-dir\") pod \"e2e-same-id-a-e1520875-kserve-757cbb86b9-7gq6c\" (UID: \"9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6\") " pod="llm/e2e-same-id-a-e1520875-kserve-757cbb86b9-7gq6c" Nov 03 03:23:16.739578 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:16.739477 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6-home\") pod \"e2e-same-id-a-e1520875-kserve-757cbb86b9-7gq6c\" (UID: \"9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6\") " pod="llm/e2e-same-id-a-e1520875-kserve-757cbb86b9-7gq6c" Nov 03 03:23:16.739578 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:16.739493 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6-model-cache\") pod \"e2e-same-id-a-e1520875-kserve-757cbb86b9-7gq6c\" (UID: \"9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6\") " pod="llm/e2e-same-id-a-e1520875-kserve-757cbb86b9-7gq6c" Nov 03 03:23:16.741528 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:16.741500 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6-dshm\") pod \"e2e-same-id-a-e1520875-kserve-757cbb86b9-7gq6c\" (UID: \"9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6\") " pod="llm/e2e-same-id-a-e1520875-kserve-757cbb86b9-7gq6c" Nov 03 03:23:16.741732 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:16.741717 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6-tls-certs\") pod \"e2e-same-id-a-e1520875-kserve-757cbb86b9-7gq6c\" (UID: \"9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6\") " pod="llm/e2e-same-id-a-e1520875-kserve-757cbb86b9-7gq6c" Nov 03 03:23:16.792718 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:16.792663 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4w9n\" (UniqueName: \"kubernetes.io/projected/9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6-kube-api-access-b4w9n\") pod \"e2e-same-id-a-e1520875-kserve-757cbb86b9-7gq6c\" (UID: \"9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6\") " pod="llm/e2e-same-id-a-e1520875-kserve-757cbb86b9-7gq6c" Nov 03 03:23:16.811633 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:16.811606 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e1520875-kserve-757cbb86b9-7gq6c" Nov 03 03:23:17.181961 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:17.181931 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-e1520875-kserve-757cbb86b9-7gq6c"] Nov 03 03:23:17.188104 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:23:17.188073 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9b37bcbe_54de_4c63_a24c_8b7f4a6d43d6.slice/crio-d89af6f66cafdeeeb55d533dd44c6a6bb44dc4b7ee32f99dfb14f877ba568735 WatchSource:0}: Error finding container d89af6f66cafdeeeb55d533dd44c6a6bb44dc4b7ee32f99dfb14f877ba568735: Status 404 returned error can't find the container with id d89af6f66cafdeeeb55d533dd44c6a6bb44dc4b7ee32f99dfb14f877ba568735 Nov 03 03:23:18.147185 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:18.147143 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e1520875-kserve-757cbb86b9-7gq6c" event={"ID":"9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6","Type":"ContainerStarted","Data":"bec66f72f6ee86d7a65dbbae33ac1de222255865fa782038fd72be9b37294ba6"} Nov 03 03:23:18.147601 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:18.147196 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e1520875-kserve-757cbb86b9-7gq6c" event={"ID":"9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6","Type":"ContainerStarted","Data":"d89af6f66cafdeeeb55d533dd44c6a6bb44dc4b7ee32f99dfb14f877ba568735"} Nov 03 03:23:18.254654 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:18.254598 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-e1520875-kserve-757cbb86b9-7gq6c" podStartSLOduration=2.254582098 podStartE2EDuration="2.254582098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:23:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:23:18.253731716 +0000 UTC m=+5475.512929871" watchObservedRunningTime="2026-11-03 03:23:18.254582098 +0000 UTC m=+5475.513780253" Nov 03 03:23:18.812309 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:18.812273 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-e1520875-kserve-757cbb86b9-7gq6c" Nov 03 03:23:22.528513 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:22.528474 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:23:23.925991 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:23:23.925950 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:23:26.812012 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:26.811978 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-e1520875-kserve-757cbb86b9-7gq6c" Nov 03 03:23:26.824089 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:26.824066 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-e1520875-kserve-757cbb86b9-7gq6c" Nov 03 03:23:27.275094 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:27.275015 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-e1520875-kserve-757cbb86b9-7gq6c" Nov 03 03:23:42.394528 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:42.394441 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-e1520875-kserve-fc9f8bcc-nh69b"] Nov 03 03:23:42.395137 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:42.394920 5605 cpu_manager.go:414] "containerMap: removing container" podUID="1437932f-1521-4f4d-a1d5-6d4be6fca6ce" containerName="istio-proxy" Nov 03 03:23:42.395137 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:42.394940 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1437932f-1521-4f4d-a1d5-6d4be6fca6ce" containerName="istio-proxy" Nov 03 03:23:42.398521 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:42.398499 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e1520875-kserve-fc9f8bcc-nh69b" Nov 03 03:23:42.420411 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:42.420379 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58dcb314-1d08-41a7-82ce-18bac7d3ddcf-tls-certs\") pod \"e2e-same-id-b-e1520875-kserve-fc9f8bcc-nh69b\" (UID: \"58dcb314-1d08-41a7-82ce-18bac7d3ddcf\") " pod="llm/e2e-same-id-b-e1520875-kserve-fc9f8bcc-nh69b" Nov 03 03:23:42.420502 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:42.420423 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58dcb314-1d08-41a7-82ce-18bac7d3ddcf-model-cache\") pod \"e2e-same-id-b-e1520875-kserve-fc9f8bcc-nh69b\" (UID: \"58dcb314-1d08-41a7-82ce-18bac7d3ddcf\") " pod="llm/e2e-same-id-b-e1520875-kserve-fc9f8bcc-nh69b" Nov 03 03:23:42.420502 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:42.420487 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58dcb314-1d08-41a7-82ce-18bac7d3ddcf-tmp-dir\") pod \"e2e-same-id-b-e1520875-kserve-fc9f8bcc-nh69b\" (UID: \"58dcb314-1d08-41a7-82ce-18bac7d3ddcf\") " pod="llm/e2e-same-id-b-e1520875-kserve-fc9f8bcc-nh69b" Nov 03 03:23:42.420582 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:42.420554 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd9lc\" (UniqueName: \"kubernetes.io/projected/58dcb314-1d08-41a7-82ce-18bac7d3ddcf-kube-api-access-vd9lc\") pod \"e2e-same-id-b-e1520875-kserve-fc9f8bcc-nh69b\" (UID: \"58dcb314-1d08-41a7-82ce-18bac7d3ddcf\") " pod="llm/e2e-same-id-b-e1520875-kserve-fc9f8bcc-nh69b" Nov 03 03:23:42.420582 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:42.420571 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58dcb314-1d08-41a7-82ce-18bac7d3ddcf-dshm\") pod \"e2e-same-id-b-e1520875-kserve-fc9f8bcc-nh69b\" (UID: \"58dcb314-1d08-41a7-82ce-18bac7d3ddcf\") " pod="llm/e2e-same-id-b-e1520875-kserve-fc9f8bcc-nh69b" Nov 03 03:23:42.420653 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:42.420633 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58dcb314-1d08-41a7-82ce-18bac7d3ddcf-home\") pod \"e2e-same-id-b-e1520875-kserve-fc9f8bcc-nh69b\" (UID: \"58dcb314-1d08-41a7-82ce-18bac7d3ddcf\") " pod="llm/e2e-same-id-b-e1520875-kserve-fc9f8bcc-nh69b" Nov 03 03:23:42.496806 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:42.496778 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-e1520875-kserve-self-signed-certs\"" Nov 03 03:23:42.497955 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:42.497933 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-e1520875-kserve-fc9f8bcc-nh69b"] Nov 03 03:23:42.521378 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:42.521321 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd9lc\" (UniqueName: \"kubernetes.io/projected/58dcb314-1d08-41a7-82ce-18bac7d3ddcf-kube-api-access-vd9lc\") pod \"e2e-same-id-b-e1520875-kserve-fc9f8bcc-nh69b\" (UID: \"58dcb314-1d08-41a7-82ce-18bac7d3ddcf\") " pod="llm/e2e-same-id-b-e1520875-kserve-fc9f8bcc-nh69b" Nov 03 03:23:42.521496 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:42.521385 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58dcb314-1d08-41a7-82ce-18bac7d3ddcf-dshm\") pod \"e2e-same-id-b-e1520875-kserve-fc9f8bcc-nh69b\" (UID: \"58dcb314-1d08-41a7-82ce-18bac7d3ddcf\") " pod="llm/e2e-same-id-b-e1520875-kserve-fc9f8bcc-nh69b" Nov 03 03:23:42.521496 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:42.521444 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58dcb314-1d08-41a7-82ce-18bac7d3ddcf-home\") pod \"e2e-same-id-b-e1520875-kserve-fc9f8bcc-nh69b\" (UID: \"58dcb314-1d08-41a7-82ce-18bac7d3ddcf\") " pod="llm/e2e-same-id-b-e1520875-kserve-fc9f8bcc-nh69b" Nov 03 03:23:42.521646 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:42.521526 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58dcb314-1d08-41a7-82ce-18bac7d3ddcf-tls-certs\") pod \"e2e-same-id-b-e1520875-kserve-fc9f8bcc-nh69b\" (UID: \"58dcb314-1d08-41a7-82ce-18bac7d3ddcf\") " pod="llm/e2e-same-id-b-e1520875-kserve-fc9f8bcc-nh69b" Nov 03 03:23:42.521646 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:42.521567 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58dcb314-1d08-41a7-82ce-18bac7d3ddcf-model-cache\") pod \"e2e-same-id-b-e1520875-kserve-fc9f8bcc-nh69b\" (UID: \"58dcb314-1d08-41a7-82ce-18bac7d3ddcf\") " pod="llm/e2e-same-id-b-e1520875-kserve-fc9f8bcc-nh69b" Nov 03 03:23:42.521646 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:42.521594 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58dcb314-1d08-41a7-82ce-18bac7d3ddcf-tmp-dir\") pod \"e2e-same-id-b-e1520875-kserve-fc9f8bcc-nh69b\" (UID: \"58dcb314-1d08-41a7-82ce-18bac7d3ddcf\") " pod="llm/e2e-same-id-b-e1520875-kserve-fc9f8bcc-nh69b" Nov 03 03:23:42.521886 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:42.521863 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58dcb314-1d08-41a7-82ce-18bac7d3ddcf-home\") pod \"e2e-same-id-b-e1520875-kserve-fc9f8bcc-nh69b\" (UID: \"58dcb314-1d08-41a7-82ce-18bac7d3ddcf\") " pod="llm/e2e-same-id-b-e1520875-kserve-fc9f8bcc-nh69b" Nov 03 03:23:42.521983 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:42.521893 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58dcb314-1d08-41a7-82ce-18bac7d3ddcf-model-cache\") pod \"e2e-same-id-b-e1520875-kserve-fc9f8bcc-nh69b\" (UID: \"58dcb314-1d08-41a7-82ce-18bac7d3ddcf\") " pod="llm/e2e-same-id-b-e1520875-kserve-fc9f8bcc-nh69b" Nov 03 03:23:42.521983 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:42.521932 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58dcb314-1d08-41a7-82ce-18bac7d3ddcf-tmp-dir\") pod \"e2e-same-id-b-e1520875-kserve-fc9f8bcc-nh69b\" (UID: \"58dcb314-1d08-41a7-82ce-18bac7d3ddcf\") " pod="llm/e2e-same-id-b-e1520875-kserve-fc9f8bcc-nh69b" Nov 03 03:23:42.523840 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:42.523818 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58dcb314-1d08-41a7-82ce-18bac7d3ddcf-dshm\") pod \"e2e-same-id-b-e1520875-kserve-fc9f8bcc-nh69b\" (UID: \"58dcb314-1d08-41a7-82ce-18bac7d3ddcf\") " pod="llm/e2e-same-id-b-e1520875-kserve-fc9f8bcc-nh69b" Nov 03 03:23:42.523976 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:42.523959 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58dcb314-1d08-41a7-82ce-18bac7d3ddcf-tls-certs\") pod \"e2e-same-id-b-e1520875-kserve-fc9f8bcc-nh69b\" (UID: \"58dcb314-1d08-41a7-82ce-18bac7d3ddcf\") " pod="llm/e2e-same-id-b-e1520875-kserve-fc9f8bcc-nh69b" Nov 03 03:23:42.576046 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:42.576007 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd9lc\" (UniqueName: \"kubernetes.io/projected/58dcb314-1d08-41a7-82ce-18bac7d3ddcf-kube-api-access-vd9lc\") pod \"e2e-same-id-b-e1520875-kserve-fc9f8bcc-nh69b\" (UID: \"58dcb314-1d08-41a7-82ce-18bac7d3ddcf\") " pod="llm/e2e-same-id-b-e1520875-kserve-fc9f8bcc-nh69b" Nov 03 03:23:42.710948 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:42.710879 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e1520875-kserve-fc9f8bcc-nh69b" Nov 03 03:23:43.080246 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:43.080198 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-e1520875-kserve-fc9f8bcc-nh69b"] Nov 03 03:23:43.088422 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:23:43.088384 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod58dcb314_1d08_41a7_82ce_18bac7d3ddcf.slice/crio-00ba492bc17ad75d2153eb5d619405f4490c60664cc66ef667d57463aa904c7d WatchSource:0}: Error finding container 00ba492bc17ad75d2153eb5d619405f4490c60664cc66ef667d57463aa904c7d: Status 404 returned error can't find the container with id 00ba492bc17ad75d2153eb5d619405f4490c60664cc66ef667d57463aa904c7d Nov 03 03:23:43.336258 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:43.336178 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e1520875-kserve-fc9f8bcc-nh69b" event={"ID":"58dcb314-1d08-41a7-82ce-18bac7d3ddcf","Type":"ContainerStarted","Data":"e197e208c62767274f01432531f7cdddfb3f2bdda7d0ca7b83dabf4d5730e0ac"} Nov 03 03:23:43.336258 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:43.336221 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e1520875-kserve-fc9f8bcc-nh69b" event={"ID":"58dcb314-1d08-41a7-82ce-18bac7d3ddcf","Type":"ContainerStarted","Data":"00ba492bc17ad75d2153eb5d619405f4490c60664cc66ef667d57463aa904c7d"} Nov 03 03:23:43.445442 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:43.445395 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-e1520875-kserve-fc9f8bcc-nh69b" podStartSLOduration=1.445382129 podStartE2EDuration="1.445382129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:23:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:23:43.443621905 +0000 UTC m=+5500.702820059" watchObservedRunningTime="2026-11-03 03:23:43.445382129 +0000 UTC m=+5500.704580286" Nov 03 03:23:43.712004 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:43.711927 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-e1520875-kserve-fc9f8bcc-nh69b" Nov 03 03:23:46.888287 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:46.888253 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:23:52.712016 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:52.711982 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-e1520875-kserve-fc9f8bcc-nh69b" Nov 03 03:23:52.724210 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:52.724177 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-e1520875-kserve-fc9f8bcc-nh69b" Nov 03 03:23:53.446905 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:23:53.446875 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-e1520875-kserve-fc9f8bcc-nh69b" Nov 03 03:23:53.910437 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:23:53.910402 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:24:05.777966 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:05.777934 5605 scope.go:122] "RemoveContainer" containerID="a57e2d0eb73e240b721a04f211acabf55e758a11b5457e42045c64d5ef9b3cb2" Nov 03 03:24:12.948257 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:12.948226 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:24:23.912191 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:24:23.912147 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:24:24.691987 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:24.691948 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-e1520875-kserve-757cbb86b9-7gq6c"] Nov 03 03:24:28.335348 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:28.335313 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:24:31.281971 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:31.281932 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-e1520875-kserve-fc9f8bcc-nh69b"] Nov 03 03:24:39.720381 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:39.719565 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-e1520875-kserve-757cbb86b9-7gq6c" podUID="9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6" containerName="main" containerID="cri-o://bec66f72f6ee86d7a65dbbae33ac1de222255865fa782038fd72be9b37294ba6" gracePeriod=45 Nov 03 03:24:39.972957 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:39.972901 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e1520875-kserve-757cbb86b9-7gq6c" Nov 03 03:24:40.068196 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:40.068168 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6-tmp-dir\") pod \"9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6\" (UID: \"9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6\") " Nov 03 03:24:40.068331 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:40.068214 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6-dshm\") pod \"9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6\" (UID: \"9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6\") " Nov 03 03:24:40.068331 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:40.068273 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6-model-cache\") pod \"9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6\" (UID: \"9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6\") " Nov 03 03:24:40.068331 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:40.068325 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6-kube-api-access-b4w9n\" (UniqueName: \"kubernetes.io/projected/9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6-kube-api-access-b4w9n\") pod \"9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6\" (UID: \"9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6\") " Nov 03 03:24:40.068546 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:40.068374 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6-home\" (UniqueName: \"kubernetes.io/empty-dir/9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6-home\") pod \"9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6\" (UID: \"9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6\") " Nov 03 03:24:40.068546 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:40.068414 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6-tls-certs\" (UniqueName: \"kubernetes.io/secret/9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6-tls-certs\") pod \"9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6\" (UID: \"9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6\") " Nov 03 03:24:40.068546 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:40.068428 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6-tmp-dir" pod "9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6" (UID: "9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:24:40.068721 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:40.068602 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6-home" pod "9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6" (UID: "9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:24:40.068721 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:40.068616 5605 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6-tmp-dir\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:24:40.068721 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:40.068617 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6-model-cache" pod "9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6" (UID: "9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:24:40.070455 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:40.070430 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6-dshm" pod "9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6" (UID: "9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:24:40.070575 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:40.070487 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6-kube-api-access-b4w9n" pod "9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6" (UID: "9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6"). InnerVolumeSpecName "kube-api-access-b4w9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:24:40.070575 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:40.070547 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6-tls-certs" pod "9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6" (UID: "9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:24:40.169452 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:40.169425 5605 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6-model-cache\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:24:40.169452 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:40.169447 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4w9n\" (UniqueName: \"kubernetes.io/projected/9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6-kube-api-access-b4w9n\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:24:40.169452 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:40.169457 5605 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6-home\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:24:40.169642 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:40.169465 5605 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6-tls-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:24:40.169642 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:40.169473 5605 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6-dshm\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:24:40.738170 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:40.738134 5605 generic.go:356] "Generic (PLEG): container finished" podID="9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6" containerID="bec66f72f6ee86d7a65dbbae33ac1de222255865fa782038fd72be9b37294ba6" exitCode=0 Nov 03 03:24:40.738631 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:40.738179 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e1520875-kserve-757cbb86b9-7gq6c" Nov 03 03:24:40.738631 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:40.738210 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e1520875-kserve-757cbb86b9-7gq6c" event={"ID":"9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6","Type":"ContainerDied","Data":"bec66f72f6ee86d7a65dbbae33ac1de222255865fa782038fd72be9b37294ba6"} Nov 03 03:24:40.738631 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:40.738250 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e1520875-kserve-757cbb86b9-7gq6c" event={"ID":"9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6","Type":"ContainerDied","Data":"d89af6f66cafdeeeb55d533dd44c6a6bb44dc4b7ee32f99dfb14f877ba568735"} Nov 03 03:24:40.738631 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:40.738266 5605 scope.go:122] "RemoveContainer" containerID="bec66f72f6ee86d7a65dbbae33ac1de222255865fa782038fd72be9b37294ba6" Nov 03 03:24:40.747088 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:40.747073 5605 scope.go:122] "RemoveContainer" containerID="bec66f72f6ee86d7a65dbbae33ac1de222255865fa782038fd72be9b37294ba6" Nov 03 03:24:40.747384 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:24:40.747347 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bec66f72f6ee86d7a65dbbae33ac1de222255865fa782038fd72be9b37294ba6\": container with ID starting with bec66f72f6ee86d7a65dbbae33ac1de222255865fa782038fd72be9b37294ba6 not found: ID does not exist" containerID="bec66f72f6ee86d7a65dbbae33ac1de222255865fa782038fd72be9b37294ba6" Nov 03 03:24:40.747475 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:40.747396 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bec66f72f6ee86d7a65dbbae33ac1de222255865fa782038fd72be9b37294ba6"} err="failed to get container status \"bec66f72f6ee86d7a65dbbae33ac1de222255865fa782038fd72be9b37294ba6\": rpc error: code = NotFound desc = could not find container \"bec66f72f6ee86d7a65dbbae33ac1de222255865fa782038fd72be9b37294ba6\": container with ID starting with bec66f72f6ee86d7a65dbbae33ac1de222255865fa782038fd72be9b37294ba6 not found: ID does not exist" Nov 03 03:24:40.890328 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:40.890299 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-e1520875-kserve-757cbb86b9-7gq6c"] Nov 03 03:24:40.937017 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:40.936992 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-e1520875-kserve-757cbb86b9-7gq6c"] Nov 03 03:24:42.848436 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:42.848394 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6" path="/var/lib/kubelet/pods/9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6/volumes" Nov 03 03:24:42.944844 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:42.944814 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:24:46.769795 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:46.769751 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-e1520875-kserve-fc9f8bcc-nh69b" podUID="58dcb314-1d08-41a7-82ce-18bac7d3ddcf" containerName="main" containerID="cri-o://e197e208c62767274f01432531f7cdddfb3f2bdda7d0ca7b83dabf4d5730e0ac" gracePeriod=45 Nov 03 03:24:47.009182 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:47.009157 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e1520875-kserve-fc9f8bcc-nh69b" Nov 03 03:24:47.129473 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:47.129441 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58dcb314-1d08-41a7-82ce-18bac7d3ddcf-tls-certs\" (UniqueName: \"kubernetes.io/secret/58dcb314-1d08-41a7-82ce-18bac7d3ddcf-tls-certs\") pod \"58dcb314-1d08-41a7-82ce-18bac7d3ddcf\" (UID: \"58dcb314-1d08-41a7-82ce-18bac7d3ddcf\") " Nov 03 03:24:47.129636 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:47.129508 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58dcb314-1d08-41a7-82ce-18bac7d3ddcf-home\" (UniqueName: \"kubernetes.io/empty-dir/58dcb314-1d08-41a7-82ce-18bac7d3ddcf-home\") pod \"58dcb314-1d08-41a7-82ce-18bac7d3ddcf\" (UID: \"58dcb314-1d08-41a7-82ce-18bac7d3ddcf\") " Nov 03 03:24:47.129636 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:47.129577 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58dcb314-1d08-41a7-82ce-18bac7d3ddcf-kube-api-access-vd9lc\" (UniqueName: \"kubernetes.io/projected/58dcb314-1d08-41a7-82ce-18bac7d3ddcf-kube-api-access-vd9lc\") pod \"58dcb314-1d08-41a7-82ce-18bac7d3ddcf\" (UID: \"58dcb314-1d08-41a7-82ce-18bac7d3ddcf\") " Nov 03 03:24:47.129734 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:47.129655 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58dcb314-1d08-41a7-82ce-18bac7d3ddcf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58dcb314-1d08-41a7-82ce-18bac7d3ddcf-tmp-dir\") pod \"58dcb314-1d08-41a7-82ce-18bac7d3ddcf\" (UID: \"58dcb314-1d08-41a7-82ce-18bac7d3ddcf\") " Nov 03 03:24:47.129734 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:47.129697 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58dcb314-1d08-41a7-82ce-18bac7d3ddcf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58dcb314-1d08-41a7-82ce-18bac7d3ddcf-model-cache\") pod \"58dcb314-1d08-41a7-82ce-18bac7d3ddcf\" (UID: \"58dcb314-1d08-41a7-82ce-18bac7d3ddcf\") " Nov 03 03:24:47.129841 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:47.129737 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58dcb314-1d08-41a7-82ce-18bac7d3ddcf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/58dcb314-1d08-41a7-82ce-18bac7d3ddcf-dshm\") pod \"58dcb314-1d08-41a7-82ce-18bac7d3ddcf\" (UID: \"58dcb314-1d08-41a7-82ce-18bac7d3ddcf\") " Nov 03 03:24:47.129841 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:47.129746 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58dcb314-1d08-41a7-82ce-18bac7d3ddcf-home" pod "58dcb314-1d08-41a7-82ce-18bac7d3ddcf" (UID: "58dcb314-1d08-41a7-82ce-18bac7d3ddcf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:24:47.130021 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:47.129952 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58dcb314-1d08-41a7-82ce-18bac7d3ddcf-tmp-dir" pod "58dcb314-1d08-41a7-82ce-18bac7d3ddcf" (UID: "58dcb314-1d08-41a7-82ce-18bac7d3ddcf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:24:47.130021 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:47.130007 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58dcb314-1d08-41a7-82ce-18bac7d3ddcf-model-cache" pod "58dcb314-1d08-41a7-82ce-18bac7d3ddcf" (UID: "58dcb314-1d08-41a7-82ce-18bac7d3ddcf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:24:47.130184 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:47.130166 5605 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58dcb314-1d08-41a7-82ce-18bac7d3ddcf-model-cache\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:24:47.130264 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:47.130188 5605 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58dcb314-1d08-41a7-82ce-18bac7d3ddcf-home\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:24:47.130264 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:47.130202 5605 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58dcb314-1d08-41a7-82ce-18bac7d3ddcf-tmp-dir\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:24:47.131661 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:47.131635 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58dcb314-1d08-41a7-82ce-18bac7d3ddcf-kube-api-access-vd9lc" pod "58dcb314-1d08-41a7-82ce-18bac7d3ddcf" (UID: "58dcb314-1d08-41a7-82ce-18bac7d3ddcf"). InnerVolumeSpecName "kube-api-access-vd9lc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:24:47.131762 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:47.131637 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58dcb314-1d08-41a7-82ce-18bac7d3ddcf-tls-certs" pod "58dcb314-1d08-41a7-82ce-18bac7d3ddcf" (UID: "58dcb314-1d08-41a7-82ce-18bac7d3ddcf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:24:47.132089 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:47.132074 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58dcb314-1d08-41a7-82ce-18bac7d3ddcf-dshm" pod "58dcb314-1d08-41a7-82ce-18bac7d3ddcf" (UID: "58dcb314-1d08-41a7-82ce-18bac7d3ddcf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:24:47.231262 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:47.231233 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd9lc\" (UniqueName: \"kubernetes.io/projected/58dcb314-1d08-41a7-82ce-18bac7d3ddcf-kube-api-access-vd9lc\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:24:47.231262 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:47.231258 5605 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58dcb314-1d08-41a7-82ce-18bac7d3ddcf-dshm\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:24:47.231262 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:47.231267 5605 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58dcb314-1d08-41a7-82ce-18bac7d3ddcf-tls-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:24:47.770223 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:47.770186 5605 generic.go:356] "Generic (PLEG): container finished" podID="58dcb314-1d08-41a7-82ce-18bac7d3ddcf" containerID="e197e208c62767274f01432531f7cdddfb3f2bdda7d0ca7b83dabf4d5730e0ac" exitCode=0 Nov 03 03:24:47.770668 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:47.770232 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e1520875-kserve-fc9f8bcc-nh69b" Nov 03 03:24:47.770668 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:47.770261 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e1520875-kserve-fc9f8bcc-nh69b" event={"ID":"58dcb314-1d08-41a7-82ce-18bac7d3ddcf","Type":"ContainerDied","Data":"e197e208c62767274f01432531f7cdddfb3f2bdda7d0ca7b83dabf4d5730e0ac"} Nov 03 03:24:47.770668 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:47.770307 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e1520875-kserve-fc9f8bcc-nh69b" event={"ID":"58dcb314-1d08-41a7-82ce-18bac7d3ddcf","Type":"ContainerDied","Data":"00ba492bc17ad75d2153eb5d619405f4490c60664cc66ef667d57463aa904c7d"} Nov 03 03:24:47.770668 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:47.770325 5605 scope.go:122] "RemoveContainer" containerID="e197e208c62767274f01432531f7cdddfb3f2bdda7d0ca7b83dabf4d5730e0ac" Nov 03 03:24:47.778843 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:47.778763 5605 scope.go:122] "RemoveContainer" containerID="e197e208c62767274f01432531f7cdddfb3f2bdda7d0ca7b83dabf4d5730e0ac" Nov 03 03:24:47.779062 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:24:47.779043 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e197e208c62767274f01432531f7cdddfb3f2bdda7d0ca7b83dabf4d5730e0ac\": container with ID starting with e197e208c62767274f01432531f7cdddfb3f2bdda7d0ca7b83dabf4d5730e0ac not found: ID does not exist" containerID="e197e208c62767274f01432531f7cdddfb3f2bdda7d0ca7b83dabf4d5730e0ac" Nov 03 03:24:47.779120 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:47.779070 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e197e208c62767274f01432531f7cdddfb3f2bdda7d0ca7b83dabf4d5730e0ac"} err="failed to get container status \"e197e208c62767274f01432531f7cdddfb3f2bdda7d0ca7b83dabf4d5730e0ac\": rpc error: code = NotFound desc = could not find container \"e197e208c62767274f01432531f7cdddfb3f2bdda7d0ca7b83dabf4d5730e0ac\": container with ID starting with e197e208c62767274f01432531f7cdddfb3f2bdda7d0ca7b83dabf4d5730e0ac not found: ID does not exist" Nov 03 03:24:47.922558 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:47.922530 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-e1520875-kserve-fc9f8bcc-nh69b"] Nov 03 03:24:47.969173 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:47.969147 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-e1520875-kserve-fc9f8bcc-nh69b"] Nov 03 03:24:48.848180 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:48.848146 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58dcb314-1d08-41a7-82ce-18bac7d3ddcf" path="/var/lib/kubelet/pods/58dcb314-1d08-41a7-82ce-18bac7d3ddcf/volumes" Nov 03 03:24:53.911332 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:24:53.911298 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:24:54.134595 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:54.134564 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:24:56.047115 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:56.047083 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f24d4143-zl2b6"] Nov 03 03:24:56.094786 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:56.094739 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f24d4143-zl2b6"] Nov 03 03:24:56.868380 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:56.865604 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8730b0b5-e304-4b23-89a1-73af5ccd0e60" path="/var/lib/kubelet/pods/8730b0b5-e304-4b23-89a1-73af5ccd0e60/volumes" Nov 03 03:24:57.303493 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:57.303461 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-p8fqq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 03:24:57.304075 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:57.304040 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p8fqq" podUID="0bedb6e8-5829-4cc2-a912-35131c941e09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 03:24:57.316473 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:57.316445 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-dbd7q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 03:24:57.316690 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:24:57.316666 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dbd7q" podUID="dc490604-2e00-49dd-8423-47fc70709678" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 03:25:12.152044 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:25:12.151959 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:25:15.042192 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:25:15.042157 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-622d7413-6lpd7"] Nov 03 03:25:15.088596 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:25:15.088565 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-622d7413-6lpd7"] Nov 03 03:25:16.849293 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:25:16.849255 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04712f50-7574-4123-8009-984532ac7a65" path="/var/lib/kubelet/pods/04712f50-7574-4123-8009-984532ac7a65/volumes" Nov 03 03:25:23.913852 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:25:23.913807 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:25:45.362566 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:25:45.362531 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-659c975df4-4wn4f"] Nov 03 03:25:45.363096 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:25:45.362857 5605 cpu_manager.go:414] "containerMap: removing container" podUID="58dcb314-1d08-41a7-82ce-18bac7d3ddcf" containerName="main" Nov 03 03:25:45.363096 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:25:45.362869 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58dcb314-1d08-41a7-82ce-18bac7d3ddcf" containerName="main" Nov 03 03:25:45.363096 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:25:45.362886 5605 cpu_manager.go:414] "containerMap: removing container" podUID="9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6" containerName="main" Nov 03 03:25:45.363096 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:25:45.362892 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6" containerName="main" Nov 03 03:25:45.363096 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:25:45.362968 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="58dcb314-1d08-41a7-82ce-18bac7d3ddcf" containerName="main" Nov 03 03:25:45.363096 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:25:45.362981 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b37bcbe-54de-4c63-a24c-8b7f4a6d43d6" containerName="main" Nov 03 03:25:45.365992 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:25:45.365970 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-659c975df4-4wn4f" Nov 03 03:25:45.461201 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:25:45.461168 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-lxnf7\"" Nov 03 03:25:45.461201 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:25:45.461171 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 03 03:25:45.462434 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:25:45.462413 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-659c975df4-4wn4f"] Nov 03 03:25:45.509664 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:25:45.509636 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3344d6d8-48b8-4aa5-afba-b7e9b504b5ee-plugins-config-volume\") pod \"payload-pre-processing-659c975df4-4wn4f\" (UID: \"3344d6d8-48b8-4aa5-afba-b7e9b504b5ee\") " pod="openshift-ingress/payload-pre-processing-659c975df4-4wn4f" Nov 03 03:25:45.509796 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:25:45.509736 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxf2s\" (UniqueName: \"kubernetes.io/projected/3344d6d8-48b8-4aa5-afba-b7e9b504b5ee-kube-api-access-jxf2s\") pod \"payload-pre-processing-659c975df4-4wn4f\" (UID: \"3344d6d8-48b8-4aa5-afba-b7e9b504b5ee\") " pod="openshift-ingress/payload-pre-processing-659c975df4-4wn4f" Nov 03 03:25:45.610124 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:25:45.610098 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxf2s\" (UniqueName: \"kubernetes.io/projected/3344d6d8-48b8-4aa5-afba-b7e9b504b5ee-kube-api-access-jxf2s\") pod \"payload-pre-processing-659c975df4-4wn4f\" (UID: \"3344d6d8-48b8-4aa5-afba-b7e9b504b5ee\") " pod="openshift-ingress/payload-pre-processing-659c975df4-4wn4f" Nov 03 03:25:45.610256 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:25:45.610158 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3344d6d8-48b8-4aa5-afba-b7e9b504b5ee-plugins-config-volume\") pod \"payload-pre-processing-659c975df4-4wn4f\" (UID: \"3344d6d8-48b8-4aa5-afba-b7e9b504b5ee\") " pod="openshift-ingress/payload-pre-processing-659c975df4-4wn4f" Nov 03 03:25:45.610805 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:25:45.610788 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3344d6d8-48b8-4aa5-afba-b7e9b504b5ee-plugins-config-volume\") pod \"payload-pre-processing-659c975df4-4wn4f\" (UID: \"3344d6d8-48b8-4aa5-afba-b7e9b504b5ee\") " pod="openshift-ingress/payload-pre-processing-659c975df4-4wn4f" Nov 03 03:25:45.663702 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:25:45.663652 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxf2s\" (UniqueName: \"kubernetes.io/projected/3344d6d8-48b8-4aa5-afba-b7e9b504b5ee-kube-api-access-jxf2s\") pod \"payload-pre-processing-659c975df4-4wn4f\" (UID: \"3344d6d8-48b8-4aa5-afba-b7e9b504b5ee\") " pod="openshift-ingress/payload-pre-processing-659c975df4-4wn4f" Nov 03 03:25:45.678902 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:25:45.678883 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-659c975df4-4wn4f" Nov 03 03:25:46.041088 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:25:46.041014 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-659c975df4-4wn4f"] Nov 03 03:25:46.047526 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:25:46.047499 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3344d6d8_48b8_4aa5_afba_b7e9b504b5ee.slice/crio-c6e5876e3447d69393b17bd4bde95479ae58f2cec6174b76020b910f9d99475c WatchSource:0}: Error finding container c6e5876e3447d69393b17bd4bde95479ae58f2cec6174b76020b910f9d99475c: Status 404 returned error can't find the container with id c6e5876e3447d69393b17bd4bde95479ae58f2cec6174b76020b910f9d99475c Nov 03 03:25:46.103558 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:25:46.103530 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-659c975df4-4wn4f" event={"ID":"3344d6d8-48b8-4aa5-afba-b7e9b504b5ee","Type":"ContainerStarted","Data":"c6e5876e3447d69393b17bd4bde95479ae58f2cec6174b76020b910f9d99475c"} Nov 03 03:25:47.108348 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:25:47.108312 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-659c975df4-4wn4f" event={"ID":"3344d6d8-48b8-4aa5-afba-b7e9b504b5ee","Type":"ContainerStarted","Data":"242bcf3b72abf1e88b8697bf277977a652cf0c043334f782d69bec70f1054231"} Nov 03 03:25:47.108788 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:25:47.108519 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-659c975df4-4wn4f" Nov 03 03:25:47.217252 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:25:47.217211 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-659c975df4-4wn4f" podStartSLOduration=2.21720016 podStartE2EDuration="2.21720016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:25:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:25:47.216554182 +0000 UTC m=+5624.475752336" watchObservedRunningTime="2026-11-03 03:25:47.21720016 +0000 UTC m=+5624.476398314" Nov 03 03:25:53.910944 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:25:53.910899 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:25:58.115345 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:25:58.115311 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-659c975df4-4wn4f" Nov 03 03:26:05.545997 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:26:05.545962 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:26:05.817273 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:26:05.817244 5605 scope.go:122] "RemoveContainer" containerID="a94ca6801a8efca54dfb6ee1792d3bedc70c59860430a54259d10e7d8eeef586" Nov 03 03:26:23.911120 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:26:23.911079 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:26:53.912651 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:26:53.912568 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:27:05.835256 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:27:05.835221 5605 scope.go:122] "RemoveContainer" containerID="4942433aecc2bc9027069a3f7a80234698e9996071ab549809fba3e30c013014" Nov 03 03:27:23.912592 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:27:23.912551 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:27:47.879060 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:27:47.879025 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-m9fhr"] Nov 03 03:27:47.882508 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:27:47.882490 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-m9fhr" Nov 03 03:27:47.915833 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:27:47.915803 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33e2648a-90d7-4a75-bc13-6ba8a4bf9784-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-m9fhr\" (UID: \"33e2648a-90d7-4a75-bc13-6ba8a4bf9784\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-m9fhr" Nov 03 03:27:47.915965 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:27:47.915865 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcwq4\" (UniqueName: \"kubernetes.io/projected/33e2648a-90d7-4a75-bc13-6ba8a4bf9784-kube-api-access-gcwq4\") pod \"maas-api-revoke-sub-e2547622-m9fhr\" (UID: \"33e2648a-90d7-4a75-bc13-6ba8a4bf9784\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-m9fhr" Nov 03 03:27:47.915965 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:27:47.915942 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33e2648a-90d7-4a75-bc13-6ba8a4bf9784-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-m9fhr\" (UID: \"33e2648a-90d7-4a75-bc13-6ba8a4bf9784\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-m9fhr" Nov 03 03:27:47.933712 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:27:47.933690 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:27:47.933805 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:27:47.933789 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:27:47.980803 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:27:47.980772 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:27:47.980950 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:27:47.980884 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sb4qh\"" Nov 03 03:27:47.981938 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:27:47.981917 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-m9fhr"] Nov 03 03:27:48.017119 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:27:48.017094 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33e2648a-90d7-4a75-bc13-6ba8a4bf9784-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-m9fhr\" (UID: \"33e2648a-90d7-4a75-bc13-6ba8a4bf9784\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-m9fhr" Nov 03 03:27:48.017209 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:27:48.017175 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcwq4\" (UniqueName: \"kubernetes.io/projected/33e2648a-90d7-4a75-bc13-6ba8a4bf9784-kube-api-access-gcwq4\") pod \"maas-api-revoke-sub-e2547622-m9fhr\" (UID: \"33e2648a-90d7-4a75-bc13-6ba8a4bf9784\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-m9fhr" Nov 03 03:27:48.017258 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:27:48.017217 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33e2648a-90d7-4a75-bc13-6ba8a4bf9784-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-m9fhr\" (UID: \"33e2648a-90d7-4a75-bc13-6ba8a4bf9784\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-m9fhr" Nov 03 03:27:48.017899 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:27:48.017877 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33e2648a-90d7-4a75-bc13-6ba8a4bf9784-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-m9fhr\" (UID: \"33e2648a-90d7-4a75-bc13-6ba8a4bf9784\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-m9fhr" Nov 03 03:27:48.019669 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:27:48.019645 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33e2648a-90d7-4a75-bc13-6ba8a4bf9784-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-m9fhr\" (UID: \"33e2648a-90d7-4a75-bc13-6ba8a4bf9784\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-m9fhr" Nov 03 03:27:48.072009 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:27:48.071987 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcwq4\" (UniqueName: \"kubernetes.io/projected/33e2648a-90d7-4a75-bc13-6ba8a4bf9784-kube-api-access-gcwq4\") pod \"maas-api-revoke-sub-e2547622-m9fhr\" (UID: \"33e2648a-90d7-4a75-bc13-6ba8a4bf9784\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-m9fhr" Nov 03 03:27:48.194509 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:27:48.194441 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-m9fhr" Nov 03 03:27:48.559679 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:27:48.559648 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-m9fhr"] Nov 03 03:27:48.569536 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:27:48.569096 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33e2648a_90d7_4a75_bc13_6ba8a4bf9784.slice/crio-a2d3a0e4decf66fd7b1cd7703745030ad1b965d719e26ca008fb6cbcc24fa212 WatchSource:0}: Error finding container a2d3a0e4decf66fd7b1cd7703745030ad1b965d719e26ca008fb6cbcc24fa212: Status 404 returned error can't find the container with id a2d3a0e4decf66fd7b1cd7703745030ad1b965d719e26ca008fb6cbcc24fa212 Nov 03 03:27:48.662913 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:27:48.662876 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-m9fhr" event={"ID":"33e2648a-90d7-4a75-bc13-6ba8a4bf9784","Type":"ContainerStarted","Data":"a2d3a0e4decf66fd7b1cd7703745030ad1b965d719e26ca008fb6cbcc24fa212"} Nov 03 03:27:49.669654 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:27:49.669561 5605 generic.go:356] "Generic (PLEG): container finished" podID="33e2648a-90d7-4a75-bc13-6ba8a4bf9784" containerID="f7ea77977128a255be23d720b8b1fa6e84d4ab94aa9e960608b516586d7cff60" exitCode=0 Nov 03 03:27:49.669654 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:27:49.669632 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-m9fhr" event={"ID":"33e2648a-90d7-4a75-bc13-6ba8a4bf9784","Type":"ContainerDied","Data":"f7ea77977128a255be23d720b8b1fa6e84d4ab94aa9e960608b516586d7cff60"} Nov 03 03:27:50.798531 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:27:50.798505 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-m9fhr" Nov 03 03:27:50.843136 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:27:50.843105 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33e2648a-90d7-4a75-bc13-6ba8a4bf9784-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33e2648a-90d7-4a75-bc13-6ba8a4bf9784-maas-api-openshift-ca\") pod \"33e2648a-90d7-4a75-bc13-6ba8a4bf9784\" (UID: \"33e2648a-90d7-4a75-bc13-6ba8a4bf9784\") " Nov 03 03:27:50.843268 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:27:50.843219 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33e2648a-90d7-4a75-bc13-6ba8a4bf9784-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33e2648a-90d7-4a75-bc13-6ba8a4bf9784-maas-api-xks-ca\") pod \"33e2648a-90d7-4a75-bc13-6ba8a4bf9784\" (UID: \"33e2648a-90d7-4a75-bc13-6ba8a4bf9784\") " Nov 03 03:27:50.843268 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:27:50.843249 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33e2648a-90d7-4a75-bc13-6ba8a4bf9784-kube-api-access-gcwq4\" (UniqueName: \"kubernetes.io/projected/33e2648a-90d7-4a75-bc13-6ba8a4bf9784-kube-api-access-gcwq4\") pod \"33e2648a-90d7-4a75-bc13-6ba8a4bf9784\" (UID: \"33e2648a-90d7-4a75-bc13-6ba8a4bf9784\") " Nov 03 03:27:50.843497 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:27:50.843476 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33e2648a-90d7-4a75-bc13-6ba8a4bf9784-maas-api-openshift-ca" pod "33e2648a-90d7-4a75-bc13-6ba8a4bf9784" (UID: "33e2648a-90d7-4a75-bc13-6ba8a4bf9784"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:27:50.845243 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:27:50.845220 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33e2648a-90d7-4a75-bc13-6ba8a4bf9784-maas-api-xks-ca" pod "33e2648a-90d7-4a75-bc13-6ba8a4bf9784" (UID: "33e2648a-90d7-4a75-bc13-6ba8a4bf9784"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:27:50.845377 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:27:50.845296 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33e2648a-90d7-4a75-bc13-6ba8a4bf9784-kube-api-access-gcwq4" pod "33e2648a-90d7-4a75-bc13-6ba8a4bf9784" (UID: "33e2648a-90d7-4a75-bc13-6ba8a4bf9784"). InnerVolumeSpecName "kube-api-access-gcwq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:27:50.943923 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:27:50.943857 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33e2648a-90d7-4a75-bc13-6ba8a4bf9784-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:27:50.943923 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:27:50.943885 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33e2648a-90d7-4a75-bc13-6ba8a4bf9784-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:27:50.943923 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:27:50.943899 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcwq4\" (UniqueName: \"kubernetes.io/projected/33e2648a-90d7-4a75-bc13-6ba8a4bf9784-kube-api-access-gcwq4\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:27:51.679580 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:27:51.679541 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-m9fhr" event={"ID":"33e2648a-90d7-4a75-bc13-6ba8a4bf9784","Type":"ContainerDied","Data":"a2d3a0e4decf66fd7b1cd7703745030ad1b965d719e26ca008fb6cbcc24fa212"} Nov 03 03:27:51.679580 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:27:51.679564 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-m9fhr" Nov 03 03:27:51.679853 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:27:51.679589 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2d3a0e4decf66fd7b1cd7703745030ad1b965d719e26ca008fb6cbcc24fa212" Nov 03 03:27:53.910848 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:27:53.910807 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:28:02.341423 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:28:02.341391 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:28:12.673574 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:28:12.673483 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:28:23.912420 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:28:23.912378 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:28:31.153701 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:28:31.153667 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:28:41.537896 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:28:41.537849 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:28:53.914876 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:28:53.914826 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:28:55.342168 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:28:55.342133 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:29:05.859998 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:29:05.859965 5605 scope.go:122] "RemoveContainer" containerID="a686a7b270d8c81965607502a89e4b1f908dcfa9da3758a6312b71bd3c88edc9" Nov 03 03:29:12.222721 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:29:12.222675 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:29:23.915176 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:29:23.915132 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:29:43.164127 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:29:43.164025 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:29:53.913855 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:29:53.913809 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:29:53.928179 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:29:53.928150 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:30:05.916849 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:30:05.916813 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:30:17.247771 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:30:17.247739 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sbhx6"] Nov 03 03:30:17.248249 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:30:17.248066 5605 cpu_manager.go:414] "containerMap: removing container" podUID="33e2648a-90d7-4a75-bc13-6ba8a4bf9784" containerName="revoke-keys" Nov 03 03:30:17.248249 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:30:17.248077 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33e2648a-90d7-4a75-bc13-6ba8a4bf9784" containerName="revoke-keys" Nov 03 03:30:17.248249 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:30:17.248154 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="33e2648a-90d7-4a75-bc13-6ba8a4bf9784" containerName="revoke-keys" Nov 03 03:30:17.251212 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:30:17.251189 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sbhx6" Nov 03 03:30:17.300515 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:30:17.300485 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:30:17.346874 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:30:17.346843 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:30:17.347043 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:30:17.346882 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:30:17.347113 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:30:17.347076 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sb4qh\"" Nov 03 03:30:17.348292 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:30:17.348269 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sbhx6"] Nov 03 03:30:17.375082 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:30:17.375056 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltxlw\" (UniqueName: \"kubernetes.io/projected/488f5888-a1ce-4f19-954d-06c7deec58b2-kube-api-access-ltxlw\") pod \"maas-api-revoke-sub-ae594979-sbhx6\" (UID: \"488f5888-a1ce-4f19-954d-06c7deec58b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sbhx6" Nov 03 03:30:17.375190 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:30:17.375105 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/488f5888-a1ce-4f19-954d-06c7deec58b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-sbhx6\" (UID: \"488f5888-a1ce-4f19-954d-06c7deec58b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sbhx6" Nov 03 03:30:17.375238 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:30:17.375213 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/488f5888-a1ce-4f19-954d-06c7deec58b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-sbhx6\" (UID: \"488f5888-a1ce-4f19-954d-06c7deec58b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sbhx6" Nov 03 03:30:17.476222 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:30:17.476198 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltxlw\" (UniqueName: \"kubernetes.io/projected/488f5888-a1ce-4f19-954d-06c7deec58b2-kube-api-access-ltxlw\") pod \"maas-api-revoke-sub-ae594979-sbhx6\" (UID: \"488f5888-a1ce-4f19-954d-06c7deec58b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sbhx6" Nov 03 03:30:17.476343 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:30:17.476232 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/488f5888-a1ce-4f19-954d-06c7deec58b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-sbhx6\" (UID: \"488f5888-a1ce-4f19-954d-06c7deec58b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sbhx6" Nov 03 03:30:17.476343 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:30:17.476275 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/488f5888-a1ce-4f19-954d-06c7deec58b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-sbhx6\" (UID: \"488f5888-a1ce-4f19-954d-06c7deec58b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sbhx6" Nov 03 03:30:17.477025 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:30:17.476999 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/488f5888-a1ce-4f19-954d-06c7deec58b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-sbhx6\" (UID: \"488f5888-a1ce-4f19-954d-06c7deec58b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sbhx6" Nov 03 03:30:17.478553 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:30:17.478535 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/488f5888-a1ce-4f19-954d-06c7deec58b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-sbhx6\" (UID: \"488f5888-a1ce-4f19-954d-06c7deec58b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sbhx6" Nov 03 03:30:17.529585 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:30:17.529522 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltxlw\" (UniqueName: \"kubernetes.io/projected/488f5888-a1ce-4f19-954d-06c7deec58b2-kube-api-access-ltxlw\") pod \"maas-api-revoke-sub-ae594979-sbhx6\" (UID: \"488f5888-a1ce-4f19-954d-06c7deec58b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sbhx6" Nov 03 03:30:17.563953 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:30:17.563932 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sbhx6" Nov 03 03:30:17.926784 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:30:17.926751 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sbhx6"] Nov 03 03:30:17.932737 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:30:17.932705 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod488f5888_a1ce_4f19_954d_06c7deec58b2.slice/crio-212a552c294bdc04ef53948ba1bf5ca775d490bc5c63b78374a3208f721a2d36 WatchSource:0}: Error finding container 212a552c294bdc04ef53948ba1bf5ca775d490bc5c63b78374a3208f721a2d36: Status 404 returned error can't find the container with id 212a552c294bdc04ef53948ba1bf5ca775d490bc5c63b78374a3208f721a2d36 Nov 03 03:30:18.437610 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:30:18.437582 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sbhx6" event={"ID":"488f5888-a1ce-4f19-954d-06c7deec58b2","Type":"ContainerStarted","Data":"fbbc241dfcf77d99f538b5b933ef07452893aeec3c607be8ce09c66839e26d55"} Nov 03 03:30:18.437924 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:30:18.437638 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sbhx6" event={"ID":"488f5888-a1ce-4f19-954d-06c7deec58b2","Type":"ContainerStarted","Data":"212a552c294bdc04ef53948ba1bf5ca775d490bc5c63b78374a3208f721a2d36"} Nov 03 03:30:18.544948 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:30:18.544875 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sbhx6" podStartSLOduration=1.5448628229999999 podStartE2EDuration="1.544862823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:30:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:30:18.543440536 +0000 UTC m=+5895.802638691" watchObservedRunningTime="2026-11-03 03:30:18.544862823 +0000 UTC m=+5895.804060977" Nov 03 03:30:19.443594 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:30:19.443562 5605 generic.go:356] "Generic (PLEG): container finished" podID="488f5888-a1ce-4f19-954d-06c7deec58b2" containerID="fbbc241dfcf77d99f538b5b933ef07452893aeec3c607be8ce09c66839e26d55" exitCode=0 Nov 03 03:30:19.443966 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:30:19.443611 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sbhx6" event={"ID":"488f5888-a1ce-4f19-954d-06c7deec58b2","Type":"ContainerDied","Data":"fbbc241dfcf77d99f538b5b933ef07452893aeec3c607be8ce09c66839e26d55"} Nov 03 03:30:20.577816 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:30:20.577789 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sbhx6" Nov 03 03:30:20.707145 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:30:20.707076 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/488f5888-a1ce-4f19-954d-06c7deec58b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/488f5888-a1ce-4f19-954d-06c7deec58b2-maas-api-openshift-ca\") pod \"488f5888-a1ce-4f19-954d-06c7deec58b2\" (UID: \"488f5888-a1ce-4f19-954d-06c7deec58b2\") " Nov 03 03:30:20.707145 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:30:20.707138 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/488f5888-a1ce-4f19-954d-06c7deec58b2-kube-api-access-ltxlw\" (UniqueName: \"kubernetes.io/projected/488f5888-a1ce-4f19-954d-06c7deec58b2-kube-api-access-ltxlw\") pod \"488f5888-a1ce-4f19-954d-06c7deec58b2\" (UID: \"488f5888-a1ce-4f19-954d-06c7deec58b2\") " Nov 03 03:30:20.707393 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:30:20.707224 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/488f5888-a1ce-4f19-954d-06c7deec58b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/488f5888-a1ce-4f19-954d-06c7deec58b2-maas-api-xks-ca\") pod \"488f5888-a1ce-4f19-954d-06c7deec58b2\" (UID: \"488f5888-a1ce-4f19-954d-06c7deec58b2\") " Nov 03 03:30:20.707539 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:30:20.707516 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/488f5888-a1ce-4f19-954d-06c7deec58b2-maas-api-openshift-ca" pod "488f5888-a1ce-4f19-954d-06c7deec58b2" (UID: "488f5888-a1ce-4f19-954d-06c7deec58b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:30:20.707624 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:30:20.707607 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/488f5888-a1ce-4f19-954d-06c7deec58b2-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:30:20.709791 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:30:20.709766 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/488f5888-a1ce-4f19-954d-06c7deec58b2-maas-api-xks-ca" pod "488f5888-a1ce-4f19-954d-06c7deec58b2" (UID: "488f5888-a1ce-4f19-954d-06c7deec58b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:30:20.709791 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:30:20.709778 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/488f5888-a1ce-4f19-954d-06c7deec58b2-kube-api-access-ltxlw" pod "488f5888-a1ce-4f19-954d-06c7deec58b2" (UID: "488f5888-a1ce-4f19-954d-06c7deec58b2"). InnerVolumeSpecName "kube-api-access-ltxlw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:30:20.808710 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:30:20.808676 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltxlw\" (UniqueName: \"kubernetes.io/projected/488f5888-a1ce-4f19-954d-06c7deec58b2-kube-api-access-ltxlw\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:30:20.808710 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:30:20.808708 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/488f5888-a1ce-4f19-954d-06c7deec58b2-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:30:21.453905 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:30:21.453873 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sbhx6" Nov 03 03:30:21.454096 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:30:21.453917 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sbhx6" event={"ID":"488f5888-a1ce-4f19-954d-06c7deec58b2","Type":"ContainerDied","Data":"212a552c294bdc04ef53948ba1bf5ca775d490bc5c63b78374a3208f721a2d36"} Nov 03 03:30:21.454096 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:30:21.453969 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="212a552c294bdc04ef53948ba1bf5ca775d490bc5c63b78374a3208f721a2d36" Nov 03 03:30:23.911226 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:30:23.911187 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:30:29.924898 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:30:29.924862 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:30:39.135114 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:30:39.135079 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:30:47.154607 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:30:47.154561 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:30:53.912124 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:30:53.912025 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:30:55.754428 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:30:55.754386 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:30:58.174316 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:30:58.174271 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-k449x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 03:30:58.174984 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:30:58.174950 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k449x" podUID="0378fcbc-4027-4275-b274-2c188f3ff442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 03:31:04.737756 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:31:04.737723 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:31:13.517929 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:31:13.517847 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:31:22.525196 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:31:22.525149 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:31:23.912125 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:31:23.912082 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:31:40.620173 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:31:40.620138 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:31:43.169535 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:31:43.169487 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-k449x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 03:31:43.170215 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:31:43.170163 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k449x" podUID="0378fcbc-4027-4275-b274-2c188f3ff442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 03:31:53.911597 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:31:53.911559 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:32:11.954030 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:32:11.953994 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:32:23.911821 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:32:23.911779 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:32:32.247134 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:32:32.247094 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:32:43.035891 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:32:43.035811 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:32:51.041962 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:32:51.041922 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-m9fhr"] Nov 03 03:32:51.088487 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:32:51.088458 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-m9fhr"] Nov 03 03:32:52.849817 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:32:52.849780 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33e2648a-90d7-4a75-bc13-6ba8a4bf9784" path="/var/lib/kubelet/pods/33e2648a-90d7-4a75-bc13-6ba8a4bf9784/volumes" Nov 03 03:32:53.911440 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:32:53.911403 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:33:13.394500 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:33:13.394458 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4e5fe5-669549f59wq894"] Nov 03 03:33:13.395096 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:33:13.395065 5605 cpu_manager.go:414] "containerMap: removing container" podUID="488f5888-a1ce-4f19-954d-06c7deec58b2" containerName="revoke-keys" Nov 03 03:33:13.395096 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:33:13.395094 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="488f5888-a1ce-4f19-954d-06c7deec58b2" containerName="revoke-keys" Nov 03 03:33:13.395384 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:33:13.395204 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="488f5888-a1ce-4f19-954d-06c7deec58b2" containerName="revoke-keys" Nov 03 03:33:13.398969 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:33:13.398945 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e5fe5-669549f59wq894" Nov 03 03:33:13.494500 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:33:13.494455 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-41393613-dockercfg-pbdmv\"" Nov 03 03:33:13.494500 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:33:13.494466 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-4e5fe5\"" Nov 03 03:33:13.495739 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:33:13.495713 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4e5fe5-669549f59wq894"] Nov 03 03:33:13.511793 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:33:13.511765 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmw7p\" (UniqueName: \"kubernetes.io/projected/3629605f-0698-4839-920d-dc3ae1c18f88-kube-api-access-qmw7p\") pod \"payload-processing-e2e-worker-single-main-4e5fe5-669549f59wq894\" (UID: \"3629605f-0698-4839-920d-dc3ae1c18f88\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e5fe5-669549f59wq894" Nov 03 03:33:13.511939 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:33:13.511814 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3629605f-0698-4839-920d-dc3ae1c18f88-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-4e5fe5-669549f59wq894\" (UID: \"3629605f-0698-4839-920d-dc3ae1c18f88\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e5fe5-669549f59wq894" Nov 03 03:33:13.613290 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:33:13.613245 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3629605f-0698-4839-920d-dc3ae1c18f88-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-4e5fe5-669549f59wq894\" (UID: \"3629605f-0698-4839-920d-dc3ae1c18f88\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e5fe5-669549f59wq894" Nov 03 03:33:13.613504 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:33:13.613344 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmw7p\" (UniqueName: \"kubernetes.io/projected/3629605f-0698-4839-920d-dc3ae1c18f88-kube-api-access-qmw7p\") pod \"payload-processing-e2e-worker-single-main-4e5fe5-669549f59wq894\" (UID: \"3629605f-0698-4839-920d-dc3ae1c18f88\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e5fe5-669549f59wq894" Nov 03 03:33:13.614026 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:33:13.614006 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3629605f-0698-4839-920d-dc3ae1c18f88-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-4e5fe5-669549f59wq894\" (UID: \"3629605f-0698-4839-920d-dc3ae1c18f88\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e5fe5-669549f59wq894" Nov 03 03:33:13.667624 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:33:13.667541 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmw7p\" (UniqueName: \"kubernetes.io/projected/3629605f-0698-4839-920d-dc3ae1c18f88-kube-api-access-qmw7p\") pod \"payload-processing-e2e-worker-single-main-4e5fe5-669549f59wq894\" (UID: \"3629605f-0698-4839-920d-dc3ae1c18f88\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e5fe5-669549f59wq894" Nov 03 03:33:13.714406 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:33:13.714337 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e5fe5-669549f59wq894" Nov 03 03:33:14.079968 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:33:14.079935 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4e5fe5-669549f59wq894"] Nov 03 03:33:14.087291 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:33:14.087257 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3629605f_0698_4839_920d_dc3ae1c18f88.slice/crio-f5a8c476990b81926d44b8f51bcc7fd986d0715794ff7bd3c29189c22ab7f4ca WatchSource:0}: Error finding container f5a8c476990b81926d44b8f51bcc7fd986d0715794ff7bd3c29189c22ab7f4ca: Status 404 returned error can't find the container with id f5a8c476990b81926d44b8f51bcc7fd986d0715794ff7bd3c29189c22ab7f4ca Nov 03 03:33:14.303863 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:33:14.303823 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e5fe5-669549f59wq894" event={"ID":"3629605f-0698-4839-920d-dc3ae1c18f88","Type":"ContainerStarted","Data":"298a8a0ec158204921254ccbe0120b9ef5942e4e31f653c9243d9038a8128eb9"} Nov 03 03:33:14.304053 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:33:14.303874 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e5fe5-669549f59wq894" event={"ID":"3629605f-0698-4839-920d-dc3ae1c18f88","Type":"ContainerStarted","Data":"f5a8c476990b81926d44b8f51bcc7fd986d0715794ff7bd3c29189c22ab7f4ca"} Nov 03 03:33:14.304053 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:33:14.303918 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e5fe5-669549f59wq894" Nov 03 03:33:14.410736 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:33:14.410616 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e5fe5-669549f59wq894" podStartSLOduration=1.410601138 podStartE2EDuration="1.410601138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:33:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:33:14.410062925 +0000 UTC m=+6071.669261078" watchObservedRunningTime="2026-11-03 03:33:14.410601138 +0000 UTC m=+6071.669799292" Nov 03 03:33:23.728116 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:33:23.728077 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:33:23.911432 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:33:23.911379 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:33:25.311785 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:33:25.311753 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e5fe5-669549f59wq894" Nov 03 03:33:37.747570 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:33:37.747525 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-4e5fe5/simulator-single-main-4e5fe5-kserve-68fc86fc88-v2z5q"] Nov 03 03:33:37.751044 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:33:37.751020 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-4e5fe5/simulator-single-main-4e5fe5-kserve-68fc86fc88-v2z5q" Nov 03 03:33:37.840715 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:33:37.840678 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ad87a47-8fba-4592-a690-68b6e60ed8dc-tls-certs\") pod \"simulator-single-main-4e5fe5-kserve-68fc86fc88-v2z5q\" (UID: \"7ad87a47-8fba-4592-a690-68b6e60ed8dc\") " pod="e2e-models-e2e-worker-single-main-4e5fe5/simulator-single-main-4e5fe5-kserve-68fc86fc88-v2z5q" Nov 03 03:33:37.840898 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:33:37.840794 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ad87a47-8fba-4592-a690-68b6e60ed8dc-home\") pod \"simulator-single-main-4e5fe5-kserve-68fc86fc88-v2z5q\" (UID: \"7ad87a47-8fba-4592-a690-68b6e60ed8dc\") " pod="e2e-models-e2e-worker-single-main-4e5fe5/simulator-single-main-4e5fe5-kserve-68fc86fc88-v2z5q" Nov 03 03:33:37.840898 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:33:37.840831 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ad87a47-8fba-4592-a690-68b6e60ed8dc-dshm\") pod \"simulator-single-main-4e5fe5-kserve-68fc86fc88-v2z5q\" (UID: \"7ad87a47-8fba-4592-a690-68b6e60ed8dc\") " pod="e2e-models-e2e-worker-single-main-4e5fe5/simulator-single-main-4e5fe5-kserve-68fc86fc88-v2z5q" Nov 03 03:33:37.840985 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:33:37.840896 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ad87a47-8fba-4592-a690-68b6e60ed8dc-tmp-dir\") pod \"simulator-single-main-4e5fe5-kserve-68fc86fc88-v2z5q\" (UID: \"7ad87a47-8fba-4592-a690-68b6e60ed8dc\") " pod="e2e-models-e2e-worker-single-main-4e5fe5/simulator-single-main-4e5fe5-kserve-68fc86fc88-v2z5q" Nov 03 03:33:37.840985 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:33:37.840927 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ad87a47-8fba-4592-a690-68b6e60ed8dc-model-cache\") pod \"simulator-single-main-4e5fe5-kserve-68fc86fc88-v2z5q\" (UID: \"7ad87a47-8fba-4592-a690-68b6e60ed8dc\") " pod="e2e-models-e2e-worker-single-main-4e5fe5/simulator-single-main-4e5fe5-kserve-68fc86fc88-v2z5q" Nov 03 03:33:37.840985 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:33:37.840950 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn6cb\" (UniqueName: \"kubernetes.io/projected/7ad87a47-8fba-4592-a690-68b6e60ed8dc-kube-api-access-bn6cb\") pod \"simulator-single-main-4e5fe5-kserve-68fc86fc88-v2z5q\" (UID: \"7ad87a47-8fba-4592-a690-68b6e60ed8dc\") " pod="e2e-models-e2e-worker-single-main-4e5fe5/simulator-single-main-4e5fe5-kserve-68fc86fc88-v2z5q" Nov 03 03:33:37.846346 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:33:37.846321 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-4e5fe5\"/\"kube-root-ca.crt\"" Nov 03 03:33:37.846526 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:33:37.846352 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-4e5fe5\"/\"openshift-service-ca.crt\"" Nov 03 03:33:37.846526 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:33:37.846390 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-4e5fe5\"/\"default-dockercfg-rrjgh\"" Nov 03 03:33:37.846526 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:33:37.846416 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-4e5fe5\"/\"simulator-single-main-4e5fe5-kserve-self-signed-certs\"" Nov 03 03:33:37.847761 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:33:37.847741 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-4e5fe5/simulator-single-main-4e5fe5-kserve-68fc86fc88-v2z5q"] Nov 03 03:33:37.941802 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:33:37.941755 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ad87a47-8fba-4592-a690-68b6e60ed8dc-home\") pod \"simulator-single-main-4e5fe5-kserve-68fc86fc88-v2z5q\" (UID: \"7ad87a47-8fba-4592-a690-68b6e60ed8dc\") " pod="e2e-models-e2e-worker-single-main-4e5fe5/simulator-single-main-4e5fe5-kserve-68fc86fc88-v2z5q" Nov 03 03:33:37.941802 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:33:37.941802 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ad87a47-8fba-4592-a690-68b6e60ed8dc-dshm\") pod \"simulator-single-main-4e5fe5-kserve-68fc86fc88-v2z5q\" (UID: \"7ad87a47-8fba-4592-a690-68b6e60ed8dc\") " pod="e2e-models-e2e-worker-single-main-4e5fe5/simulator-single-main-4e5fe5-kserve-68fc86fc88-v2z5q" Nov 03 03:33:37.942092 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:33:37.941835 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ad87a47-8fba-4592-a690-68b6e60ed8dc-tmp-dir\") pod \"simulator-single-main-4e5fe5-kserve-68fc86fc88-v2z5q\" (UID: \"7ad87a47-8fba-4592-a690-68b6e60ed8dc\") " pod="e2e-models-e2e-worker-single-main-4e5fe5/simulator-single-main-4e5fe5-kserve-68fc86fc88-v2z5q" Nov 03 03:33:37.942092 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:33:37.941877 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ad87a47-8fba-4592-a690-68b6e60ed8dc-model-cache\") pod \"simulator-single-main-4e5fe5-kserve-68fc86fc88-v2z5q\" (UID: \"7ad87a47-8fba-4592-a690-68b6e60ed8dc\") " pod="e2e-models-e2e-worker-single-main-4e5fe5/simulator-single-main-4e5fe5-kserve-68fc86fc88-v2z5q" Nov 03 03:33:37.942092 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:33:37.941909 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn6cb\" (UniqueName: \"kubernetes.io/projected/7ad87a47-8fba-4592-a690-68b6e60ed8dc-kube-api-access-bn6cb\") pod \"simulator-single-main-4e5fe5-kserve-68fc86fc88-v2z5q\" (UID: \"7ad87a47-8fba-4592-a690-68b6e60ed8dc\") " pod="e2e-models-e2e-worker-single-main-4e5fe5/simulator-single-main-4e5fe5-kserve-68fc86fc88-v2z5q" Nov 03 03:33:37.942092 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:33:37.941960 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ad87a47-8fba-4592-a690-68b6e60ed8dc-tls-certs\") pod \"simulator-single-main-4e5fe5-kserve-68fc86fc88-v2z5q\" (UID: \"7ad87a47-8fba-4592-a690-68b6e60ed8dc\") " pod="e2e-models-e2e-worker-single-main-4e5fe5/simulator-single-main-4e5fe5-kserve-68fc86fc88-v2z5q" Nov 03 03:33:37.942340 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:33:37.942242 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ad87a47-8fba-4592-a690-68b6e60ed8dc-home\") pod \"simulator-single-main-4e5fe5-kserve-68fc86fc88-v2z5q\" (UID: \"7ad87a47-8fba-4592-a690-68b6e60ed8dc\") " pod="e2e-models-e2e-worker-single-main-4e5fe5/simulator-single-main-4e5fe5-kserve-68fc86fc88-v2z5q" Nov 03 03:33:37.942340 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:33:37.942303 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ad87a47-8fba-4592-a690-68b6e60ed8dc-model-cache\") pod \"simulator-single-main-4e5fe5-kserve-68fc86fc88-v2z5q\" (UID: \"7ad87a47-8fba-4592-a690-68b6e60ed8dc\") " pod="e2e-models-e2e-worker-single-main-4e5fe5/simulator-single-main-4e5fe5-kserve-68fc86fc88-v2z5q" Nov 03 03:33:37.942480 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:33:37.942338 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ad87a47-8fba-4592-a690-68b6e60ed8dc-tmp-dir\") pod \"simulator-single-main-4e5fe5-kserve-68fc86fc88-v2z5q\" (UID: \"7ad87a47-8fba-4592-a690-68b6e60ed8dc\") " pod="e2e-models-e2e-worker-single-main-4e5fe5/simulator-single-main-4e5fe5-kserve-68fc86fc88-v2z5q" Nov 03 03:33:37.944447 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:33:37.944420 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ad87a47-8fba-4592-a690-68b6e60ed8dc-dshm\") pod \"simulator-single-main-4e5fe5-kserve-68fc86fc88-v2z5q\" (UID: \"7ad87a47-8fba-4592-a690-68b6e60ed8dc\") " pod="e2e-models-e2e-worker-single-main-4e5fe5/simulator-single-main-4e5fe5-kserve-68fc86fc88-v2z5q" Nov 03 03:33:37.944652 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:33:37.944633 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ad87a47-8fba-4592-a690-68b6e60ed8dc-tls-certs\") pod \"simulator-single-main-4e5fe5-kserve-68fc86fc88-v2z5q\" (UID: \"7ad87a47-8fba-4592-a690-68b6e60ed8dc\") " pod="e2e-models-e2e-worker-single-main-4e5fe5/simulator-single-main-4e5fe5-kserve-68fc86fc88-v2z5q" Nov 03 03:33:37.995243 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:33:37.995208 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn6cb\" (UniqueName: \"kubernetes.io/projected/7ad87a47-8fba-4592-a690-68b6e60ed8dc-kube-api-access-bn6cb\") pod \"simulator-single-main-4e5fe5-kserve-68fc86fc88-v2z5q\" (UID: \"7ad87a47-8fba-4592-a690-68b6e60ed8dc\") " pod="e2e-models-e2e-worker-single-main-4e5fe5/simulator-single-main-4e5fe5-kserve-68fc86fc88-v2z5q" Nov 03 03:33:38.062691 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:33:38.062646 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-4e5fe5/simulator-single-main-4e5fe5-kserve-68fc86fc88-v2z5q" Nov 03 03:33:38.440351 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:33:38.440257 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-4e5fe5/simulator-single-main-4e5fe5-kserve-68fc86fc88-v2z5q"] Nov 03 03:33:38.446405 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:33:38.446369 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ad87a47_8fba_4592_a690_68b6e60ed8dc.slice/crio-aa61b947e269c427254a70231f9ba618400afafbff153b649363bd566797fcc6 WatchSource:0}: Error finding container aa61b947e269c427254a70231f9ba618400afafbff153b649363bd566797fcc6: Status 404 returned error can't find the container with id aa61b947e269c427254a70231f9ba618400afafbff153b649363bd566797fcc6 Nov 03 03:33:39.429623 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:33:39.429581 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-4e5fe5/simulator-single-main-4e5fe5-kserve-68fc86fc88-v2z5q" event={"ID":"7ad87a47-8fba-4592-a690-68b6e60ed8dc","Type":"ContainerStarted","Data":"d596128b266d728244dd83e861a8246a71fb61b6b23575db024124f1eec73f5a"} Nov 03 03:33:39.429623 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:33:39.429629 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-4e5fe5/simulator-single-main-4e5fe5-kserve-68fc86fc88-v2z5q" event={"ID":"7ad87a47-8fba-4592-a690-68b6e60ed8dc","Type":"ContainerStarted","Data":"aa61b947e269c427254a70231f9ba618400afafbff153b649363bd566797fcc6"} Nov 03 03:33:39.536735 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:33:39.536677 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-4e5fe5/simulator-single-main-4e5fe5-kserve-68fc86fc88-v2z5q" podStartSLOduration=2.536664671 podStartE2EDuration="2.536664671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:33:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:33:39.536253493 +0000 UTC m=+6096.795451647" watchObservedRunningTime="2026-11-03 03:33:39.536664671 +0000 UTC m=+6096.795862824" Nov 03 03:33:40.063308 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:33:40.063265 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-4e5fe5/simulator-single-main-4e5fe5-kserve-68fc86fc88-v2z5q" Nov 03 03:33:42.263690 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:33:42.263656 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:33:48.062908 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:33:48.062874 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-4e5fe5/simulator-single-main-4e5fe5-kserve-68fc86fc88-v2z5q" Nov 03 03:33:48.075352 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:33:48.075329 5605 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-4e5fe5/simulator-single-main-4e5fe5-kserve-68fc86fc88-v2z5q" Nov 03 03:33:48.478847 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:33:48.478764 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-4e5fe5/simulator-single-main-4e5fe5-kserve-68fc86fc88-v2z5q" Nov 03 03:33:53.911677 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:33:53.911631 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:34:05.906073 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:05.906042 5605 scope.go:122] "RemoveContainer" containerID="f7ea77977128a255be23d720b8b1fa6e84d4ab94aa9e960608b516586d7cff60" Nov 03 03:34:08.227630 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:08.227593 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:34:22.810904 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:22.810826 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6c94f3e-679hl"] Nov 03 03:34:22.814074 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:22.814053 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c94f3e-679hl" Nov 03 03:34:22.863408 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:22.863385 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:34:22.909846 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:22.909822 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:34:22.909976 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:22.909887 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:34:22.909976 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:22.909898 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sb4qh\"" Nov 03 03:34:22.911261 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:22.911242 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6c94f3e-679hl"] Nov 03 03:34:22.953855 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:22.953833 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec0cd6b7-f158-4057-a2d1-b57a506c360f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6c94f3e-679hl\" (UID: \"ec0cd6b7-f158-4057-a2d1-b57a506c360f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c94f3e-679hl" Nov 03 03:34:22.953981 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:22.953868 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzxvk\" (UniqueName: \"kubernetes.io/projected/ec0cd6b7-f158-4057-a2d1-b57a506c360f-kube-api-access-pzxvk\") pod \"maas-api-revoke-sub-a6c94f3e-679hl\" (UID: \"ec0cd6b7-f158-4057-a2d1-b57a506c360f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c94f3e-679hl" Nov 03 03:34:22.953981 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:22.953890 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec0cd6b7-f158-4057-a2d1-b57a506c360f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6c94f3e-679hl\" (UID: \"ec0cd6b7-f158-4057-a2d1-b57a506c360f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c94f3e-679hl" Nov 03 03:34:23.054545 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:23.054518 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec0cd6b7-f158-4057-a2d1-b57a506c360f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6c94f3e-679hl\" (UID: \"ec0cd6b7-f158-4057-a2d1-b57a506c360f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c94f3e-679hl" Nov 03 03:34:23.054681 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:23.054551 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzxvk\" (UniqueName: \"kubernetes.io/projected/ec0cd6b7-f158-4057-a2d1-b57a506c360f-kube-api-access-pzxvk\") pod \"maas-api-revoke-sub-a6c94f3e-679hl\" (UID: \"ec0cd6b7-f158-4057-a2d1-b57a506c360f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c94f3e-679hl" Nov 03 03:34:23.054681 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:23.054574 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec0cd6b7-f158-4057-a2d1-b57a506c360f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6c94f3e-679hl\" (UID: \"ec0cd6b7-f158-4057-a2d1-b57a506c360f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c94f3e-679hl" Nov 03 03:34:23.055296 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:23.055278 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec0cd6b7-f158-4057-a2d1-b57a506c360f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6c94f3e-679hl\" (UID: \"ec0cd6b7-f158-4057-a2d1-b57a506c360f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c94f3e-679hl" Nov 03 03:34:23.056865 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:23.056847 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec0cd6b7-f158-4057-a2d1-b57a506c360f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6c94f3e-679hl\" (UID: \"ec0cd6b7-f158-4057-a2d1-b57a506c360f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c94f3e-679hl" Nov 03 03:34:23.109433 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:23.109375 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzxvk\" (UniqueName: \"kubernetes.io/projected/ec0cd6b7-f158-4057-a2d1-b57a506c360f-kube-api-access-pzxvk\") pod \"maas-api-revoke-sub-a6c94f3e-679hl\" (UID: \"ec0cd6b7-f158-4057-a2d1-b57a506c360f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c94f3e-679hl" Nov 03 03:34:23.126198 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:23.126174 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c94f3e-679hl" Nov 03 03:34:23.490192 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:23.490115 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6c94f3e-679hl"] Nov 03 03:34:23.497542 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:34:23.497501 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec0cd6b7_f158_4057_a2d1_b57a506c360f.slice/crio-1c7fd5ba0299090d61f66a609e1368084176d31819b2c6495e00e855302d1abf WatchSource:0}: Error finding container 1c7fd5ba0299090d61f66a609e1368084176d31819b2c6495e00e855302d1abf: Status 404 returned error can't find the container with id 1c7fd5ba0299090d61f66a609e1368084176d31819b2c6495e00e855302d1abf Nov 03 03:34:23.620414 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:23.620374 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c94f3e-679hl" event={"ID":"ec0cd6b7-f158-4057-a2d1-b57a506c360f","Type":"ContainerStarted","Data":"1c7fd5ba0299090d61f66a609e1368084176d31819b2c6495e00e855302d1abf"} Nov 03 03:34:23.911463 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:34:23.911433 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:34:24.625799 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:24.625712 5605 generic.go:356] "Generic (PLEG): container finished" podID="ec0cd6b7-f158-4057-a2d1-b57a506c360f" containerID="20a85ca70c5bb1590b259134ed7b45cb358f6a794204aaf4bdb5391202a482d9" exitCode=0 Nov 03 03:34:24.625952 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:24.625795 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c94f3e-679hl" event={"ID":"ec0cd6b7-f158-4057-a2d1-b57a506c360f","Type":"ContainerDied","Data":"20a85ca70c5bb1590b259134ed7b45cb358f6a794204aaf4bdb5391202a482d9"} Nov 03 03:34:25.756735 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:25.756707 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c94f3e-679hl" Nov 03 03:34:25.876660 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:25.876629 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec0cd6b7-f158-4057-a2d1-b57a506c360f-kube-api-access-pzxvk\" (UniqueName: \"kubernetes.io/projected/ec0cd6b7-f158-4057-a2d1-b57a506c360f-kube-api-access-pzxvk\") pod \"ec0cd6b7-f158-4057-a2d1-b57a506c360f\" (UID: \"ec0cd6b7-f158-4057-a2d1-b57a506c360f\") " Nov 03 03:34:25.876803 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:25.876756 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec0cd6b7-f158-4057-a2d1-b57a506c360f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec0cd6b7-f158-4057-a2d1-b57a506c360f-maas-api-openshift-ca\") pod \"ec0cd6b7-f158-4057-a2d1-b57a506c360f\" (UID: \"ec0cd6b7-f158-4057-a2d1-b57a506c360f\") " Nov 03 03:34:25.876856 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:25.876809 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec0cd6b7-f158-4057-a2d1-b57a506c360f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec0cd6b7-f158-4057-a2d1-b57a506c360f-maas-api-xks-ca\") pod \"ec0cd6b7-f158-4057-a2d1-b57a506c360f\" (UID: \"ec0cd6b7-f158-4057-a2d1-b57a506c360f\") " Nov 03 03:34:25.877124 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:25.877095 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec0cd6b7-f158-4057-a2d1-b57a506c360f-maas-api-openshift-ca" pod "ec0cd6b7-f158-4057-a2d1-b57a506c360f" (UID: "ec0cd6b7-f158-4057-a2d1-b57a506c360f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:34:25.878839 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:25.878815 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec0cd6b7-f158-4057-a2d1-b57a506c360f-maas-api-xks-ca" pod "ec0cd6b7-f158-4057-a2d1-b57a506c360f" (UID: "ec0cd6b7-f158-4057-a2d1-b57a506c360f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:34:25.878947 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:25.878841 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec0cd6b7-f158-4057-a2d1-b57a506c360f-kube-api-access-pzxvk" pod "ec0cd6b7-f158-4057-a2d1-b57a506c360f" (UID: "ec0cd6b7-f158-4057-a2d1-b57a506c360f"). InnerVolumeSpecName "kube-api-access-pzxvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:34:25.977533 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:25.977506 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec0cd6b7-f158-4057-a2d1-b57a506c360f-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:34:25.977533 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:25.977531 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec0cd6b7-f158-4057-a2d1-b57a506c360f-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:34:25.977681 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:25.977541 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzxvk\" (UniqueName: \"kubernetes.io/projected/ec0cd6b7-f158-4057-a2d1-b57a506c360f-kube-api-access-pzxvk\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:34:26.635880 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:26.635850 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c94f3e-679hl" Nov 03 03:34:26.636119 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:26.635881 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c94f3e-679hl" event={"ID":"ec0cd6b7-f158-4057-a2d1-b57a506c360f","Type":"ContainerDied","Data":"1c7fd5ba0299090d61f66a609e1368084176d31819b2c6495e00e855302d1abf"} Nov 03 03:34:26.636119 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:26.635926 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c7fd5ba0299090d61f66a609e1368084176d31819b2c6495e00e855302d1abf" Nov 03 03:34:37.464029 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:37.463994 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:34:39.499429 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:39.499399 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-4e5fe5/simulator-single-main-4e5fe5-kserve-68fc86fc88-v2z5q"] Nov 03 03:34:41.651723 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:41.651683 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4e5fe5-669549f59wq894"] Nov 03 03:34:46.652772 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:46.652711 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e5fe5-669549f59wq894" podUID="3629605f-0698-4839-920d-dc3ae1c18f88" containerName="payload-processing" containerID="cri-o://298a8a0ec158204921254ccbe0120b9ef5942e4e31f653c9243d9038a8128eb9" gracePeriod=25 Nov 03 03:34:50.919807 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:50.919772 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:34:53.913735 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:34:53.913695 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:34:54.519353 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:54.519311 5605 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-4e5fe5/simulator-single-main-4e5fe5-kserve-68fc86fc88-v2z5q" podUID="7ad87a47-8fba-4592-a690-68b6e60ed8dc" containerName="main" containerID="cri-o://d596128b266d728244dd83e861a8246a71fb61b6b23575db024124f1eec73f5a" gracePeriod=45 Nov 03 03:34:54.758783 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:54.758757 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-4e5fe5/simulator-single-main-4e5fe5-kserve-68fc86fc88-v2z5q" Nov 03 03:34:54.765636 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:54.765607 5605 generic.go:356] "Generic (PLEG): container finished" podID="7ad87a47-8fba-4592-a690-68b6e60ed8dc" containerID="d596128b266d728244dd83e861a8246a71fb61b6b23575db024124f1eec73f5a" exitCode=0 Nov 03 03:34:54.765762 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:54.765643 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-4e5fe5/simulator-single-main-4e5fe5-kserve-68fc86fc88-v2z5q" Nov 03 03:34:54.765762 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:54.765645 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-4e5fe5/simulator-single-main-4e5fe5-kserve-68fc86fc88-v2z5q" event={"ID":"7ad87a47-8fba-4592-a690-68b6e60ed8dc","Type":"ContainerDied","Data":"d596128b266d728244dd83e861a8246a71fb61b6b23575db024124f1eec73f5a"} Nov 03 03:34:54.765839 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:54.765761 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-4e5fe5/simulator-single-main-4e5fe5-kserve-68fc86fc88-v2z5q" event={"ID":"7ad87a47-8fba-4592-a690-68b6e60ed8dc","Type":"ContainerDied","Data":"aa61b947e269c427254a70231f9ba618400afafbff153b649363bd566797fcc6"} Nov 03 03:34:54.765839 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:54.765783 5605 scope.go:122] "RemoveContainer" containerID="d596128b266d728244dd83e861a8246a71fb61b6b23575db024124f1eec73f5a" Nov 03 03:34:54.773482 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:54.773462 5605 scope.go:122] "RemoveContainer" containerID="d596128b266d728244dd83e861a8246a71fb61b6b23575db024124f1eec73f5a" Nov 03 03:34:54.773745 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:34:54.773722 5605 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d596128b266d728244dd83e861a8246a71fb61b6b23575db024124f1eec73f5a\": container with ID starting with d596128b266d728244dd83e861a8246a71fb61b6b23575db024124f1eec73f5a not found: ID does not exist" containerID="d596128b266d728244dd83e861a8246a71fb61b6b23575db024124f1eec73f5a" Nov 03 03:34:54.773845 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:54.773751 5605 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d596128b266d728244dd83e861a8246a71fb61b6b23575db024124f1eec73f5a"} err="failed to get container status \"d596128b266d728244dd83e861a8246a71fb61b6b23575db024124f1eec73f5a\": rpc error: code = NotFound desc = could not find container \"d596128b266d728244dd83e861a8246a71fb61b6b23575db024124f1eec73f5a\": container with ID starting with d596128b266d728244dd83e861a8246a71fb61b6b23575db024124f1eec73f5a not found: ID does not exist" Nov 03 03:34:54.821675 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:54.821650 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ad87a47-8fba-4592-a690-68b6e60ed8dc-home\" (UniqueName: \"kubernetes.io/empty-dir/7ad87a47-8fba-4592-a690-68b6e60ed8dc-home\") pod \"7ad87a47-8fba-4592-a690-68b6e60ed8dc\" (UID: \"7ad87a47-8fba-4592-a690-68b6e60ed8dc\") " Nov 03 03:34:54.821787 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:54.821760 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ad87a47-8fba-4592-a690-68b6e60ed8dc-tls-certs\" (UniqueName: \"kubernetes.io/secret/7ad87a47-8fba-4592-a690-68b6e60ed8dc-tls-certs\") pod \"7ad87a47-8fba-4592-a690-68b6e60ed8dc\" (UID: \"7ad87a47-8fba-4592-a690-68b6e60ed8dc\") " Nov 03 03:34:54.821854 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:54.821799 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ad87a47-8fba-4592-a690-68b6e60ed8dc-kube-api-access-bn6cb\" (UniqueName: \"kubernetes.io/projected/7ad87a47-8fba-4592-a690-68b6e60ed8dc-kube-api-access-bn6cb\") pod \"7ad87a47-8fba-4592-a690-68b6e60ed8dc\" (UID: \"7ad87a47-8fba-4592-a690-68b6e60ed8dc\") " Nov 03 03:34:54.821854 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:54.821830 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ad87a47-8fba-4592-a690-68b6e60ed8dc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ad87a47-8fba-4592-a690-68b6e60ed8dc-dshm\") pod \"7ad87a47-8fba-4592-a690-68b6e60ed8dc\" (UID: \"7ad87a47-8fba-4592-a690-68b6e60ed8dc\") " Nov 03 03:34:54.821969 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:54.821867 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ad87a47-8fba-4592-a690-68b6e60ed8dc-home" pod "7ad87a47-8fba-4592-a690-68b6e60ed8dc" (UID: "7ad87a47-8fba-4592-a690-68b6e60ed8dc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:34:54.821969 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:54.821886 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ad87a47-8fba-4592-a690-68b6e60ed8dc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ad87a47-8fba-4592-a690-68b6e60ed8dc-model-cache\") pod \"7ad87a47-8fba-4592-a690-68b6e60ed8dc\" (UID: \"7ad87a47-8fba-4592-a690-68b6e60ed8dc\") " Nov 03 03:34:54.821969 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:54.821947 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ad87a47-8fba-4592-a690-68b6e60ed8dc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ad87a47-8fba-4592-a690-68b6e60ed8dc-tmp-dir\") pod \"7ad87a47-8fba-4592-a690-68b6e60ed8dc\" (UID: \"7ad87a47-8fba-4592-a690-68b6e60ed8dc\") " Nov 03 03:34:54.822232 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:54.822209 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ad87a47-8fba-4592-a690-68b6e60ed8dc-model-cache" pod "7ad87a47-8fba-4592-a690-68b6e60ed8dc" (UID: "7ad87a47-8fba-4592-a690-68b6e60ed8dc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:34:54.822232 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:54.822226 5605 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ad87a47-8fba-4592-a690-68b6e60ed8dc-home\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:34:54.822347 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:54.822295 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ad87a47-8fba-4592-a690-68b6e60ed8dc-tmp-dir" pod "7ad87a47-8fba-4592-a690-68b6e60ed8dc" (UID: "7ad87a47-8fba-4592-a690-68b6e60ed8dc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:34:54.823866 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:54.823847 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ad87a47-8fba-4592-a690-68b6e60ed8dc-dshm" pod "7ad87a47-8fba-4592-a690-68b6e60ed8dc" (UID: "7ad87a47-8fba-4592-a690-68b6e60ed8dc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:34:54.823992 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:54.823975 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ad87a47-8fba-4592-a690-68b6e60ed8dc-kube-api-access-bn6cb" pod "7ad87a47-8fba-4592-a690-68b6e60ed8dc" (UID: "7ad87a47-8fba-4592-a690-68b6e60ed8dc"). InnerVolumeSpecName "kube-api-access-bn6cb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:34:54.824045 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:54.824009 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ad87a47-8fba-4592-a690-68b6e60ed8dc-tls-certs" pod "7ad87a47-8fba-4592-a690-68b6e60ed8dc" (UID: "7ad87a47-8fba-4592-a690-68b6e60ed8dc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:34:54.923598 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:54.923567 5605 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ad87a47-8fba-4592-a690-68b6e60ed8dc-dshm\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:34:54.923598 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:54.923596 5605 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ad87a47-8fba-4592-a690-68b6e60ed8dc-model-cache\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:34:54.923993 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:54.923608 5605 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ad87a47-8fba-4592-a690-68b6e60ed8dc-tmp-dir\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:34:54.923993 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:54.923620 5605 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ad87a47-8fba-4592-a690-68b6e60ed8dc-tls-certs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:34:54.923993 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:54.923632 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn6cb\" (UniqueName: \"kubernetes.io/projected/7ad87a47-8fba-4592-a690-68b6e60ed8dc-kube-api-access-bn6cb\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:34:55.172215 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:55.172182 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-4e5fe5/simulator-single-main-4e5fe5-kserve-68fc86fc88-v2z5q"] Nov 03 03:34:55.177510 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:55.177483 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-4e5fe5/simulator-single-main-4e5fe5-kserve-68fc86fc88-v2z5q"] Nov 03 03:34:55.309253 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:55.309213 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e5fe5-669549f59wq894" podUID="3629605f-0698-4839-920d-dc3ae1c18f88" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:34:56.848348 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:34:56.848313 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ad87a47-8fba-4592-a690-68b6e60ed8dc" path="/var/lib/kubelet/pods/7ad87a47-8fba-4592-a690-68b6e60ed8dc/volumes" Nov 03 03:35:05.309989 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:35:05.309951 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e5fe5-669549f59wq894" podUID="3629605f-0698-4839-920d-dc3ae1c18f88" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:35:06.823706 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:35:06.823671 5605 generic.go:356] "Generic (PLEG): container finished" podID="3629605f-0698-4839-920d-dc3ae1c18f88" containerID="298a8a0ec158204921254ccbe0120b9ef5942e4e31f653c9243d9038a8128eb9" exitCode=0 Nov 03 03:35:06.824149 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:35:06.823787 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e5fe5-669549f59wq894" event={"ID":"3629605f-0698-4839-920d-dc3ae1c18f88","Type":"ContainerDied","Data":"298a8a0ec158204921254ccbe0120b9ef5942e4e31f653c9243d9038a8128eb9"} Nov 03 03:35:06.891731 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:35:06.891704 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e5fe5-669549f59wq894" Nov 03 03:35:07.018290 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:35:07.018211 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3629605f-0698-4839-920d-dc3ae1c18f88-kube-api-access-qmw7p\" (UniqueName: \"kubernetes.io/projected/3629605f-0698-4839-920d-dc3ae1c18f88-kube-api-access-qmw7p\") pod \"3629605f-0698-4839-920d-dc3ae1c18f88\" (UID: \"3629605f-0698-4839-920d-dc3ae1c18f88\") " Nov 03 03:35:07.018290 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:35:07.018278 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3629605f-0698-4839-920d-dc3ae1c18f88-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3629605f-0698-4839-920d-dc3ae1c18f88-plugins-config-volume\") pod \"3629605f-0698-4839-920d-dc3ae1c18f88\" (UID: \"3629605f-0698-4839-920d-dc3ae1c18f88\") " Nov 03 03:35:07.018694 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:35:07.018670 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3629605f-0698-4839-920d-dc3ae1c18f88-plugins-config-volume" pod "3629605f-0698-4839-920d-dc3ae1c18f88" (UID: "3629605f-0698-4839-920d-dc3ae1c18f88"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:35:07.020303 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:35:07.020282 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3629605f-0698-4839-920d-dc3ae1c18f88-kube-api-access-qmw7p" pod "3629605f-0698-4839-920d-dc3ae1c18f88" (UID: "3629605f-0698-4839-920d-dc3ae1c18f88"). InnerVolumeSpecName "kube-api-access-qmw7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:35:07.119849 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:35:07.119826 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmw7p\" (UniqueName: \"kubernetes.io/projected/3629605f-0698-4839-920d-dc3ae1c18f88-kube-api-access-qmw7p\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:35:07.119849 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:35:07.119846 5605 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3629605f-0698-4839-920d-dc3ae1c18f88-plugins-config-volume\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:35:07.829086 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:35:07.829063 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e5fe5-669549f59wq894" Nov 03 03:35:07.829517 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:35:07.829061 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e5fe5-669549f59wq894" event={"ID":"3629605f-0698-4839-920d-dc3ae1c18f88","Type":"ContainerDied","Data":"f5a8c476990b81926d44b8f51bcc7fd986d0715794ff7bd3c29189c22ab7f4ca"} Nov 03 03:35:07.829517 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:35:07.829203 5605 scope.go:122] "RemoveContainer" containerID="298a8a0ec158204921254ccbe0120b9ef5942e4e31f653c9243d9038a8128eb9" Nov 03 03:35:07.981525 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:35:07.981494 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4e5fe5-669549f59wq894"] Nov 03 03:35:08.028098 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:35:08.028072 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4e5fe5-669549f59wq894"] Nov 03 03:35:08.848732 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:35:08.848695 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3629605f-0698-4839-920d-dc3ae1c18f88" path="/var/lib/kubelet/pods/3629605f-0698-4839-920d-dc3ae1c18f88/volumes" Nov 03 03:35:20.973019 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:35:20.972986 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sfzzw"] Nov 03 03:35:20.973425 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:35:20.973306 5605 cpu_manager.go:414] "containerMap: removing container" podUID="3629605f-0698-4839-920d-dc3ae1c18f88" containerName="payload-processing" Nov 03 03:35:20.973425 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:35:20.973316 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3629605f-0698-4839-920d-dc3ae1c18f88" containerName="payload-processing" Nov 03 03:35:20.973425 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:35:20.973339 5605 cpu_manager.go:414] "containerMap: removing container" podUID="ec0cd6b7-f158-4057-a2d1-b57a506c360f" containerName="revoke-keys" Nov 03 03:35:20.973425 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:35:20.973345 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec0cd6b7-f158-4057-a2d1-b57a506c360f" containerName="revoke-keys" Nov 03 03:35:20.973425 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:35:20.973367 5605 cpu_manager.go:414] "containerMap: removing container" podUID="7ad87a47-8fba-4592-a690-68b6e60ed8dc" containerName="main" Nov 03 03:35:20.973425 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:35:20.973372 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ad87a47-8fba-4592-a690-68b6e60ed8dc" containerName="main" Nov 03 03:35:20.973425 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:35:20.973425 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="3629605f-0698-4839-920d-dc3ae1c18f88" containerName="payload-processing" Nov 03 03:35:20.973712 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:35:20.973437 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec0cd6b7-f158-4057-a2d1-b57a506c360f" containerName="revoke-keys" Nov 03 03:35:20.973712 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:35:20.973444 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ad87a47-8fba-4592-a690-68b6e60ed8dc" containerName="main" Nov 03 03:35:20.977714 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:35:20.977690 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sfzzw" Nov 03 03:35:21.026869 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:35:21.026841 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:35:21.026973 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:35:21.026886 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:35:21.027018 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:35:21.026977 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:35:21.073854 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:35:21.073831 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sb4qh\"" Nov 03 03:35:21.075284 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:35:21.075263 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sfzzw"] Nov 03 03:35:21.076308 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:35:21.076291 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sbhx6"] Nov 03 03:35:21.077149 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:35:21.077130 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sbhx6"] Nov 03 03:35:21.139751 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:35:21.139729 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qmcf\" (UniqueName: \"kubernetes.io/projected/b028cc05-9223-4ef0-8147-a0d4df641ff3-kube-api-access-5qmcf\") pod \"maas-api-revoke-sub-3d4156f1-sfzzw\" (UID: \"b028cc05-9223-4ef0-8147-a0d4df641ff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sfzzw" Nov 03 03:35:21.139868 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:35:21.139765 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b028cc05-9223-4ef0-8147-a0d4df641ff3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-sfzzw\" (UID: \"b028cc05-9223-4ef0-8147-a0d4df641ff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sfzzw" Nov 03 03:35:21.139868 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:35:21.139806 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b028cc05-9223-4ef0-8147-a0d4df641ff3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-sfzzw\" (UID: \"b028cc05-9223-4ef0-8147-a0d4df641ff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sfzzw" Nov 03 03:35:21.240716 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:35:21.240656 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qmcf\" (UniqueName: \"kubernetes.io/projected/b028cc05-9223-4ef0-8147-a0d4df641ff3-kube-api-access-5qmcf\") pod \"maas-api-revoke-sub-3d4156f1-sfzzw\" (UID: \"b028cc05-9223-4ef0-8147-a0d4df641ff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sfzzw" Nov 03 03:35:21.240716 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:35:21.240691 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b028cc05-9223-4ef0-8147-a0d4df641ff3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-sfzzw\" (UID: \"b028cc05-9223-4ef0-8147-a0d4df641ff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sfzzw" Nov 03 03:35:21.240869 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:35:21.240724 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b028cc05-9223-4ef0-8147-a0d4df641ff3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-sfzzw\" (UID: \"b028cc05-9223-4ef0-8147-a0d4df641ff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sfzzw" Nov 03 03:35:21.241331 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:35:21.241313 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b028cc05-9223-4ef0-8147-a0d4df641ff3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-sfzzw\" (UID: \"b028cc05-9223-4ef0-8147-a0d4df641ff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sfzzw" Nov 03 03:35:21.243153 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:35:21.243131 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b028cc05-9223-4ef0-8147-a0d4df641ff3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-sfzzw\" (UID: \"b028cc05-9223-4ef0-8147-a0d4df641ff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sfzzw" Nov 03 03:35:21.294829 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:35:21.294802 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qmcf\" (UniqueName: \"kubernetes.io/projected/b028cc05-9223-4ef0-8147-a0d4df641ff3-kube-api-access-5qmcf\") pod \"maas-api-revoke-sub-3d4156f1-sfzzw\" (UID: \"b028cc05-9223-4ef0-8147-a0d4df641ff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sfzzw" Nov 03 03:35:21.589960 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:35:21.589921 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sfzzw" Nov 03 03:35:21.954619 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:35:21.954545 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sfzzw"] Nov 03 03:35:21.961216 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:35:21.961188 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb028cc05_9223_4ef0_8147_a0d4df641ff3.slice/crio-892b272e3f1a9cedccdfd6f17fe734238577787afdcf4683d24718dbc76448c4 WatchSource:0}: Error finding container 892b272e3f1a9cedccdfd6f17fe734238577787afdcf4683d24718dbc76448c4: Status 404 returned error can't find the container with id 892b272e3f1a9cedccdfd6f17fe734238577787afdcf4683d24718dbc76448c4 Nov 03 03:35:22.848617 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:35:22.848591 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="488f5888-a1ce-4f19-954d-06c7deec58b2" path="/var/lib/kubelet/pods/488f5888-a1ce-4f19-954d-06c7deec58b2/volumes" Nov 03 03:35:22.890590 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:35:22.890560 5605 generic.go:356] "Generic (PLEG): container finished" podID="b028cc05-9223-4ef0-8147-a0d4df641ff3" containerID="fcc66b42d23cfbfb159d9e3bd029ccc254723f825a229efe1f3277dc0e7862ae" exitCode=0 Nov 03 03:35:22.890709 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:35:22.890605 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sfzzw" event={"ID":"b028cc05-9223-4ef0-8147-a0d4df641ff3","Type":"ContainerDied","Data":"fcc66b42d23cfbfb159d9e3bd029ccc254723f825a229efe1f3277dc0e7862ae"} Nov 03 03:35:22.890709 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:35:22.890656 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sfzzw" event={"ID":"b028cc05-9223-4ef0-8147-a0d4df641ff3","Type":"ContainerStarted","Data":"892b272e3f1a9cedccdfd6f17fe734238577787afdcf4683d24718dbc76448c4"} Nov 03 03:35:23.912330 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:35:23.912296 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:35:24.020791 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:35:24.020765 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sfzzw" Nov 03 03:35:24.166651 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:35:24.166576 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b028cc05-9223-4ef0-8147-a0d4df641ff3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b028cc05-9223-4ef0-8147-a0d4df641ff3-maas-api-xks-ca\") pod \"b028cc05-9223-4ef0-8147-a0d4df641ff3\" (UID: \"b028cc05-9223-4ef0-8147-a0d4df641ff3\") " Nov 03 03:35:24.166651 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:35:24.166646 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b028cc05-9223-4ef0-8147-a0d4df641ff3-kube-api-access-5qmcf\" (UniqueName: \"kubernetes.io/projected/b028cc05-9223-4ef0-8147-a0d4df641ff3-kube-api-access-5qmcf\") pod \"b028cc05-9223-4ef0-8147-a0d4df641ff3\" (UID: \"b028cc05-9223-4ef0-8147-a0d4df641ff3\") " Nov 03 03:35:24.166866 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:35:24.166737 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b028cc05-9223-4ef0-8147-a0d4df641ff3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b028cc05-9223-4ef0-8147-a0d4df641ff3-maas-api-openshift-ca\") pod \"b028cc05-9223-4ef0-8147-a0d4df641ff3\" (UID: \"b028cc05-9223-4ef0-8147-a0d4df641ff3\") " Nov 03 03:35:24.167099 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:35:24.167075 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b028cc05-9223-4ef0-8147-a0d4df641ff3-maas-api-openshift-ca" pod "b028cc05-9223-4ef0-8147-a0d4df641ff3" (UID: "b028cc05-9223-4ef0-8147-a0d4df641ff3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:35:24.168678 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:35:24.168657 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b028cc05-9223-4ef0-8147-a0d4df641ff3-maas-api-xks-ca" pod "b028cc05-9223-4ef0-8147-a0d4df641ff3" (UID: "b028cc05-9223-4ef0-8147-a0d4df641ff3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:35:24.168780 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:35:24.168715 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b028cc05-9223-4ef0-8147-a0d4df641ff3-kube-api-access-5qmcf" pod "b028cc05-9223-4ef0-8147-a0d4df641ff3" (UID: "b028cc05-9223-4ef0-8147-a0d4df641ff3"). InnerVolumeSpecName "kube-api-access-5qmcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:35:24.267970 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:35:24.267948 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qmcf\" (UniqueName: \"kubernetes.io/projected/b028cc05-9223-4ef0-8147-a0d4df641ff3-kube-api-access-5qmcf\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:35:24.267970 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:35:24.267969 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b028cc05-9223-4ef0-8147-a0d4df641ff3-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:35:24.268111 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:35:24.267981 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b028cc05-9223-4ef0-8147-a0d4df641ff3-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:35:24.900131 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:35:24.900101 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sfzzw" event={"ID":"b028cc05-9223-4ef0-8147-a0d4df641ff3","Type":"ContainerDied","Data":"892b272e3f1a9cedccdfd6f17fe734238577787afdcf4683d24718dbc76448c4"} Nov 03 03:35:24.900280 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:35:24.900140 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sfzzw" Nov 03 03:35:24.900280 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:35:24.900144 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="892b272e3f1a9cedccdfd6f17fe734238577787afdcf4683d24718dbc76448c4" Nov 03 03:35:35.367269 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:35:35.367232 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:35:53.912897 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:35:53.912786 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:36:23.910875 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:36:23.910840 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:36:53.911351 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:36:53.911312 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:37:05.944064 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:37:05.944031 5605 scope.go:122] "RemoveContainer" containerID="fbbc241dfcf77d99f538b5b933ef07452893aeec3c607be8ce09c66839e26d55" Nov 03 03:37:11.665943 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:37:11.665863 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:37:23.911859 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:37:23.911816 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:37:24.064265 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:37:24.064234 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:37:33.534881 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:37:33.534843 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:37:41.534346 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:37:41.534309 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:37:53.912060 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:37:53.912017 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:37:59.032496 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:37:59.032457 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:38:12.677584 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:12.677549 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c6e4f9a-z2m2v"] Nov 03 03:38:12.678054 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:12.677908 5605 cpu_manager.go:414] "containerMap: removing container" podUID="b028cc05-9223-4ef0-8147-a0d4df641ff3" containerName="revoke-keys" Nov 03 03:38:12.678054 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:12.677920 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b028cc05-9223-4ef0-8147-a0d4df641ff3" containerName="revoke-keys" Nov 03 03:38:12.678054 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:12.677998 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="b028cc05-9223-4ef0-8147-a0d4df641ff3" containerName="revoke-keys" Nov 03 03:38:12.681081 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:12.681053 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c6e4f9a-z2m2v" Nov 03 03:38:12.777148 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:12.777116 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:38:12.777148 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:12.777135 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:38:12.777412 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:12.777201 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sb4qh\"" Nov 03 03:38:12.777412 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:12.777207 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:38:12.778563 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:12.778541 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c6e4f9a-z2m2v"] Nov 03 03:38:12.784871 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:12.784848 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee6e54e2-3009-41a1-a617-d2d4dc6166d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c6e4f9a-z2m2v\" (UID: \"ee6e54e2-3009-41a1-a617-d2d4dc6166d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c6e4f9a-z2m2v" Nov 03 03:38:12.784993 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:12.784900 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee6e54e2-3009-41a1-a617-d2d4dc6166d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c6e4f9a-z2m2v\" (UID: \"ee6e54e2-3009-41a1-a617-d2d4dc6166d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c6e4f9a-z2m2v" Nov 03 03:38:12.785063 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:12.785007 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs6bj\" (UniqueName: \"kubernetes.io/projected/ee6e54e2-3009-41a1-a617-d2d4dc6166d7-kube-api-access-qs6bj\") pod \"maas-api-revoke-sub-2c6e4f9a-z2m2v\" (UID: \"ee6e54e2-3009-41a1-a617-d2d4dc6166d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c6e4f9a-z2m2v" Nov 03 03:38:12.885523 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:12.885488 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee6e54e2-3009-41a1-a617-d2d4dc6166d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c6e4f9a-z2m2v\" (UID: \"ee6e54e2-3009-41a1-a617-d2d4dc6166d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c6e4f9a-z2m2v" Nov 03 03:38:12.885741 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:12.885544 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee6e54e2-3009-41a1-a617-d2d4dc6166d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c6e4f9a-z2m2v\" (UID: \"ee6e54e2-3009-41a1-a617-d2d4dc6166d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c6e4f9a-z2m2v" Nov 03 03:38:12.885741 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:12.885591 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs6bj\" (UniqueName: \"kubernetes.io/projected/ee6e54e2-3009-41a1-a617-d2d4dc6166d7-kube-api-access-qs6bj\") pod \"maas-api-revoke-sub-2c6e4f9a-z2m2v\" (UID: \"ee6e54e2-3009-41a1-a617-d2d4dc6166d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c6e4f9a-z2m2v" Nov 03 03:38:12.886271 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:12.886244 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee6e54e2-3009-41a1-a617-d2d4dc6166d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c6e4f9a-z2m2v\" (UID: \"ee6e54e2-3009-41a1-a617-d2d4dc6166d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c6e4f9a-z2m2v" Nov 03 03:38:12.887913 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:12.887889 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee6e54e2-3009-41a1-a617-d2d4dc6166d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c6e4f9a-z2m2v\" (UID: \"ee6e54e2-3009-41a1-a617-d2d4dc6166d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c6e4f9a-z2m2v" Nov 03 03:38:12.939065 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:12.938989 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs6bj\" (UniqueName: \"kubernetes.io/projected/ee6e54e2-3009-41a1-a617-d2d4dc6166d7-kube-api-access-qs6bj\") pod \"maas-api-revoke-sub-2c6e4f9a-z2m2v\" (UID: \"ee6e54e2-3009-41a1-a617-d2d4dc6166d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c6e4f9a-z2m2v" Nov 03 03:38:12.993696 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:12.993658 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c6e4f9a-z2m2v" Nov 03 03:38:13.358810 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:13.358772 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c6e4f9a-z2m2v"] Nov 03 03:38:13.366171 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:38:13.366140 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee6e54e2_3009_41a1_a617_d2d4dc6166d7.slice/crio-7a8d5ac3ea542d1c68ae964af1dec9e5e5f5630acd754d3d8ead7340015d81f0 WatchSource:0}: Error finding container 7a8d5ac3ea542d1c68ae964af1dec9e5e5f5630acd754d3d8ead7340015d81f0: Status 404 returned error can't find the container with id 7a8d5ac3ea542d1c68ae964af1dec9e5e5f5630acd754d3d8ead7340015d81f0 Nov 03 03:38:13.913453 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:13.913369 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c6e4f9a-z2m2v" event={"ID":"ee6e54e2-3009-41a1-a617-d2d4dc6166d7","Type":"ContainerStarted","Data":"d0e94d49f7b3db5855d02258681a3dc63277f5552ded67c5fa6e387588e6d379"} Nov 03 03:38:13.913837 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:13.913455 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c6e4f9a-z2m2v" event={"ID":"ee6e54e2-3009-41a1-a617-d2d4dc6166d7","Type":"ContainerStarted","Data":"7a8d5ac3ea542d1c68ae964af1dec9e5e5f5630acd754d3d8ead7340015d81f0"} Nov 03 03:38:14.020140 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:14.020089 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c6e4f9a-z2m2v" podStartSLOduration=2.020074189 podStartE2EDuration="2.020074189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:38:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:38:14.018852721 +0000 UTC m=+6371.278050875" watchObservedRunningTime="2026-11-03 03:38:14.020074189 +0000 UTC m=+6371.279272337" Nov 03 03:38:14.919440 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:14.919402 5605 generic.go:356] "Generic (PLEG): container finished" podID="ee6e54e2-3009-41a1-a617-d2d4dc6166d7" containerID="d0e94d49f7b3db5855d02258681a3dc63277f5552ded67c5fa6e387588e6d379" exitCode=0 Nov 03 03:38:14.919849 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:14.919488 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c6e4f9a-z2m2v" event={"ID":"ee6e54e2-3009-41a1-a617-d2d4dc6166d7","Type":"ContainerDied","Data":"d0e94d49f7b3db5855d02258681a3dc63277f5552ded67c5fa6e387588e6d379"} Nov 03 03:38:16.053333 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:16.053308 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c6e4f9a-z2m2v" Nov 03 03:38:16.118299 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:16.118262 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee6e54e2-3009-41a1-a617-d2d4dc6166d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee6e54e2-3009-41a1-a617-d2d4dc6166d7-maas-api-openshift-ca\") pod \"ee6e54e2-3009-41a1-a617-d2d4dc6166d7\" (UID: \"ee6e54e2-3009-41a1-a617-d2d4dc6166d7\") " Nov 03 03:38:16.118508 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:16.118442 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee6e54e2-3009-41a1-a617-d2d4dc6166d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee6e54e2-3009-41a1-a617-d2d4dc6166d7-maas-api-xks-ca\") pod \"ee6e54e2-3009-41a1-a617-d2d4dc6166d7\" (UID: \"ee6e54e2-3009-41a1-a617-d2d4dc6166d7\") " Nov 03 03:38:16.118508 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:16.118482 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee6e54e2-3009-41a1-a617-d2d4dc6166d7-kube-api-access-qs6bj\" (UniqueName: \"kubernetes.io/projected/ee6e54e2-3009-41a1-a617-d2d4dc6166d7-kube-api-access-qs6bj\") pod \"ee6e54e2-3009-41a1-a617-d2d4dc6166d7\" (UID: \"ee6e54e2-3009-41a1-a617-d2d4dc6166d7\") " Nov 03 03:38:16.118732 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:16.118709 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee6e54e2-3009-41a1-a617-d2d4dc6166d7-maas-api-openshift-ca" pod "ee6e54e2-3009-41a1-a617-d2d4dc6166d7" (UID: "ee6e54e2-3009-41a1-a617-d2d4dc6166d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:38:16.120613 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:16.120580 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee6e54e2-3009-41a1-a617-d2d4dc6166d7-maas-api-xks-ca" pod "ee6e54e2-3009-41a1-a617-d2d4dc6166d7" (UID: "ee6e54e2-3009-41a1-a617-d2d4dc6166d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:38:16.120760 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:16.120642 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee6e54e2-3009-41a1-a617-d2d4dc6166d7-kube-api-access-qs6bj" pod "ee6e54e2-3009-41a1-a617-d2d4dc6166d7" (UID: "ee6e54e2-3009-41a1-a617-d2d4dc6166d7"). InnerVolumeSpecName "kube-api-access-qs6bj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:38:16.219900 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:16.219810 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee6e54e2-3009-41a1-a617-d2d4dc6166d7-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:38:16.219900 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:16.219847 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qs6bj\" (UniqueName: \"kubernetes.io/projected/ee6e54e2-3009-41a1-a617-d2d4dc6166d7-kube-api-access-qs6bj\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:38:16.219900 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:16.219858 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee6e54e2-3009-41a1-a617-d2d4dc6166d7-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:38:16.928695 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:16.928669 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c6e4f9a-z2m2v" Nov 03 03:38:16.928918 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:16.928713 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c6e4f9a-z2m2v" event={"ID":"ee6e54e2-3009-41a1-a617-d2d4dc6166d7","Type":"ContainerDied","Data":"7a8d5ac3ea542d1c68ae964af1dec9e5e5f5630acd754d3d8ead7340015d81f0"} Nov 03 03:38:16.928918 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:16.928756 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a8d5ac3ea542d1c68ae964af1dec9e5e5f5630acd754d3d8ead7340015d81f0" Nov 03 03:38:23.431155 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:23.431119 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e7b8ba1-5zr5h"] Nov 03 03:38:23.431606 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:23.431497 5605 cpu_manager.go:414] "containerMap: removing container" podUID="ee6e54e2-3009-41a1-a617-d2d4dc6166d7" containerName="revoke-keys" Nov 03 03:38:23.431606 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:23.431509 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee6e54e2-3009-41a1-a617-d2d4dc6166d7" containerName="revoke-keys" Nov 03 03:38:23.431606 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:23.431572 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee6e54e2-3009-41a1-a617-d2d4dc6166d7" containerName="revoke-keys" Nov 03 03:38:23.434638 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:23.434616 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e7b8ba1-5zr5h" Nov 03 03:38:23.483777 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:23.483744 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:38:23.488266 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:23.488239 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eee62b00-65d0-42a7-8a77-2d1a1d8bf7c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e7b8ba1-5zr5h\" (UID: \"eee62b00-65d0-42a7-8a77-2d1a1d8bf7c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e7b8ba1-5zr5h" Nov 03 03:38:23.488408 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:23.488281 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdfw9\" (UniqueName: \"kubernetes.io/projected/eee62b00-65d0-42a7-8a77-2d1a1d8bf7c4-kube-api-access-mdfw9\") pod \"maas-api-revoke-sub-1e7b8ba1-5zr5h\" (UID: \"eee62b00-65d0-42a7-8a77-2d1a1d8bf7c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e7b8ba1-5zr5h" Nov 03 03:38:23.488408 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:23.488383 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eee62b00-65d0-42a7-8a77-2d1a1d8bf7c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e7b8ba1-5zr5h\" (UID: \"eee62b00-65d0-42a7-8a77-2d1a1d8bf7c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e7b8ba1-5zr5h" Nov 03 03:38:23.530436 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:23.530399 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:38:23.530436 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:23.530399 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sb4qh\"" Nov 03 03:38:23.530651 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:23.530463 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:38:23.531773 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:23.531754 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e7b8ba1-5zr5h"] Nov 03 03:38:23.589839 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:23.589796 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eee62b00-65d0-42a7-8a77-2d1a1d8bf7c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e7b8ba1-5zr5h\" (UID: \"eee62b00-65d0-42a7-8a77-2d1a1d8bf7c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e7b8ba1-5zr5h" Nov 03 03:38:23.589839 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:23.589851 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdfw9\" (UniqueName: \"kubernetes.io/projected/eee62b00-65d0-42a7-8a77-2d1a1d8bf7c4-kube-api-access-mdfw9\") pod \"maas-api-revoke-sub-1e7b8ba1-5zr5h\" (UID: \"eee62b00-65d0-42a7-8a77-2d1a1d8bf7c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e7b8ba1-5zr5h" Nov 03 03:38:23.590089 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:23.589894 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eee62b00-65d0-42a7-8a77-2d1a1d8bf7c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e7b8ba1-5zr5h\" (UID: \"eee62b00-65d0-42a7-8a77-2d1a1d8bf7c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e7b8ba1-5zr5h" Nov 03 03:38:23.590699 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:23.590671 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eee62b00-65d0-42a7-8a77-2d1a1d8bf7c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e7b8ba1-5zr5h\" (UID: \"eee62b00-65d0-42a7-8a77-2d1a1d8bf7c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e7b8ba1-5zr5h" Nov 03 03:38:23.592423 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:23.592395 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eee62b00-65d0-42a7-8a77-2d1a1d8bf7c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e7b8ba1-5zr5h\" (UID: \"eee62b00-65d0-42a7-8a77-2d1a1d8bf7c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e7b8ba1-5zr5h" Nov 03 03:38:23.643487 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:23.643454 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdfw9\" (UniqueName: \"kubernetes.io/projected/eee62b00-65d0-42a7-8a77-2d1a1d8bf7c4-kube-api-access-mdfw9\") pod \"maas-api-revoke-sub-1e7b8ba1-5zr5h\" (UID: \"eee62b00-65d0-42a7-8a77-2d1a1d8bf7c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e7b8ba1-5zr5h" Nov 03 03:38:23.747392 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:23.747291 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e7b8ba1-5zr5h" Nov 03 03:38:23.911695 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:38:23.911658 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:38:24.112522 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:24.112487 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e7b8ba1-5zr5h"] Nov 03 03:38:24.119000 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:38:24.118963 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeee62b00_65d0_42a7_8a77_2d1a1d8bf7c4.slice/crio-402a81e47989f26dc0c2503783d68f4b19e1538746bb48373915f7ce3d3a1ab6 WatchSource:0}: Error finding container 402a81e47989f26dc0c2503783d68f4b19e1538746bb48373915f7ce3d3a1ab6: Status 404 returned error can't find the container with id 402a81e47989f26dc0c2503783d68f4b19e1538746bb48373915f7ce3d3a1ab6 Nov 03 03:38:24.967694 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:24.967654 5605 generic.go:356] "Generic (PLEG): container finished" podID="eee62b00-65d0-42a7-8a77-2d1a1d8bf7c4" containerID="9cf3bedf69f7babb9c69647a6ee231d99804f5858f52dd5e3d1d37b49b76d13b" exitCode=0 Nov 03 03:38:24.968134 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:24.967739 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e7b8ba1-5zr5h" event={"ID":"eee62b00-65d0-42a7-8a77-2d1a1d8bf7c4","Type":"ContainerDied","Data":"9cf3bedf69f7babb9c69647a6ee231d99804f5858f52dd5e3d1d37b49b76d13b"} Nov 03 03:38:24.968134 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:24.967810 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e7b8ba1-5zr5h" event={"ID":"eee62b00-65d0-42a7-8a77-2d1a1d8bf7c4","Type":"ContainerStarted","Data":"402a81e47989f26dc0c2503783d68f4b19e1538746bb48373915f7ce3d3a1ab6"} Nov 03 03:38:26.100256 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:26.100232 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e7b8ba1-5zr5h" Nov 03 03:38:26.209744 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:26.209712 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eee62b00-65d0-42a7-8a77-2d1a1d8bf7c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eee62b00-65d0-42a7-8a77-2d1a1d8bf7c4-maas-api-xks-ca\") pod \"eee62b00-65d0-42a7-8a77-2d1a1d8bf7c4\" (UID: \"eee62b00-65d0-42a7-8a77-2d1a1d8bf7c4\") " Nov 03 03:38:26.209981 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:26.209853 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eee62b00-65d0-42a7-8a77-2d1a1d8bf7c4-kube-api-access-mdfw9\" (UniqueName: \"kubernetes.io/projected/eee62b00-65d0-42a7-8a77-2d1a1d8bf7c4-kube-api-access-mdfw9\") pod \"eee62b00-65d0-42a7-8a77-2d1a1d8bf7c4\" (UID: \"eee62b00-65d0-42a7-8a77-2d1a1d8bf7c4\") " Nov 03 03:38:26.209981 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:26.209884 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eee62b00-65d0-42a7-8a77-2d1a1d8bf7c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eee62b00-65d0-42a7-8a77-2d1a1d8bf7c4-maas-api-openshift-ca\") pod \"eee62b00-65d0-42a7-8a77-2d1a1d8bf7c4\" (UID: \"eee62b00-65d0-42a7-8a77-2d1a1d8bf7c4\") " Nov 03 03:38:26.210275 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:26.210248 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eee62b00-65d0-42a7-8a77-2d1a1d8bf7c4-maas-api-openshift-ca" pod "eee62b00-65d0-42a7-8a77-2d1a1d8bf7c4" (UID: "eee62b00-65d0-42a7-8a77-2d1a1d8bf7c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:38:26.211970 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:26.211945 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eee62b00-65d0-42a7-8a77-2d1a1d8bf7c4-kube-api-access-mdfw9" pod "eee62b00-65d0-42a7-8a77-2d1a1d8bf7c4" (UID: "eee62b00-65d0-42a7-8a77-2d1a1d8bf7c4"). InnerVolumeSpecName "kube-api-access-mdfw9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:38:26.212237 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:26.212212 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eee62b00-65d0-42a7-8a77-2d1a1d8bf7c4-maas-api-xks-ca" pod "eee62b00-65d0-42a7-8a77-2d1a1d8bf7c4" (UID: "eee62b00-65d0-42a7-8a77-2d1a1d8bf7c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:38:26.310836 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:26.310798 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdfw9\" (UniqueName: \"kubernetes.io/projected/eee62b00-65d0-42a7-8a77-2d1a1d8bf7c4-kube-api-access-mdfw9\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:38:26.310836 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:26.310833 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eee62b00-65d0-42a7-8a77-2d1a1d8bf7c4-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:38:26.311063 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:26.310844 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eee62b00-65d0-42a7-8a77-2d1a1d8bf7c4-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:38:26.977146 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:26.977104 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e7b8ba1-5zr5h" event={"ID":"eee62b00-65d0-42a7-8a77-2d1a1d8bf7c4","Type":"ContainerDied","Data":"402a81e47989f26dc0c2503783d68f4b19e1538746bb48373915f7ce3d3a1ab6"} Nov 03 03:38:26.977348 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:26.977160 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="402a81e47989f26dc0c2503783d68f4b19e1538746bb48373915f7ce3d3a1ab6" Nov 03 03:38:26.977348 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:26.977112 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e7b8ba1-5zr5h" Nov 03 03:38:36.153796 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:36.153760 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:38:48.058006 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:48.057926 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:38:53.912529 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:38:53.912488 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:38:57.015345 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:57.015303 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fc94408-m8qwl"] Nov 03 03:38:57.015913 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:57.015876 5605 cpu_manager.go:414] "containerMap: removing container" podUID="eee62b00-65d0-42a7-8a77-2d1a1d8bf7c4" containerName="revoke-keys" Nov 03 03:38:57.015913 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:57.015896 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eee62b00-65d0-42a7-8a77-2d1a1d8bf7c4" containerName="revoke-keys" Nov 03 03:38:57.016046 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:57.016001 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="eee62b00-65d0-42a7-8a77-2d1a1d8bf7c4" containerName="revoke-keys" Nov 03 03:38:57.019374 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:57.019337 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fc94408-m8qwl" Nov 03 03:38:57.026307 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:57.026282 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fcbfb33-bfd4-401c-9054-bc3abd0ea4e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9fc94408-m8qwl\" (UID: \"0fcbfb33-bfd4-401c-9054-bc3abd0ea4e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fc94408-m8qwl" Nov 03 03:38:57.026468 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:57.026331 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m54m\" (UniqueName: \"kubernetes.io/projected/0fcbfb33-bfd4-401c-9054-bc3abd0ea4e3-kube-api-access-6m54m\") pod \"maas-api-revoke-sub-9fc94408-m8qwl\" (UID: \"0fcbfb33-bfd4-401c-9054-bc3abd0ea4e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fc94408-m8qwl" Nov 03 03:38:57.026529 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:57.026484 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fcbfb33-bfd4-401c-9054-bc3abd0ea4e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9fc94408-m8qwl\" (UID: \"0fcbfb33-bfd4-401c-9054-bc3abd0ea4e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fc94408-m8qwl" Nov 03 03:38:57.068605 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:57.068571 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:38:57.068605 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:57.068588 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:38:57.115190 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:57.115157 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:38:57.115417 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:57.115398 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sb4qh\"" Nov 03 03:38:57.116711 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:57.116689 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fc94408-m8qwl"] Nov 03 03:38:57.126868 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:57.126841 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fcbfb33-bfd4-401c-9054-bc3abd0ea4e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9fc94408-m8qwl\" (UID: \"0fcbfb33-bfd4-401c-9054-bc3abd0ea4e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fc94408-m8qwl" Nov 03 03:38:57.127001 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:57.126885 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m54m\" (UniqueName: \"kubernetes.io/projected/0fcbfb33-bfd4-401c-9054-bc3abd0ea4e3-kube-api-access-6m54m\") pod \"maas-api-revoke-sub-9fc94408-m8qwl\" (UID: \"0fcbfb33-bfd4-401c-9054-bc3abd0ea4e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fc94408-m8qwl" Nov 03 03:38:57.127046 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:57.126995 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fcbfb33-bfd4-401c-9054-bc3abd0ea4e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9fc94408-m8qwl\" (UID: \"0fcbfb33-bfd4-401c-9054-bc3abd0ea4e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fc94408-m8qwl" Nov 03 03:38:57.127648 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:57.127624 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fcbfb33-bfd4-401c-9054-bc3abd0ea4e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9fc94408-m8qwl\" (UID: \"0fcbfb33-bfd4-401c-9054-bc3abd0ea4e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fc94408-m8qwl" Nov 03 03:38:57.129330 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:57.129308 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fcbfb33-bfd4-401c-9054-bc3abd0ea4e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9fc94408-m8qwl\" (UID: \"0fcbfb33-bfd4-401c-9054-bc3abd0ea4e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fc94408-m8qwl" Nov 03 03:38:57.181058 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:57.181019 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m54m\" (UniqueName: \"kubernetes.io/projected/0fcbfb33-bfd4-401c-9054-bc3abd0ea4e3-kube-api-access-6m54m\") pod \"maas-api-revoke-sub-9fc94408-m8qwl\" (UID: \"0fcbfb33-bfd4-401c-9054-bc3abd0ea4e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fc94408-m8qwl" Nov 03 03:38:57.333941 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:57.333900 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fc94408-m8qwl" Nov 03 03:38:57.700579 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:57.700489 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fc94408-m8qwl"] Nov 03 03:38:57.707106 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:38:57.707081 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fcbfb33_bfd4_401c_9054_bc3abd0ea4e3.slice/crio-ed961e78a35bc41b81b83f8beed88be8a36885afd69b43f918bad0f5f3e6ab8e WatchSource:0}: Error finding container ed961e78a35bc41b81b83f8beed88be8a36885afd69b43f918bad0f5f3e6ab8e: Status 404 returned error can't find the container with id ed961e78a35bc41b81b83f8beed88be8a36885afd69b43f918bad0f5f3e6ab8e Nov 03 03:38:58.132188 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:58.132136 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fc94408-m8qwl" event={"ID":"0fcbfb33-bfd4-401c-9054-bc3abd0ea4e3","Type":"ContainerStarted","Data":"f0c7cf2509a22694cd7adcabcd8fcfc43e0db6a2be73f6a591f17273d0d002f8"} Nov 03 03:38:58.132615 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:58.132211 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fc94408-m8qwl" event={"ID":"0fcbfb33-bfd4-401c-9054-bc3abd0ea4e3","Type":"ContainerStarted","Data":"ed961e78a35bc41b81b83f8beed88be8a36885afd69b43f918bad0f5f3e6ab8e"} Nov 03 03:38:58.240394 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:58.240259 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fc94408-m8qwl" podStartSLOduration=2.240246308 podStartE2EDuration="2.240246308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:38:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:38:58.239780828 +0000 UTC m=+6415.498978978" watchObservedRunningTime="2026-11-03 03:38:58.240246308 +0000 UTC m=+6415.499444462" Nov 03 03:38:59.137541 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:59.137505 5605 generic.go:356] "Generic (PLEG): container finished" podID="0fcbfb33-bfd4-401c-9054-bc3abd0ea4e3" containerID="f0c7cf2509a22694cd7adcabcd8fcfc43e0db6a2be73f6a591f17273d0d002f8" exitCode=0 Nov 03 03:38:59.137950 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:38:59.137581 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fc94408-m8qwl" event={"ID":"0fcbfb33-bfd4-401c-9054-bc3abd0ea4e3","Type":"ContainerDied","Data":"f0c7cf2509a22694cd7adcabcd8fcfc43e0db6a2be73f6a591f17273d0d002f8"} Nov 03 03:39:00.267921 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:00.267888 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fc94408-m8qwl" Nov 03 03:39:00.351868 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:00.351833 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fcbfb33-bfd4-401c-9054-bc3abd0ea4e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fcbfb33-bfd4-401c-9054-bc3abd0ea4e3-maas-api-xks-ca\") pod \"0fcbfb33-bfd4-401c-9054-bc3abd0ea4e3\" (UID: \"0fcbfb33-bfd4-401c-9054-bc3abd0ea4e3\") " Nov 03 03:39:00.352052 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:00.351890 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fcbfb33-bfd4-401c-9054-bc3abd0ea4e3-kube-api-access-6m54m\" (UniqueName: \"kubernetes.io/projected/0fcbfb33-bfd4-401c-9054-bc3abd0ea4e3-kube-api-access-6m54m\") pod \"0fcbfb33-bfd4-401c-9054-bc3abd0ea4e3\" (UID: \"0fcbfb33-bfd4-401c-9054-bc3abd0ea4e3\") " Nov 03 03:39:00.352052 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:00.351938 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fcbfb33-bfd4-401c-9054-bc3abd0ea4e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fcbfb33-bfd4-401c-9054-bc3abd0ea4e3-maas-api-openshift-ca\") pod \"0fcbfb33-bfd4-401c-9054-bc3abd0ea4e3\" (UID: \"0fcbfb33-bfd4-401c-9054-bc3abd0ea4e3\") " Nov 03 03:39:00.352402 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:00.352342 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fcbfb33-bfd4-401c-9054-bc3abd0ea4e3-maas-api-openshift-ca" pod "0fcbfb33-bfd4-401c-9054-bc3abd0ea4e3" (UID: "0fcbfb33-bfd4-401c-9054-bc3abd0ea4e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:39:00.354052 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:00.354026 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fcbfb33-bfd4-401c-9054-bc3abd0ea4e3-maas-api-xks-ca" pod "0fcbfb33-bfd4-401c-9054-bc3abd0ea4e3" (UID: "0fcbfb33-bfd4-401c-9054-bc3abd0ea4e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:39:00.354152 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:00.354048 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fcbfb33-bfd4-401c-9054-bc3abd0ea4e3-kube-api-access-6m54m" pod "0fcbfb33-bfd4-401c-9054-bc3abd0ea4e3" (UID: "0fcbfb33-bfd4-401c-9054-bc3abd0ea4e3"). InnerVolumeSpecName "kube-api-access-6m54m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:39:00.452621 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:00.452535 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m54m\" (UniqueName: \"kubernetes.io/projected/0fcbfb33-bfd4-401c-9054-bc3abd0ea4e3-kube-api-access-6m54m\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:39:00.452621 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:00.452566 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fcbfb33-bfd4-401c-9054-bc3abd0ea4e3-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:39:00.452621 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:00.452576 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fcbfb33-bfd4-401c-9054-bc3abd0ea4e3-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:39:01.148142 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:01.148049 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fc94408-m8qwl" event={"ID":"0fcbfb33-bfd4-401c-9054-bc3abd0ea4e3","Type":"ContainerDied","Data":"ed961e78a35bc41b81b83f8beed88be8a36885afd69b43f918bad0f5f3e6ab8e"} Nov 03 03:39:01.148142 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:01.148081 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fc94408-m8qwl" Nov 03 03:39:01.148142 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:01.148099 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed961e78a35bc41b81b83f8beed88be8a36885afd69b43f918bad0f5f3e6ab8e" Nov 03 03:39:09.662834 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:09.662799 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:39:21.470206 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:21.470166 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:39:24.010338 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:39:24.010299 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:39:26.041959 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:26.041914 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6c94f3e-679hl"] Nov 03 03:39:26.088729 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:26.088680 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6c94f3e-679hl"] Nov 03 03:39:26.849737 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:26.849701 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec0cd6b7-f158-4057-a2d1-b57a506c360f" path="/var/lib/kubelet/pods/ec0cd6b7-f158-4057-a2d1-b57a506c360f/volumes" Nov 03 03:39:39.841502 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:39.841464 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20d6150f-4ztgk"] Nov 03 03:39:39.841947 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:39.841838 5605 cpu_manager.go:414] "containerMap: removing container" podUID="0fcbfb33-bfd4-401c-9054-bc3abd0ea4e3" containerName="revoke-keys" Nov 03 03:39:39.841947 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:39.841849 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fcbfb33-bfd4-401c-9054-bc3abd0ea4e3" containerName="revoke-keys" Nov 03 03:39:39.841947 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:39.841918 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fcbfb33-bfd4-401c-9054-bc3abd0ea4e3" containerName="revoke-keys" Nov 03 03:39:39.845485 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:39.845456 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20d6150f-4ztgk" Nov 03 03:39:39.894747 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:39.894717 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:39:39.941349 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:39.941305 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:39:39.941349 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:39.941328 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:39:39.941580 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:39.941428 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sb4qh\"" Nov 03 03:39:39.942854 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:39.942836 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20d6150f-4ztgk"] Nov 03 03:39:39.992531 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:39.992495 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f270da8f-bf1a-46b7-8d66-3cba86c1f267-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20d6150f-4ztgk\" (UID: \"f270da8f-bf1a-46b7-8d66-3cba86c1f267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20d6150f-4ztgk" Nov 03 03:39:39.992724 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:39.992556 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f270da8f-bf1a-46b7-8d66-3cba86c1f267-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20d6150f-4ztgk\" (UID: \"f270da8f-bf1a-46b7-8d66-3cba86c1f267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20d6150f-4ztgk" Nov 03 03:39:39.992768 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:39.992719 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzz2q\" (UniqueName: \"kubernetes.io/projected/f270da8f-bf1a-46b7-8d66-3cba86c1f267-kube-api-access-kzz2q\") pod \"maas-api-revoke-sub-20d6150f-4ztgk\" (UID: \"f270da8f-bf1a-46b7-8d66-3cba86c1f267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20d6150f-4ztgk" Nov 03 03:39:40.093606 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:40.093514 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f270da8f-bf1a-46b7-8d66-3cba86c1f267-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20d6150f-4ztgk\" (UID: \"f270da8f-bf1a-46b7-8d66-3cba86c1f267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20d6150f-4ztgk" Nov 03 03:39:40.093606 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:40.093569 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f270da8f-bf1a-46b7-8d66-3cba86c1f267-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20d6150f-4ztgk\" (UID: \"f270da8f-bf1a-46b7-8d66-3cba86c1f267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20d6150f-4ztgk" Nov 03 03:39:40.093874 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:40.093686 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzz2q\" (UniqueName: \"kubernetes.io/projected/f270da8f-bf1a-46b7-8d66-3cba86c1f267-kube-api-access-kzz2q\") pod \"maas-api-revoke-sub-20d6150f-4ztgk\" (UID: \"f270da8f-bf1a-46b7-8d66-3cba86c1f267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20d6150f-4ztgk" Nov 03 03:39:40.100447 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:40.094721 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f270da8f-bf1a-46b7-8d66-3cba86c1f267-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20d6150f-4ztgk\" (UID: \"f270da8f-bf1a-46b7-8d66-3cba86c1f267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20d6150f-4ztgk" Nov 03 03:39:40.100447 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:40.097090 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f270da8f-bf1a-46b7-8d66-3cba86c1f267-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20d6150f-4ztgk\" (UID: \"f270da8f-bf1a-46b7-8d66-3cba86c1f267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20d6150f-4ztgk" Nov 03 03:39:40.147621 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:40.147586 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzz2q\" (UniqueName: \"kubernetes.io/projected/f270da8f-bf1a-46b7-8d66-3cba86c1f267-kube-api-access-kzz2q\") pod \"maas-api-revoke-sub-20d6150f-4ztgk\" (UID: \"f270da8f-bf1a-46b7-8d66-3cba86c1f267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20d6150f-4ztgk" Nov 03 03:39:40.157284 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:40.157253 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20d6150f-4ztgk" Nov 03 03:39:40.525162 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:40.525074 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20d6150f-4ztgk"] Nov 03 03:39:40.531669 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:39:40.531635 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf270da8f_bf1a_46b7_8d66_3cba86c1f267.slice/crio-9851c72713a4c14c104e549dac030441795c65a2cfcef46439ba979d195bbab8 WatchSource:0}: Error finding container 9851c72713a4c14c104e549dac030441795c65a2cfcef46439ba979d195bbab8: Status 404 returned error can't find the container with id 9851c72713a4c14c104e549dac030441795c65a2cfcef46439ba979d195bbab8 Nov 03 03:39:41.363724 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:41.363685 5605 generic.go:356] "Generic (PLEG): container finished" podID="f270da8f-bf1a-46b7-8d66-3cba86c1f267" containerID="30dace4febdcc77048e3383133a9c8fee14d83d4206c60886c7c3cadb21e0539" exitCode=0 Nov 03 03:39:41.363724 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:41.363717 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20d6150f-4ztgk" event={"ID":"f270da8f-bf1a-46b7-8d66-3cba86c1f267","Type":"ContainerDied","Data":"30dace4febdcc77048e3383133a9c8fee14d83d4206c60886c7c3cadb21e0539"} Nov 03 03:39:41.364177 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:41.363769 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20d6150f-4ztgk" event={"ID":"f270da8f-bf1a-46b7-8d66-3cba86c1f267","Type":"ContainerStarted","Data":"9851c72713a4c14c104e549dac030441795c65a2cfcef46439ba979d195bbab8"} Nov 03 03:39:42.498756 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:42.498733 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20d6150f-4ztgk" Nov 03 03:39:42.617316 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:42.617278 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f270da8f-bf1a-46b7-8d66-3cba86c1f267-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f270da8f-bf1a-46b7-8d66-3cba86c1f267-maas-api-openshift-ca\") pod \"f270da8f-bf1a-46b7-8d66-3cba86c1f267\" (UID: \"f270da8f-bf1a-46b7-8d66-3cba86c1f267\") " Nov 03 03:39:42.617517 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:42.617399 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f270da8f-bf1a-46b7-8d66-3cba86c1f267-kube-api-access-kzz2q\" (UniqueName: \"kubernetes.io/projected/f270da8f-bf1a-46b7-8d66-3cba86c1f267-kube-api-access-kzz2q\") pod \"f270da8f-bf1a-46b7-8d66-3cba86c1f267\" (UID: \"f270da8f-bf1a-46b7-8d66-3cba86c1f267\") " Nov 03 03:39:42.617517 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:42.617486 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f270da8f-bf1a-46b7-8d66-3cba86c1f267-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f270da8f-bf1a-46b7-8d66-3cba86c1f267-maas-api-xks-ca\") pod \"f270da8f-bf1a-46b7-8d66-3cba86c1f267\" (UID: \"f270da8f-bf1a-46b7-8d66-3cba86c1f267\") " Nov 03 03:39:42.618169 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:42.618098 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f270da8f-bf1a-46b7-8d66-3cba86c1f267-maas-api-openshift-ca" pod "f270da8f-bf1a-46b7-8d66-3cba86c1f267" (UID: "f270da8f-bf1a-46b7-8d66-3cba86c1f267"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:39:42.619633 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:42.619608 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f270da8f-bf1a-46b7-8d66-3cba86c1f267-maas-api-xks-ca" pod "f270da8f-bf1a-46b7-8d66-3cba86c1f267" (UID: "f270da8f-bf1a-46b7-8d66-3cba86c1f267"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:39:42.619746 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:42.619654 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f270da8f-bf1a-46b7-8d66-3cba86c1f267-kube-api-access-kzz2q" pod "f270da8f-bf1a-46b7-8d66-3cba86c1f267" (UID: "f270da8f-bf1a-46b7-8d66-3cba86c1f267"). InnerVolumeSpecName "kube-api-access-kzz2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:39:42.718297 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:42.718259 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzz2q\" (UniqueName: \"kubernetes.io/projected/f270da8f-bf1a-46b7-8d66-3cba86c1f267-kube-api-access-kzz2q\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:39:42.718297 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:42.718291 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f270da8f-bf1a-46b7-8d66-3cba86c1f267-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:39:42.718297 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:42.718301 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f270da8f-bf1a-46b7-8d66-3cba86c1f267-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:39:43.373730 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:43.373691 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20d6150f-4ztgk" event={"ID":"f270da8f-bf1a-46b7-8d66-3cba86c1f267","Type":"ContainerDied","Data":"9851c72713a4c14c104e549dac030441795c65a2cfcef46439ba979d195bbab8"} Nov 03 03:39:43.373920 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:43.373733 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20d6150f-4ztgk" Nov 03 03:39:43.373920 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:43.373743 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9851c72713a4c14c104e549dac030441795c65a2cfcef46439ba979d195bbab8" Nov 03 03:39:50.652527 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:50.652493 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfb5a0e2-hm9wv"] Nov 03 03:39:50.652943 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:50.652857 5605 cpu_manager.go:414] "containerMap: removing container" podUID="f270da8f-bf1a-46b7-8d66-3cba86c1f267" containerName="revoke-keys" Nov 03 03:39:50.652943 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:50.652868 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f270da8f-bf1a-46b7-8d66-3cba86c1f267" containerName="revoke-keys" Nov 03 03:39:50.652943 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:50.652926 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="f270da8f-bf1a-46b7-8d66-3cba86c1f267" containerName="revoke-keys" Nov 03 03:39:50.656120 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:50.656095 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfb5a0e2-hm9wv" Nov 03 03:39:50.705280 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:50.705234 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:39:50.705497 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:50.705433 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sb4qh\"" Nov 03 03:39:50.705563 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:50.705517 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:39:50.752084 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:50.752046 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:39:50.753606 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:50.753582 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfb5a0e2-hm9wv"] Nov 03 03:39:50.797718 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:50.797681 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0fa70c6-7511-4f78-9775-704c55afca0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dfb5a0e2-hm9wv\" (UID: \"c0fa70c6-7511-4f78-9775-704c55afca0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfb5a0e2-hm9wv" Nov 03 03:39:50.797718 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:50.797720 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0fa70c6-7511-4f78-9775-704c55afca0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dfb5a0e2-hm9wv\" (UID: \"c0fa70c6-7511-4f78-9775-704c55afca0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfb5a0e2-hm9wv" Nov 03 03:39:50.797935 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:50.797755 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4jqx\" (UniqueName: \"kubernetes.io/projected/c0fa70c6-7511-4f78-9775-704c55afca0b-kube-api-access-f4jqx\") pod \"maas-api-revoke-sub-dfb5a0e2-hm9wv\" (UID: \"c0fa70c6-7511-4f78-9775-704c55afca0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfb5a0e2-hm9wv" Nov 03 03:39:50.898683 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:50.898647 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0fa70c6-7511-4f78-9775-704c55afca0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dfb5a0e2-hm9wv\" (UID: \"c0fa70c6-7511-4f78-9775-704c55afca0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfb5a0e2-hm9wv" Nov 03 03:39:50.898683 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:50.898685 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0fa70c6-7511-4f78-9775-704c55afca0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dfb5a0e2-hm9wv\" (UID: \"c0fa70c6-7511-4f78-9775-704c55afca0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfb5a0e2-hm9wv" Nov 03 03:39:50.898905 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:50.898723 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4jqx\" (UniqueName: \"kubernetes.io/projected/c0fa70c6-7511-4f78-9775-704c55afca0b-kube-api-access-f4jqx\") pod \"maas-api-revoke-sub-dfb5a0e2-hm9wv\" (UID: \"c0fa70c6-7511-4f78-9775-704c55afca0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfb5a0e2-hm9wv" Nov 03 03:39:50.899527 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:50.899497 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0fa70c6-7511-4f78-9775-704c55afca0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dfb5a0e2-hm9wv\" (UID: \"c0fa70c6-7511-4f78-9775-704c55afca0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfb5a0e2-hm9wv" Nov 03 03:39:50.901103 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:50.901077 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0fa70c6-7511-4f78-9775-704c55afca0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dfb5a0e2-hm9wv\" (UID: \"c0fa70c6-7511-4f78-9775-704c55afca0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfb5a0e2-hm9wv" Nov 03 03:39:50.953152 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:50.953075 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4jqx\" (UniqueName: \"kubernetes.io/projected/c0fa70c6-7511-4f78-9775-704c55afca0b-kube-api-access-f4jqx\") pod \"maas-api-revoke-sub-dfb5a0e2-hm9wv\" (UID: \"c0fa70c6-7511-4f78-9775-704c55afca0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfb5a0e2-hm9wv" Nov 03 03:39:50.969870 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:50.969843 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfb5a0e2-hm9wv" Nov 03 03:39:51.334045 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:51.334015 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfb5a0e2-hm9wv"] Nov 03 03:39:51.341058 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:39:51.341030 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0fa70c6_7511_4f78_9775_704c55afca0b.slice/crio-38a4a06479f50c1e26c2cdec319da0f1e0d119158a752adef7f6fe42d9eb70ec WatchSource:0}: Error finding container 38a4a06479f50c1e26c2cdec319da0f1e0d119158a752adef7f6fe42d9eb70ec: Status 404 returned error can't find the container with id 38a4a06479f50c1e26c2cdec319da0f1e0d119158a752adef7f6fe42d9eb70ec Nov 03 03:39:51.410893 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:51.410862 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfb5a0e2-hm9wv" event={"ID":"c0fa70c6-7511-4f78-9775-704c55afca0b","Type":"ContainerStarted","Data":"38a4a06479f50c1e26c2cdec319da0f1e0d119158a752adef7f6fe42d9eb70ec"} Nov 03 03:39:52.419709 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:52.419676 5605 generic.go:356] "Generic (PLEG): container finished" podID="c0fa70c6-7511-4f78-9775-704c55afca0b" containerID="8eb9f2c027eb4566f68abf1ea07921b4852752f6f9ea83ea3e9a048e97afbd26" exitCode=0 Nov 03 03:39:52.420346 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:52.419774 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfb5a0e2-hm9wv" event={"ID":"c0fa70c6-7511-4f78-9775-704c55afca0b","Type":"ContainerDied","Data":"8eb9f2c027eb4566f68abf1ea07921b4852752f6f9ea83ea3e9a048e97afbd26"} Nov 03 03:39:52.547473 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:52.547388 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:39:53.553216 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:53.553180 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfb5a0e2-hm9wv" Nov 03 03:39:53.627037 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:53.627001 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0fa70c6-7511-4f78-9775-704c55afca0b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0fa70c6-7511-4f78-9775-704c55afca0b-maas-api-openshift-ca\") pod \"c0fa70c6-7511-4f78-9775-704c55afca0b\" (UID: \"c0fa70c6-7511-4f78-9775-704c55afca0b\") " Nov 03 03:39:53.627251 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:53.627067 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0fa70c6-7511-4f78-9775-704c55afca0b-kube-api-access-f4jqx\" (UniqueName: \"kubernetes.io/projected/c0fa70c6-7511-4f78-9775-704c55afca0b-kube-api-access-f4jqx\") pod \"c0fa70c6-7511-4f78-9775-704c55afca0b\" (UID: \"c0fa70c6-7511-4f78-9775-704c55afca0b\") " Nov 03 03:39:53.627251 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:53.627111 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0fa70c6-7511-4f78-9775-704c55afca0b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0fa70c6-7511-4f78-9775-704c55afca0b-maas-api-xks-ca\") pod \"c0fa70c6-7511-4f78-9775-704c55afca0b\" (UID: \"c0fa70c6-7511-4f78-9775-704c55afca0b\") " Nov 03 03:39:53.627460 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:53.627436 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0fa70c6-7511-4f78-9775-704c55afca0b-maas-api-openshift-ca" pod "c0fa70c6-7511-4f78-9775-704c55afca0b" (UID: "c0fa70c6-7511-4f78-9775-704c55afca0b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:39:53.629176 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:53.629157 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0fa70c6-7511-4f78-9775-704c55afca0b-maas-api-xks-ca" pod "c0fa70c6-7511-4f78-9775-704c55afca0b" (UID: "c0fa70c6-7511-4f78-9775-704c55afca0b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:39:53.629235 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:53.629216 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0fa70c6-7511-4f78-9775-704c55afca0b-kube-api-access-f4jqx" pod "c0fa70c6-7511-4f78-9775-704c55afca0b" (UID: "c0fa70c6-7511-4f78-9775-704c55afca0b"). InnerVolumeSpecName "kube-api-access-f4jqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:39:53.728094 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:53.728007 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4jqx\" (UniqueName: \"kubernetes.io/projected/c0fa70c6-7511-4f78-9775-704c55afca0b-kube-api-access-f4jqx\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:39:53.728094 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:53.728039 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0fa70c6-7511-4f78-9775-704c55afca0b-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:39:53.728094 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:53.728048 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0fa70c6-7511-4f78-9775-704c55afca0b-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:39:53.911987 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:39:53.911950 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:39:54.431368 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:54.431332 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfb5a0e2-hm9wv" Nov 03 03:39:54.431573 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:54.431384 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfb5a0e2-hm9wv" event={"ID":"c0fa70c6-7511-4f78-9775-704c55afca0b","Type":"ContainerDied","Data":"38a4a06479f50c1e26c2cdec319da0f1e0d119158a752adef7f6fe42d9eb70ec"} Nov 03 03:39:54.431573 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:39:54.431446 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38a4a06479f50c1e26c2cdec319da0f1e0d119158a752adef7f6fe42d9eb70ec" Nov 03 03:40:03.429375 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:03.429321 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:40:12.305605 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:12.305382 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-p8fqq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 03:40:12.306301 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:12.306260 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p8fqq" podUID="0bedb6e8-5829-4cc2-a912-35131c941e09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 03:40:12.320492 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:12.320455 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-dbd7q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 03:40:12.320727 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:12.320696 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dbd7q" podUID="dc490604-2e00-49dd-8423-47fc70709678" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 03:40:12.835748 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:12.835697 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-bpbq7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 03:40:12.835969 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:12.835802 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bpbq7" podUID="563c5b47-58ea-4fb8-b9f1-332cf8dae641" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 03:40:15.543559 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:15.543524 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:40:23.912151 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:40:23.912111 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:40:25.042435 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:25.042392 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sfzzw"] Nov 03 03:40:25.089156 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:25.089122 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sfzzw"] Nov 03 03:40:25.824316 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:25.824279 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:40:26.852277 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:26.852232 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b028cc05-9223-4ef0-8147-a0d4df641ff3" path="/var/lib/kubelet/pods/b028cc05-9223-4ef0-8147-a0d4df641ff3/volumes" Nov 03 03:40:28.168382 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:28.167511 5605 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-k449x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 03:40:28.168382 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:28.167654 5605 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k449x" podUID="0378fcbc-4027-4275-b274-2c188f3ff442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 03:40:35.604510 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:35.604469 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-442c374b-lxwqb"] Nov 03 03:40:35.604932 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:35.604817 5605 cpu_manager.go:414] "containerMap: removing container" podUID="c0fa70c6-7511-4f78-9775-704c55afca0b" containerName="revoke-keys" Nov 03 03:40:35.604932 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:35.604828 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0fa70c6-7511-4f78-9775-704c55afca0b" containerName="revoke-keys" Nov 03 03:40:35.604932 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:35.604904 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0fa70c6-7511-4f78-9775-704c55afca0b" containerName="revoke-keys" Nov 03 03:40:35.608324 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:35.608302 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-442c374b-lxwqb" Nov 03 03:40:35.657546 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:35.657510 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:40:35.657724 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:35.657550 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:40:35.672562 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:35.672527 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srnv7\" (UniqueName: \"kubernetes.io/projected/6320e544-6957-4312-9a62-6c1338837e2a-kube-api-access-srnv7\") pod \"maas-api-revoke-sub-442c374b-lxwqb\" (UID: \"6320e544-6957-4312-9a62-6c1338837e2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-442c374b-lxwqb" Nov 03 03:40:35.672711 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:35.672585 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6320e544-6957-4312-9a62-6c1338837e2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-442c374b-lxwqb\" (UID: \"6320e544-6957-4312-9a62-6c1338837e2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-442c374b-lxwqb" Nov 03 03:40:35.672757 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:35.672705 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6320e544-6957-4312-9a62-6c1338837e2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-442c374b-lxwqb\" (UID: \"6320e544-6957-4312-9a62-6c1338837e2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-442c374b-lxwqb" Nov 03 03:40:35.704321 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:35.704290 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sb4qh\"" Nov 03 03:40:35.704321 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:35.704291 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:40:35.706020 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:35.705999 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-442c374b-lxwqb"] Nov 03 03:40:35.773385 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:35.773325 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6320e544-6957-4312-9a62-6c1338837e2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-442c374b-lxwqb\" (UID: \"6320e544-6957-4312-9a62-6c1338837e2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-442c374b-lxwqb" Nov 03 03:40:35.773582 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:35.773429 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srnv7\" (UniqueName: \"kubernetes.io/projected/6320e544-6957-4312-9a62-6c1338837e2a-kube-api-access-srnv7\") pod \"maas-api-revoke-sub-442c374b-lxwqb\" (UID: \"6320e544-6957-4312-9a62-6c1338837e2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-442c374b-lxwqb" Nov 03 03:40:35.773582 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:35.773462 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6320e544-6957-4312-9a62-6c1338837e2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-442c374b-lxwqb\" (UID: \"6320e544-6957-4312-9a62-6c1338837e2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-442c374b-lxwqb" Nov 03 03:40:35.774094 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:35.774073 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6320e544-6957-4312-9a62-6c1338837e2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-442c374b-lxwqb\" (UID: \"6320e544-6957-4312-9a62-6c1338837e2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-442c374b-lxwqb" Nov 03 03:40:35.775671 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:35.775651 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6320e544-6957-4312-9a62-6c1338837e2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-442c374b-lxwqb\" (UID: \"6320e544-6957-4312-9a62-6c1338837e2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-442c374b-lxwqb" Nov 03 03:40:35.827689 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:35.827652 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srnv7\" (UniqueName: \"kubernetes.io/projected/6320e544-6957-4312-9a62-6c1338837e2a-kube-api-access-srnv7\") pod \"maas-api-revoke-sub-442c374b-lxwqb\" (UID: \"6320e544-6957-4312-9a62-6c1338837e2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-442c374b-lxwqb" Nov 03 03:40:35.920978 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:35.920895 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-442c374b-lxwqb" Nov 03 03:40:36.286015 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:36.285918 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-442c374b-lxwqb"] Nov 03 03:40:36.293046 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:40:36.293018 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6320e544_6957_4312_9a62_6c1338837e2a.slice/crio-214c4af07cb36db3ee294631513ad5b5cdd5cee469aea242a6e431a87b598db8 WatchSource:0}: Error finding container 214c4af07cb36db3ee294631513ad5b5cdd5cee469aea242a6e431a87b598db8: Status 404 returned error can't find the container with id 214c4af07cb36db3ee294631513ad5b5cdd5cee469aea242a6e431a87b598db8 Nov 03 03:40:36.697264 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:36.697226 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-442c374b-lxwqb" event={"ID":"6320e544-6957-4312-9a62-6c1338837e2a","Type":"ContainerStarted","Data":"4b3c0d553f11ad9e554ab888c0838d3cdaef4aa9c4f4179f05281a974ebc44e2"} Nov 03 03:40:36.697791 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:36.697279 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-442c374b-lxwqb" event={"ID":"6320e544-6957-4312-9a62-6c1338837e2a","Type":"ContainerStarted","Data":"214c4af07cb36db3ee294631513ad5b5cdd5cee469aea242a6e431a87b598db8"} Nov 03 03:40:36.804811 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:36.804757 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-442c374b-lxwqb" podStartSLOduration=1.804744382 podStartE2EDuration="1.804744382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:40:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:40:36.804049847 +0000 UTC m=+6514.063248009" watchObservedRunningTime="2026-11-03 03:40:36.804744382 +0000 UTC m=+6514.063942536" Nov 03 03:40:37.703278 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:37.703242 5605 generic.go:356] "Generic (PLEG): container finished" podID="6320e544-6957-4312-9a62-6c1338837e2a" containerID="4b3c0d553f11ad9e554ab888c0838d3cdaef4aa9c4f4179f05281a974ebc44e2" exitCode=0 Nov 03 03:40:37.703719 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:37.703291 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-442c374b-lxwqb" event={"ID":"6320e544-6957-4312-9a62-6c1338837e2a","Type":"ContainerDied","Data":"4b3c0d553f11ad9e554ab888c0838d3cdaef4aa9c4f4179f05281a974ebc44e2"} Nov 03 03:40:38.840183 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:38.840155 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-442c374b-lxwqb" Nov 03 03:40:38.904879 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:38.904845 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6320e544-6957-4312-9a62-6c1338837e2a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6320e544-6957-4312-9a62-6c1338837e2a-maas-api-openshift-ca\") pod \"6320e544-6957-4312-9a62-6c1338837e2a\" (UID: \"6320e544-6957-4312-9a62-6c1338837e2a\") " Nov 03 03:40:38.905083 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:38.904940 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6320e544-6957-4312-9a62-6c1338837e2a-kube-api-access-srnv7\" (UniqueName: \"kubernetes.io/projected/6320e544-6957-4312-9a62-6c1338837e2a-kube-api-access-srnv7\") pod \"6320e544-6957-4312-9a62-6c1338837e2a\" (UID: \"6320e544-6957-4312-9a62-6c1338837e2a\") " Nov 03 03:40:38.905083 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:38.904975 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6320e544-6957-4312-9a62-6c1338837e2a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6320e544-6957-4312-9a62-6c1338837e2a-maas-api-xks-ca\") pod \"6320e544-6957-4312-9a62-6c1338837e2a\" (UID: \"6320e544-6957-4312-9a62-6c1338837e2a\") " Nov 03 03:40:38.905280 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:38.905255 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6320e544-6957-4312-9a62-6c1338837e2a-maas-api-openshift-ca" pod "6320e544-6957-4312-9a62-6c1338837e2a" (UID: "6320e544-6957-4312-9a62-6c1338837e2a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:40:38.907103 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:38.907076 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6320e544-6957-4312-9a62-6c1338837e2a-maas-api-xks-ca" pod "6320e544-6957-4312-9a62-6c1338837e2a" (UID: "6320e544-6957-4312-9a62-6c1338837e2a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:40:38.907216 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:38.907096 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6320e544-6957-4312-9a62-6c1338837e2a-kube-api-access-srnv7" pod "6320e544-6957-4312-9a62-6c1338837e2a" (UID: "6320e544-6957-4312-9a62-6c1338837e2a"). InnerVolumeSpecName "kube-api-access-srnv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:40:39.006144 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:39.006056 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6320e544-6957-4312-9a62-6c1338837e2a-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:40:39.006144 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:39.006088 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srnv7\" (UniqueName: \"kubernetes.io/projected/6320e544-6957-4312-9a62-6c1338837e2a-kube-api-access-srnv7\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:40:39.006144 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:39.006099 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6320e544-6957-4312-9a62-6c1338837e2a-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:40:39.713662 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:39.713612 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-442c374b-lxwqb" event={"ID":"6320e544-6957-4312-9a62-6c1338837e2a","Type":"ContainerDied","Data":"214c4af07cb36db3ee294631513ad5b5cdd5cee469aea242a6e431a87b598db8"} Nov 03 03:40:39.713662 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:39.713643 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-442c374b-lxwqb" Nov 03 03:40:39.713893 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:39.713680 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="214c4af07cb36db3ee294631513ad5b5cdd5cee469aea242a6e431a87b598db8" Nov 03 03:40:46.409390 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:46.409340 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-986b3c70-tnprm"] Nov 03 03:40:46.409843 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:46.409748 5605 cpu_manager.go:414] "containerMap: removing container" podUID="6320e544-6957-4312-9a62-6c1338837e2a" containerName="revoke-keys" Nov 03 03:40:46.409843 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:46.409761 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6320e544-6957-4312-9a62-6c1338837e2a" containerName="revoke-keys" Nov 03 03:40:46.409919 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:46.409848 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="6320e544-6957-4312-9a62-6c1338837e2a" containerName="revoke-keys" Nov 03 03:40:46.413147 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:46.413124 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-986b3c70-tnprm" Nov 03 03:40:46.462154 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:46.462123 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:40:46.475700 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:46.475655 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7z4q\" (UniqueName: \"kubernetes.io/projected/21584a67-219d-4646-86b7-90719a04de2b-kube-api-access-c7z4q\") pod \"maas-api-revoke-sub-986b3c70-tnprm\" (UID: \"21584a67-219d-4646-86b7-90719a04de2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-986b3c70-tnprm" Nov 03 03:40:46.475882 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:46.475723 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21584a67-219d-4646-86b7-90719a04de2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-986b3c70-tnprm\" (UID: \"21584a67-219d-4646-86b7-90719a04de2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-986b3c70-tnprm" Nov 03 03:40:46.475882 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:46.475812 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21584a67-219d-4646-86b7-90719a04de2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-986b3c70-tnprm\" (UID: \"21584a67-219d-4646-86b7-90719a04de2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-986b3c70-tnprm" Nov 03 03:40:46.476709 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:46.476680 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sb4qh\"" Nov 03 03:40:46.508830 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:46.508785 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:40:46.508830 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:46.508785 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:40:46.510392 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:46.510342 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-986b3c70-tnprm"] Nov 03 03:40:46.577044 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:46.577004 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21584a67-219d-4646-86b7-90719a04de2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-986b3c70-tnprm\" (UID: \"21584a67-219d-4646-86b7-90719a04de2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-986b3c70-tnprm" Nov 03 03:40:46.577257 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:46.577053 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21584a67-219d-4646-86b7-90719a04de2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-986b3c70-tnprm\" (UID: \"21584a67-219d-4646-86b7-90719a04de2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-986b3c70-tnprm" Nov 03 03:40:46.577257 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:46.577119 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7z4q\" (UniqueName: \"kubernetes.io/projected/21584a67-219d-4646-86b7-90719a04de2b-kube-api-access-c7z4q\") pod \"maas-api-revoke-sub-986b3c70-tnprm\" (UID: \"21584a67-219d-4646-86b7-90719a04de2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-986b3c70-tnprm" Nov 03 03:40:46.577801 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:46.577773 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21584a67-219d-4646-86b7-90719a04de2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-986b3c70-tnprm\" (UID: \"21584a67-219d-4646-86b7-90719a04de2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-986b3c70-tnprm" Nov 03 03:40:46.579482 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:46.579459 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21584a67-219d-4646-86b7-90719a04de2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-986b3c70-tnprm\" (UID: \"21584a67-219d-4646-86b7-90719a04de2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-986b3c70-tnprm" Nov 03 03:40:46.630939 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:46.630908 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7z4q\" (UniqueName: \"kubernetes.io/projected/21584a67-219d-4646-86b7-90719a04de2b-kube-api-access-c7z4q\") pod \"maas-api-revoke-sub-986b3c70-tnprm\" (UID: \"21584a67-219d-4646-86b7-90719a04de2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-986b3c70-tnprm" Nov 03 03:40:46.726080 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:46.725988 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-986b3c70-tnprm" Nov 03 03:40:47.096616 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:47.096580 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-986b3c70-tnprm"] Nov 03 03:40:47.102805 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:40:47.102773 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21584a67_219d_4646_86b7_90719a04de2b.slice/crio-25d3cf1fab786017678052e1d47e05b9be5d2d5b6b206486fda9e7e11b2fa4a7 WatchSource:0}: Error finding container 25d3cf1fab786017678052e1d47e05b9be5d2d5b6b206486fda9e7e11b2fa4a7: Status 404 returned error can't find the container with id 25d3cf1fab786017678052e1d47e05b9be5d2d5b6b206486fda9e7e11b2fa4a7 Nov 03 03:40:47.749907 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:47.749813 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-986b3c70-tnprm" event={"ID":"21584a67-219d-4646-86b7-90719a04de2b","Type":"ContainerStarted","Data":"3d591718886b7dd66e2589540d9a80332e32f0148604f1439c39312fd8de46f4"} Nov 03 03:40:47.749907 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:47.749869 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-986b3c70-tnprm" event={"ID":"21584a67-219d-4646-86b7-90719a04de2b","Type":"ContainerStarted","Data":"25d3cf1fab786017678052e1d47e05b9be5d2d5b6b206486fda9e7e11b2fa4a7"} Nov 03 03:40:47.857465 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:47.857405 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-986b3c70-tnprm" podStartSLOduration=1.857387586 podStartE2EDuration="1.857387586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:40:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:40:47.856903864 +0000 UTC m=+6525.116102020" watchObservedRunningTime="2026-11-03 03:40:47.857387586 +0000 UTC m=+6525.116585741" Nov 03 03:40:48.421153 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:48.421108 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:40:48.760409 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:48.760218 5605 generic.go:356] "Generic (PLEG): container finished" podID="21584a67-219d-4646-86b7-90719a04de2b" containerID="3d591718886b7dd66e2589540d9a80332e32f0148604f1439c39312fd8de46f4" exitCode=0 Nov 03 03:40:48.760409 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:48.760262 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-986b3c70-tnprm" event={"ID":"21584a67-219d-4646-86b7-90719a04de2b","Type":"ContainerDied","Data":"3d591718886b7dd66e2589540d9a80332e32f0148604f1439c39312fd8de46f4"} Nov 03 03:40:50.530086 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:50.529772 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-986b3c70-tnprm" Nov 03 03:40:50.692804 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:50.692749 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21584a67-219d-4646-86b7-90719a04de2b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21584a67-219d-4646-86b7-90719a04de2b-maas-api-xks-ca\") pod \"21584a67-219d-4646-86b7-90719a04de2b\" (UID: \"21584a67-219d-4646-86b7-90719a04de2b\") " Nov 03 03:40:50.708612 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:50.708515 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21584a67-219d-4646-86b7-90719a04de2b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21584a67-219d-4646-86b7-90719a04de2b-maas-api-openshift-ca\") pod \"21584a67-219d-4646-86b7-90719a04de2b\" (UID: \"21584a67-219d-4646-86b7-90719a04de2b\") " Nov 03 03:40:50.731516 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:50.709264 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21584a67-219d-4646-86b7-90719a04de2b-maas-api-openshift-ca" pod "21584a67-219d-4646-86b7-90719a04de2b" (UID: "21584a67-219d-4646-86b7-90719a04de2b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:40:50.734486 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:50.731847 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21584a67-219d-4646-86b7-90719a04de2b-kube-api-access-c7z4q\" (UniqueName: \"kubernetes.io/projected/21584a67-219d-4646-86b7-90719a04de2b-kube-api-access-c7z4q\") pod \"21584a67-219d-4646-86b7-90719a04de2b\" (UID: \"21584a67-219d-4646-86b7-90719a04de2b\") " Nov 03 03:40:50.763392 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:50.748903 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21584a67-219d-4646-86b7-90719a04de2b-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:40:50.763764 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:50.760964 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21584a67-219d-4646-86b7-90719a04de2b-kube-api-access-c7z4q" pod "21584a67-219d-4646-86b7-90719a04de2b" (UID: "21584a67-219d-4646-86b7-90719a04de2b"). InnerVolumeSpecName "kube-api-access-c7z4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:40:50.763924 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:50.761009 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21584a67-219d-4646-86b7-90719a04de2b-maas-api-xks-ca" pod "21584a67-219d-4646-86b7-90719a04de2b" (UID: "21584a67-219d-4646-86b7-90719a04de2b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:40:50.852342 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:50.852312 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-986b3c70-tnprm" Nov 03 03:40:50.853385 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:50.852771 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-986b3c70-tnprm" event={"ID":"21584a67-219d-4646-86b7-90719a04de2b","Type":"ContainerDied","Data":"25d3cf1fab786017678052e1d47e05b9be5d2d5b6b206486fda9e7e11b2fa4a7"} Nov 03 03:40:50.853385 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:50.852832 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25d3cf1fab786017678052e1d47e05b9be5d2d5b6b206486fda9e7e11b2fa4a7" Nov 03 03:40:50.906383 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:50.902989 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7z4q\" (UniqueName: \"kubernetes.io/projected/21584a67-219d-4646-86b7-90719a04de2b-kube-api-access-c7z4q\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:40:50.910449 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:50.910413 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21584a67-219d-4646-86b7-90719a04de2b-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:40:53.911852 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:40:53.911816 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:40:59.134294 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:40:59.134260 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:41:05.989699 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:05.989660 5605 scope.go:122] "RemoveContainer" containerID="20a85ca70c5bb1590b259134ed7b45cb358f6a794204aaf4bdb5391202a482d9" Nov 03 03:41:07.225809 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:07.225772 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:41:23.911090 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:41:23.911045 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:41:25.721783 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:25.721752 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:41:35.186056 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:35.186013 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6644eaa-nrq72"] Nov 03 03:41:35.186751 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:35.186586 5605 cpu_manager.go:414] "containerMap: removing container" podUID="21584a67-219d-4646-86b7-90719a04de2b" containerName="revoke-keys" Nov 03 03:41:35.186751 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:35.186608 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21584a67-219d-4646-86b7-90719a04de2b" containerName="revoke-keys" Nov 03 03:41:35.186751 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:35.186731 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="21584a67-219d-4646-86b7-90719a04de2b" containerName="revoke-keys" Nov 03 03:41:35.190022 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:35.189999 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6644eaa-nrq72" Nov 03 03:41:35.239172 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:35.239130 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:41:35.239383 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:35.239211 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:41:35.285950 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:35.285914 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:41:35.285950 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:35.285941 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sb4qh\"" Nov 03 03:41:35.287558 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:35.287534 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6644eaa-nrq72"] Nov 03 03:41:35.354060 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:35.354017 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf9xs\" (UniqueName: \"kubernetes.io/projected/4819c8b3-339a-4a9b-9c54-81cda7e94236-kube-api-access-cf9xs\") pod \"maas-api-revoke-sub-e6644eaa-nrq72\" (UID: \"4819c8b3-339a-4a9b-9c54-81cda7e94236\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6644eaa-nrq72" Nov 03 03:41:35.354282 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:35.354087 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4819c8b3-339a-4a9b-9c54-81cda7e94236-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6644eaa-nrq72\" (UID: \"4819c8b3-339a-4a9b-9c54-81cda7e94236\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6644eaa-nrq72" Nov 03 03:41:35.354282 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:35.354266 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4819c8b3-339a-4a9b-9c54-81cda7e94236-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6644eaa-nrq72\" (UID: \"4819c8b3-339a-4a9b-9c54-81cda7e94236\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6644eaa-nrq72" Nov 03 03:41:35.455269 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:35.455183 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf9xs\" (UniqueName: \"kubernetes.io/projected/4819c8b3-339a-4a9b-9c54-81cda7e94236-kube-api-access-cf9xs\") pod \"maas-api-revoke-sub-e6644eaa-nrq72\" (UID: \"4819c8b3-339a-4a9b-9c54-81cda7e94236\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6644eaa-nrq72" Nov 03 03:41:35.455269 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:35.455232 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4819c8b3-339a-4a9b-9c54-81cda7e94236-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6644eaa-nrq72\" (UID: \"4819c8b3-339a-4a9b-9c54-81cda7e94236\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6644eaa-nrq72" Nov 03 03:41:35.455269 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:35.455270 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4819c8b3-339a-4a9b-9c54-81cda7e94236-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6644eaa-nrq72\" (UID: \"4819c8b3-339a-4a9b-9c54-81cda7e94236\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6644eaa-nrq72" Nov 03 03:41:35.455979 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:35.455956 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4819c8b3-339a-4a9b-9c54-81cda7e94236-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6644eaa-nrq72\" (UID: \"4819c8b3-339a-4a9b-9c54-81cda7e94236\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6644eaa-nrq72" Nov 03 03:41:35.457772 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:35.457747 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4819c8b3-339a-4a9b-9c54-81cda7e94236-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6644eaa-nrq72\" (UID: \"4819c8b3-339a-4a9b-9c54-81cda7e94236\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6644eaa-nrq72" Nov 03 03:41:35.509657 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:35.509625 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf9xs\" (UniqueName: \"kubernetes.io/projected/4819c8b3-339a-4a9b-9c54-81cda7e94236-kube-api-access-cf9xs\") pod \"maas-api-revoke-sub-e6644eaa-nrq72\" (UID: \"4819c8b3-339a-4a9b-9c54-81cda7e94236\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6644eaa-nrq72" Nov 03 03:41:35.803012 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:35.802977 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6644eaa-nrq72" Nov 03 03:41:36.173281 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:36.172111 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6644eaa-nrq72"] Nov 03 03:41:36.182489 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:41:36.182458 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4819c8b3_339a_4a9b_9c54_81cda7e94236.slice/crio-27b5f35ff223161e3b9a437a4ee169b7ef171d22aa0d783edfb15374a74d45fd WatchSource:0}: Error finding container 27b5f35ff223161e3b9a437a4ee169b7ef171d22aa0d783edfb15374a74d45fd: Status 404 returned error can't find the container with id 27b5f35ff223161e3b9a437a4ee169b7ef171d22aa0d783edfb15374a74d45fd Nov 03 03:41:37.157198 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:37.157164 5605 generic.go:356] "Generic (PLEG): container finished" podID="4819c8b3-339a-4a9b-9c54-81cda7e94236" containerID="1e7cf491f86527cc158d3e725448bff58c10aa9e4e8bf302903989bb7925f31e" exitCode=0 Nov 03 03:41:37.157628 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:37.157249 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6644eaa-nrq72" event={"ID":"4819c8b3-339a-4a9b-9c54-81cda7e94236","Type":"ContainerDied","Data":"1e7cf491f86527cc158d3e725448bff58c10aa9e4e8bf302903989bb7925f31e"} Nov 03 03:41:37.157628 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:37.157301 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6644eaa-nrq72" event={"ID":"4819c8b3-339a-4a9b-9c54-81cda7e94236","Type":"ContainerStarted","Data":"27b5f35ff223161e3b9a437a4ee169b7ef171d22aa0d783edfb15374a74d45fd"} Nov 03 03:41:38.297112 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:38.297085 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6644eaa-nrq72" Nov 03 03:41:38.381919 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:38.381878 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4819c8b3-339a-4a9b-9c54-81cda7e94236-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4819c8b3-339a-4a9b-9c54-81cda7e94236-maas-api-xks-ca\") pod \"4819c8b3-339a-4a9b-9c54-81cda7e94236\" (UID: \"4819c8b3-339a-4a9b-9c54-81cda7e94236\") " Nov 03 03:41:38.382179 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:38.381956 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4819c8b3-339a-4a9b-9c54-81cda7e94236-kube-api-access-cf9xs\" (UniqueName: \"kubernetes.io/projected/4819c8b3-339a-4a9b-9c54-81cda7e94236-kube-api-access-cf9xs\") pod \"4819c8b3-339a-4a9b-9c54-81cda7e94236\" (UID: \"4819c8b3-339a-4a9b-9c54-81cda7e94236\") " Nov 03 03:41:38.382179 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:38.382054 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4819c8b3-339a-4a9b-9c54-81cda7e94236-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4819c8b3-339a-4a9b-9c54-81cda7e94236-maas-api-openshift-ca\") pod \"4819c8b3-339a-4a9b-9c54-81cda7e94236\" (UID: \"4819c8b3-339a-4a9b-9c54-81cda7e94236\") " Nov 03 03:41:38.382457 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:38.382431 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4819c8b3-339a-4a9b-9c54-81cda7e94236-maas-api-openshift-ca" pod "4819c8b3-339a-4a9b-9c54-81cda7e94236" (UID: "4819c8b3-339a-4a9b-9c54-81cda7e94236"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:41:38.384095 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:38.384075 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4819c8b3-339a-4a9b-9c54-81cda7e94236-maas-api-xks-ca" pod "4819c8b3-339a-4a9b-9c54-81cda7e94236" (UID: "4819c8b3-339a-4a9b-9c54-81cda7e94236"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:41:38.384227 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:38.384205 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4819c8b3-339a-4a9b-9c54-81cda7e94236-kube-api-access-cf9xs" pod "4819c8b3-339a-4a9b-9c54-81cda7e94236" (UID: "4819c8b3-339a-4a9b-9c54-81cda7e94236"). InnerVolumeSpecName "kube-api-access-cf9xs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:41:38.483480 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:38.483388 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cf9xs\" (UniqueName: \"kubernetes.io/projected/4819c8b3-339a-4a9b-9c54-81cda7e94236-kube-api-access-cf9xs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:41:38.483480 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:38.483420 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4819c8b3-339a-4a9b-9c54-81cda7e94236-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:41:38.483480 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:38.483430 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4819c8b3-339a-4a9b-9c54-81cda7e94236-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:41:39.168379 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:39.168264 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6644eaa-nrq72" event={"ID":"4819c8b3-339a-4a9b-9c54-81cda7e94236","Type":"ContainerDied","Data":"27b5f35ff223161e3b9a437a4ee169b7ef171d22aa0d783edfb15374a74d45fd"} Nov 03 03:41:39.168379 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:39.168308 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6644eaa-nrq72" Nov 03 03:41:39.168379 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:39.168322 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27b5f35ff223161e3b9a437a4ee169b7ef171d22aa0d783edfb15374a74d45fd" Nov 03 03:41:46.202876 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:46.202788 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7451fda3-q7dx9"] Nov 03 03:41:46.203269 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:46.203112 5605 cpu_manager.go:414] "containerMap: removing container" podUID="4819c8b3-339a-4a9b-9c54-81cda7e94236" containerName="revoke-keys" Nov 03 03:41:46.203269 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:46.203123 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4819c8b3-339a-4a9b-9c54-81cda7e94236" containerName="revoke-keys" Nov 03 03:41:46.203269 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:46.203229 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="4819c8b3-339a-4a9b-9c54-81cda7e94236" containerName="revoke-keys" Nov 03 03:41:46.206384 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:46.206342 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7451fda3-q7dx9" Nov 03 03:41:46.249531 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:46.249494 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqppr\" (UniqueName: \"kubernetes.io/projected/ca706570-f5be-404a-8a82-25c8466d74c4-kube-api-access-hqppr\") pod \"maas-api-revoke-sub-7451fda3-q7dx9\" (UID: \"ca706570-f5be-404a-8a82-25c8466d74c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7451fda3-q7dx9" Nov 03 03:41:46.249732 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:46.249639 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca706570-f5be-404a-8a82-25c8466d74c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7451fda3-q7dx9\" (UID: \"ca706570-f5be-404a-8a82-25c8466d74c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7451fda3-q7dx9" Nov 03 03:41:46.249732 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:46.249706 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca706570-f5be-404a-8a82-25c8466d74c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7451fda3-q7dx9\" (UID: \"ca706570-f5be-404a-8a82-25c8466d74c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7451fda3-q7dx9" Nov 03 03:41:46.255833 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:46.255812 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:41:46.302367 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:46.302328 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:41:46.302563 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:46.302543 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:41:46.302661 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:46.302566 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sb4qh\"" Nov 03 03:41:46.304340 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:46.304314 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7451fda3-q7dx9"] Nov 03 03:41:46.351107 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:46.351081 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca706570-f5be-404a-8a82-25c8466d74c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7451fda3-q7dx9\" (UID: \"ca706570-f5be-404a-8a82-25c8466d74c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7451fda3-q7dx9" Nov 03 03:41:46.351217 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:46.351122 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca706570-f5be-404a-8a82-25c8466d74c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7451fda3-q7dx9\" (UID: \"ca706570-f5be-404a-8a82-25c8466d74c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7451fda3-q7dx9" Nov 03 03:41:46.351293 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:46.351229 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqppr\" (UniqueName: \"kubernetes.io/projected/ca706570-f5be-404a-8a82-25c8466d74c4-kube-api-access-hqppr\") pod \"maas-api-revoke-sub-7451fda3-q7dx9\" (UID: \"ca706570-f5be-404a-8a82-25c8466d74c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7451fda3-q7dx9" Nov 03 03:41:46.351752 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:46.351732 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca706570-f5be-404a-8a82-25c8466d74c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7451fda3-q7dx9\" (UID: \"ca706570-f5be-404a-8a82-25c8466d74c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7451fda3-q7dx9" Nov 03 03:41:46.353432 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:46.353412 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca706570-f5be-404a-8a82-25c8466d74c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7451fda3-q7dx9\" (UID: \"ca706570-f5be-404a-8a82-25c8466d74c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7451fda3-q7dx9" Nov 03 03:41:46.404705 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:46.404686 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqppr\" (UniqueName: \"kubernetes.io/projected/ca706570-f5be-404a-8a82-25c8466d74c4-kube-api-access-hqppr\") pod \"maas-api-revoke-sub-7451fda3-q7dx9\" (UID: \"ca706570-f5be-404a-8a82-25c8466d74c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7451fda3-q7dx9" Nov 03 03:41:46.518500 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:46.518448 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7451fda3-q7dx9" Nov 03 03:41:46.883660 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:46.883624 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7451fda3-q7dx9"] Nov 03 03:41:46.889866 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:41:46.889839 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca706570_f5be_404a_8a82_25c8466d74c4.slice/crio-da68b577017c1e666df2cd5cc5abc230ecf6a12c78e0b7237398fbc51057fbee WatchSource:0}: Error finding container da68b577017c1e666df2cd5cc5abc230ecf6a12c78e0b7237398fbc51057fbee: Status 404 returned error can't find the container with id da68b577017c1e666df2cd5cc5abc230ecf6a12c78e0b7237398fbc51057fbee Nov 03 03:41:47.205263 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:47.205199 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7451fda3-q7dx9" event={"ID":"ca706570-f5be-404a-8a82-25c8466d74c4","Type":"ContainerStarted","Data":"43f9f8fb28000434320c859de4ca52425b3dd7af9b168c453b3b71c841f10271"} Nov 03 03:41:47.205263 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:47.205251 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7451fda3-q7dx9" event={"ID":"ca706570-f5be-404a-8a82-25c8466d74c4","Type":"ContainerStarted","Data":"da68b577017c1e666df2cd5cc5abc230ecf6a12c78e0b7237398fbc51057fbee"} Nov 03 03:41:47.313592 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:47.313547 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7451fda3-q7dx9" podStartSLOduration=1.313533246 podStartE2EDuration="1.313533246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:41:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:41:47.312649382 +0000 UTC m=+6584.571847535" watchObservedRunningTime="2026-11-03 03:41:47.313533246 +0000 UTC m=+6584.572731400" Nov 03 03:41:48.211048 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:48.211012 5605 generic.go:356] "Generic (PLEG): container finished" podID="ca706570-f5be-404a-8a82-25c8466d74c4" containerID="43f9f8fb28000434320c859de4ca52425b3dd7af9b168c453b3b71c841f10271" exitCode=0 Nov 03 03:41:48.211476 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:48.211094 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7451fda3-q7dx9" event={"ID":"ca706570-f5be-404a-8a82-25c8466d74c4","Type":"ContainerDied","Data":"43f9f8fb28000434320c859de4ca52425b3dd7af9b168c453b3b71c841f10271"} Nov 03 03:41:49.353508 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:49.353481 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7451fda3-q7dx9" Nov 03 03:41:49.476817 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:49.476747 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca706570-f5be-404a-8a82-25c8466d74c4-kube-api-access-hqppr\" (UniqueName: \"kubernetes.io/projected/ca706570-f5be-404a-8a82-25c8466d74c4-kube-api-access-hqppr\") pod \"ca706570-f5be-404a-8a82-25c8466d74c4\" (UID: \"ca706570-f5be-404a-8a82-25c8466d74c4\") " Nov 03 03:41:49.476817 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:49.476785 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca706570-f5be-404a-8a82-25c8466d74c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca706570-f5be-404a-8a82-25c8466d74c4-maas-api-xks-ca\") pod \"ca706570-f5be-404a-8a82-25c8466d74c4\" (UID: \"ca706570-f5be-404a-8a82-25c8466d74c4\") " Nov 03 03:41:49.476817 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:49.476814 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca706570-f5be-404a-8a82-25c8466d74c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca706570-f5be-404a-8a82-25c8466d74c4-maas-api-openshift-ca\") pod \"ca706570-f5be-404a-8a82-25c8466d74c4\" (UID: \"ca706570-f5be-404a-8a82-25c8466d74c4\") " Nov 03 03:41:49.477228 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:49.477205 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca706570-f5be-404a-8a82-25c8466d74c4-maas-api-openshift-ca" pod "ca706570-f5be-404a-8a82-25c8466d74c4" (UID: "ca706570-f5be-404a-8a82-25c8466d74c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:41:49.478910 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:49.478886 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca706570-f5be-404a-8a82-25c8466d74c4-kube-api-access-hqppr" pod "ca706570-f5be-404a-8a82-25c8466d74c4" (UID: "ca706570-f5be-404a-8a82-25c8466d74c4"). InnerVolumeSpecName "kube-api-access-hqppr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:41:49.478910 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:49.478892 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca706570-f5be-404a-8a82-25c8466d74c4-maas-api-xks-ca" pod "ca706570-f5be-404a-8a82-25c8466d74c4" (UID: "ca706570-f5be-404a-8a82-25c8466d74c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:41:49.577897 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:49.577870 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqppr\" (UniqueName: \"kubernetes.io/projected/ca706570-f5be-404a-8a82-25c8466d74c4-kube-api-access-hqppr\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:41:49.577897 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:49.577895 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca706570-f5be-404a-8a82-25c8466d74c4-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:41:49.578057 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:49.577905 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca706570-f5be-404a-8a82-25c8466d74c4-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:41:49.646070 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:49.646048 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:41:50.223492 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:50.223450 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7451fda3-q7dx9" event={"ID":"ca706570-f5be-404a-8a82-25c8466d74c4","Type":"ContainerDied","Data":"da68b577017c1e666df2cd5cc5abc230ecf6a12c78e0b7237398fbc51057fbee"} Nov 03 03:41:50.223711 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:50.223499 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7451fda3-q7dx9" Nov 03 03:41:50.223711 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:50.223514 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da68b577017c1e666df2cd5cc5abc230ecf6a12c78e0b7237398fbc51057fbee" Nov 03 03:41:53.915273 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:41:53.915229 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:41:57.273281 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:57.273247 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c81be95d-2phsq"] Nov 03 03:41:57.273669 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:57.273620 5605 cpu_manager.go:414] "containerMap: removing container" podUID="ca706570-f5be-404a-8a82-25c8466d74c4" containerName="revoke-keys" Nov 03 03:41:57.273669 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:57.273632 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca706570-f5be-404a-8a82-25c8466d74c4" containerName="revoke-keys" Nov 03 03:41:57.273760 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:57.273702 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca706570-f5be-404a-8a82-25c8466d74c4" containerName="revoke-keys" Nov 03 03:41:57.276724 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:57.276701 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81be95d-2phsq" Nov 03 03:41:57.356265 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:57.356237 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:41:57.356421 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:57.356263 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:41:57.380934 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:57.380903 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:41:57.381131 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:57.380933 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sb4qh\"" Nov 03 03:41:57.382668 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:57.382645 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c81be95d-2phsq"] Nov 03 03:41:57.385165 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:57.385146 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53ed8432-b78a-4b9b-ad15-56ff5772752b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c81be95d-2phsq\" (UID: \"53ed8432-b78a-4b9b-ad15-56ff5772752b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81be95d-2phsq" Nov 03 03:41:57.385232 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:57.385179 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53ed8432-b78a-4b9b-ad15-56ff5772752b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c81be95d-2phsq\" (UID: \"53ed8432-b78a-4b9b-ad15-56ff5772752b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81be95d-2phsq" Nov 03 03:41:57.385232 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:57.385211 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzzzg\" (UniqueName: \"kubernetes.io/projected/53ed8432-b78a-4b9b-ad15-56ff5772752b-kube-api-access-gzzzg\") pod \"maas-api-revoke-sub-c81be95d-2phsq\" (UID: \"53ed8432-b78a-4b9b-ad15-56ff5772752b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81be95d-2phsq" Nov 03 03:41:57.485905 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:57.485880 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzzzg\" (UniqueName: \"kubernetes.io/projected/53ed8432-b78a-4b9b-ad15-56ff5772752b-kube-api-access-gzzzg\") pod \"maas-api-revoke-sub-c81be95d-2phsq\" (UID: \"53ed8432-b78a-4b9b-ad15-56ff5772752b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81be95d-2phsq" Nov 03 03:41:57.486021 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:57.485949 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53ed8432-b78a-4b9b-ad15-56ff5772752b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c81be95d-2phsq\" (UID: \"53ed8432-b78a-4b9b-ad15-56ff5772752b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81be95d-2phsq" Nov 03 03:41:57.486021 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:57.485974 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53ed8432-b78a-4b9b-ad15-56ff5772752b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c81be95d-2phsq\" (UID: \"53ed8432-b78a-4b9b-ad15-56ff5772752b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81be95d-2phsq" Nov 03 03:41:57.486663 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:57.486647 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53ed8432-b78a-4b9b-ad15-56ff5772752b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c81be95d-2phsq\" (UID: \"53ed8432-b78a-4b9b-ad15-56ff5772752b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81be95d-2phsq" Nov 03 03:41:57.488316 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:57.488295 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53ed8432-b78a-4b9b-ad15-56ff5772752b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c81be95d-2phsq\" (UID: \"53ed8432-b78a-4b9b-ad15-56ff5772752b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81be95d-2phsq" Nov 03 03:41:57.539342 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:57.539290 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzzzg\" (UniqueName: \"kubernetes.io/projected/53ed8432-b78a-4b9b-ad15-56ff5772752b-kube-api-access-gzzzg\") pod \"maas-api-revoke-sub-c81be95d-2phsq\" (UID: \"53ed8432-b78a-4b9b-ad15-56ff5772752b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81be95d-2phsq" Nov 03 03:41:57.589402 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:57.589379 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81be95d-2phsq" Nov 03 03:41:57.955514 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:57.955483 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c81be95d-2phsq"] Nov 03 03:41:57.962125 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:41:57.962096 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53ed8432_b78a_4b9b_ad15_56ff5772752b.slice/crio-35208364fa2a39df6afe301c444375052e718606749656974d14cf37a5a490aa WatchSource:0}: Error finding container 35208364fa2a39df6afe301c444375052e718606749656974d14cf37a5a490aa: Status 404 returned error can't find the container with id 35208364fa2a39df6afe301c444375052e718606749656974d14cf37a5a490aa Nov 03 03:41:58.271851 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:58.271794 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81be95d-2phsq" event={"ID":"53ed8432-b78a-4b9b-ad15-56ff5772752b","Type":"ContainerStarted","Data":"a2032cf5c173a2570a4f4d3f83806b81da9b5ae7ebab1febf5a88e16dc9290b7"} Nov 03 03:41:58.271851 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:58.271846 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81be95d-2phsq" event={"ID":"53ed8432-b78a-4b9b-ad15-56ff5772752b","Type":"ContainerStarted","Data":"35208364fa2a39df6afe301c444375052e718606749656974d14cf37a5a490aa"} Nov 03 03:41:58.377464 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:58.377419 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81be95d-2phsq" podStartSLOduration=1.377406025 podStartE2EDuration="1.377406025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:41:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:41:58.376550087 +0000 UTC m=+6595.635748240" watchObservedRunningTime="2026-11-03 03:41:58.377406025 +0000 UTC m=+6595.636604178" Nov 03 03:41:59.277811 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:59.277773 5605 generic.go:356] "Generic (PLEG): container finished" podID="53ed8432-b78a-4b9b-ad15-56ff5772752b" containerID="a2032cf5c173a2570a4f4d3f83806b81da9b5ae7ebab1febf5a88e16dc9290b7" exitCode=0 Nov 03 03:41:59.278009 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:41:59.277864 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81be95d-2phsq" event={"ID":"53ed8432-b78a-4b9b-ad15-56ff5772752b","Type":"ContainerDied","Data":"a2032cf5c173a2570a4f4d3f83806b81da9b5ae7ebab1febf5a88e16dc9290b7"} Nov 03 03:42:00.408962 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:00.408937 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81be95d-2phsq" Nov 03 03:42:00.510666 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:00.510640 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53ed8432-b78a-4b9b-ad15-56ff5772752b-kube-api-access-gzzzg\" (UniqueName: \"kubernetes.io/projected/53ed8432-b78a-4b9b-ad15-56ff5772752b-kube-api-access-gzzzg\") pod \"53ed8432-b78a-4b9b-ad15-56ff5772752b\" (UID: \"53ed8432-b78a-4b9b-ad15-56ff5772752b\") " Nov 03 03:42:00.510774 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:00.510696 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53ed8432-b78a-4b9b-ad15-56ff5772752b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53ed8432-b78a-4b9b-ad15-56ff5772752b-maas-api-openshift-ca\") pod \"53ed8432-b78a-4b9b-ad15-56ff5772752b\" (UID: \"53ed8432-b78a-4b9b-ad15-56ff5772752b\") " Nov 03 03:42:00.510774 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:00.510734 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53ed8432-b78a-4b9b-ad15-56ff5772752b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53ed8432-b78a-4b9b-ad15-56ff5772752b-maas-api-xks-ca\") pod \"53ed8432-b78a-4b9b-ad15-56ff5772752b\" (UID: \"53ed8432-b78a-4b9b-ad15-56ff5772752b\") " Nov 03 03:42:00.511064 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:00.511043 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53ed8432-b78a-4b9b-ad15-56ff5772752b-maas-api-openshift-ca" pod "53ed8432-b78a-4b9b-ad15-56ff5772752b" (UID: "53ed8432-b78a-4b9b-ad15-56ff5772752b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:42:00.512791 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:00.512766 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53ed8432-b78a-4b9b-ad15-56ff5772752b-kube-api-access-gzzzg" pod "53ed8432-b78a-4b9b-ad15-56ff5772752b" (UID: "53ed8432-b78a-4b9b-ad15-56ff5772752b"). InnerVolumeSpecName "kube-api-access-gzzzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:42:00.512791 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:00.512778 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53ed8432-b78a-4b9b-ad15-56ff5772752b-maas-api-xks-ca" pod "53ed8432-b78a-4b9b-ad15-56ff5772752b" (UID: "53ed8432-b78a-4b9b-ad15-56ff5772752b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:42:00.612302 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:00.612278 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzzzg\" (UniqueName: \"kubernetes.io/projected/53ed8432-b78a-4b9b-ad15-56ff5772752b-kube-api-access-gzzzg\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:42:00.612302 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:00.612302 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53ed8432-b78a-4b9b-ad15-56ff5772752b-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:42:00.612486 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:00.612313 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53ed8432-b78a-4b9b-ad15-56ff5772752b-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:42:01.289872 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:01.289838 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81be95d-2phsq" Nov 03 03:42:01.290047 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:01.289875 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81be95d-2phsq" event={"ID":"53ed8432-b78a-4b9b-ad15-56ff5772752b","Type":"ContainerDied","Data":"35208364fa2a39df6afe301c444375052e718606749656974d14cf37a5a490aa"} Nov 03 03:42:01.290047 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:01.289933 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35208364fa2a39df6afe301c444375052e718606749656974d14cf37a5a490aa" Nov 03 03:42:06.089935 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:06.089898 5605 scope.go:122] "RemoveContainer" containerID="fcc66b42d23cfbfb159d9e3bd029ccc254723f825a229efe1f3277dc0e7862ae" Nov 03 03:42:08.467097 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:08.467062 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b18aaf4c-mgbvb"] Nov 03 03:42:08.467516 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:08.467453 5605 cpu_manager.go:414] "containerMap: removing container" podUID="53ed8432-b78a-4b9b-ad15-56ff5772752b" containerName="revoke-keys" Nov 03 03:42:08.467516 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:08.467466 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53ed8432-b78a-4b9b-ad15-56ff5772752b" containerName="revoke-keys" Nov 03 03:42:08.467601 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:08.467540 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="53ed8432-b78a-4b9b-ad15-56ff5772752b" containerName="revoke-keys" Nov 03 03:42:08.472148 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:08.472128 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18aaf4c-mgbvb" Nov 03 03:42:08.521457 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:08.521431 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:42:08.567875 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:08.567825 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:42:08.567875 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:08.567844 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:42:08.567875 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:08.567868 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sb4qh\"" Nov 03 03:42:08.569783 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:08.569762 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b18aaf4c-mgbvb"] Nov 03 03:42:08.578826 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:08.578808 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwsl9\" (UniqueName: \"kubernetes.io/projected/e6261b7a-3a8d-4da1-90ab-775aa83aa9ad-kube-api-access-dwsl9\") pod \"maas-api-revoke-sub-b18aaf4c-mgbvb\" (UID: \"e6261b7a-3a8d-4da1-90ab-775aa83aa9ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18aaf4c-mgbvb" Nov 03 03:42:08.578897 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:08.578839 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6261b7a-3a8d-4da1-90ab-775aa83aa9ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b18aaf4c-mgbvb\" (UID: \"e6261b7a-3a8d-4da1-90ab-775aa83aa9ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18aaf4c-mgbvb" Nov 03 03:42:08.578897 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:08.578880 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6261b7a-3a8d-4da1-90ab-775aa83aa9ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b18aaf4c-mgbvb\" (UID: \"e6261b7a-3a8d-4da1-90ab-775aa83aa9ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18aaf4c-mgbvb" Nov 03 03:42:08.680272 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:08.680248 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwsl9\" (UniqueName: \"kubernetes.io/projected/e6261b7a-3a8d-4da1-90ab-775aa83aa9ad-kube-api-access-dwsl9\") pod \"maas-api-revoke-sub-b18aaf4c-mgbvb\" (UID: \"e6261b7a-3a8d-4da1-90ab-775aa83aa9ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18aaf4c-mgbvb" Nov 03 03:42:08.680415 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:08.680278 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6261b7a-3a8d-4da1-90ab-775aa83aa9ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b18aaf4c-mgbvb\" (UID: \"e6261b7a-3a8d-4da1-90ab-775aa83aa9ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18aaf4c-mgbvb" Nov 03 03:42:08.680415 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:08.680312 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6261b7a-3a8d-4da1-90ab-775aa83aa9ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b18aaf4c-mgbvb\" (UID: \"e6261b7a-3a8d-4da1-90ab-775aa83aa9ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18aaf4c-mgbvb" Nov 03 03:42:08.681065 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:08.681039 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6261b7a-3a8d-4da1-90ab-775aa83aa9ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b18aaf4c-mgbvb\" (UID: \"e6261b7a-3a8d-4da1-90ab-775aa83aa9ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18aaf4c-mgbvb" Nov 03 03:42:08.682695 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:08.682673 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6261b7a-3a8d-4da1-90ab-775aa83aa9ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b18aaf4c-mgbvb\" (UID: \"e6261b7a-3a8d-4da1-90ab-775aa83aa9ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18aaf4c-mgbvb" Nov 03 03:42:08.734191 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:08.734131 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwsl9\" (UniqueName: \"kubernetes.io/projected/e6261b7a-3a8d-4da1-90ab-775aa83aa9ad-kube-api-access-dwsl9\") pod \"maas-api-revoke-sub-b18aaf4c-mgbvb\" (UID: \"e6261b7a-3a8d-4da1-90ab-775aa83aa9ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18aaf4c-mgbvb" Nov 03 03:42:08.784681 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:08.784659 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18aaf4c-mgbvb" Nov 03 03:42:09.149654 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:09.149624 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b18aaf4c-mgbvb"] Nov 03 03:42:09.155481 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:42:09.155452 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6261b7a_3a8d_4da1_90ab_775aa83aa9ad.slice/crio-18f7613a406490d3b6600c68c33889f8e979efa03c5746c78dec566c9ef17bcf WatchSource:0}: Error finding container 18f7613a406490d3b6600c68c33889f8e979efa03c5746c78dec566c9ef17bcf: Status 404 returned error can't find the container with id 18f7613a406490d3b6600c68c33889f8e979efa03c5746c78dec566c9ef17bcf Nov 03 03:42:09.327061 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:09.327021 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18aaf4c-mgbvb" event={"ID":"e6261b7a-3a8d-4da1-90ab-775aa83aa9ad","Type":"ContainerStarted","Data":"18f7613a406490d3b6600c68c33889f8e979efa03c5746c78dec566c9ef17bcf"} Nov 03 03:42:10.333649 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:10.333618 5605 generic.go:356] "Generic (PLEG): container finished" podID="e6261b7a-3a8d-4da1-90ab-775aa83aa9ad" containerID="9bd13f182af36d93b287ee902fa4488871a3cff94a4cd1fdbe9fac73b127a2e8" exitCode=0 Nov 03 03:42:10.333988 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:10.333687 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18aaf4c-mgbvb" event={"ID":"e6261b7a-3a8d-4da1-90ab-775aa83aa9ad","Type":"ContainerDied","Data":"9bd13f182af36d93b287ee902fa4488871a3cff94a4cd1fdbe9fac73b127a2e8"} Nov 03 03:42:11.458683 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:11.458661 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18aaf4c-mgbvb" Nov 03 03:42:11.613347 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:11.613272 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6261b7a-3a8d-4da1-90ab-775aa83aa9ad-kube-api-access-dwsl9\" (UniqueName: \"kubernetes.io/projected/e6261b7a-3a8d-4da1-90ab-775aa83aa9ad-kube-api-access-dwsl9\") pod \"e6261b7a-3a8d-4da1-90ab-775aa83aa9ad\" (UID: \"e6261b7a-3a8d-4da1-90ab-775aa83aa9ad\") " Nov 03 03:42:11.613347 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:11.613335 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6261b7a-3a8d-4da1-90ab-775aa83aa9ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6261b7a-3a8d-4da1-90ab-775aa83aa9ad-maas-api-openshift-ca\") pod \"e6261b7a-3a8d-4da1-90ab-775aa83aa9ad\" (UID: \"e6261b7a-3a8d-4da1-90ab-775aa83aa9ad\") " Nov 03 03:42:11.613585 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:11.613419 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6261b7a-3a8d-4da1-90ab-775aa83aa9ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6261b7a-3a8d-4da1-90ab-775aa83aa9ad-maas-api-xks-ca\") pod \"e6261b7a-3a8d-4da1-90ab-775aa83aa9ad\" (UID: \"e6261b7a-3a8d-4da1-90ab-775aa83aa9ad\") " Nov 03 03:42:11.613663 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:11.613642 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6261b7a-3a8d-4da1-90ab-775aa83aa9ad-maas-api-openshift-ca" pod "e6261b7a-3a8d-4da1-90ab-775aa83aa9ad" (UID: "e6261b7a-3a8d-4da1-90ab-775aa83aa9ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:42:11.613752 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:11.613735 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6261b7a-3a8d-4da1-90ab-775aa83aa9ad-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:42:11.615347 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:11.615325 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6261b7a-3a8d-4da1-90ab-775aa83aa9ad-maas-api-xks-ca" pod "e6261b7a-3a8d-4da1-90ab-775aa83aa9ad" (UID: "e6261b7a-3a8d-4da1-90ab-775aa83aa9ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:42:11.615447 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:11.615397 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6261b7a-3a8d-4da1-90ab-775aa83aa9ad-kube-api-access-dwsl9" pod "e6261b7a-3a8d-4da1-90ab-775aa83aa9ad" (UID: "e6261b7a-3a8d-4da1-90ab-775aa83aa9ad"). InnerVolumeSpecName "kube-api-access-dwsl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:42:11.714690 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:11.714664 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6261b7a-3a8d-4da1-90ab-775aa83aa9ad-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:42:11.714690 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:11.714688 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwsl9\" (UniqueName: \"kubernetes.io/projected/e6261b7a-3a8d-4da1-90ab-775aa83aa9ad-kube-api-access-dwsl9\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:42:12.347835 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:12.347744 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18aaf4c-mgbvb" event={"ID":"e6261b7a-3a8d-4da1-90ab-775aa83aa9ad","Type":"ContainerDied","Data":"18f7613a406490d3b6600c68c33889f8e979efa03c5746c78dec566c9ef17bcf"} Nov 03 03:42:12.347835 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:12.347794 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18aaf4c-mgbvb" Nov 03 03:42:12.347835 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:12.347805 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18f7613a406490d3b6600c68c33889f8e979efa03c5746c78dec566c9ef17bcf" Nov 03 03:42:19.213036 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:19.212996 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ff431bf-xxlhd"] Nov 03 03:42:19.213486 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:19.213337 5605 cpu_manager.go:414] "containerMap: removing container" podUID="e6261b7a-3a8d-4da1-90ab-775aa83aa9ad" containerName="revoke-keys" Nov 03 03:42:19.213486 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:19.213347 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6261b7a-3a8d-4da1-90ab-775aa83aa9ad" containerName="revoke-keys" Nov 03 03:42:19.213486 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:19.213449 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6261b7a-3a8d-4da1-90ab-775aa83aa9ad" containerName="revoke-keys" Nov 03 03:42:19.216554 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:19.216532 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff431bf-xxlhd" Nov 03 03:42:19.223531 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:19.223504 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcb23d87-df37-4ad7-bfc7-e376c1e420b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ff431bf-xxlhd\" (UID: \"bcb23d87-df37-4ad7-bfc7-e376c1e420b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff431bf-xxlhd" Nov 03 03:42:19.223637 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:19.223557 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcb23d87-df37-4ad7-bfc7-e376c1e420b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ff431bf-xxlhd\" (UID: \"bcb23d87-df37-4ad7-bfc7-e376c1e420b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff431bf-xxlhd" Nov 03 03:42:19.223686 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:19.223627 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rhwv\" (UniqueName: \"kubernetes.io/projected/bcb23d87-df37-4ad7-bfc7-e376c1e420b1-kube-api-access-9rhwv\") pod \"maas-api-revoke-sub-2ff431bf-xxlhd\" (UID: \"bcb23d87-df37-4ad7-bfc7-e376c1e420b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff431bf-xxlhd" Nov 03 03:42:19.265640 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:19.265614 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:42:19.265749 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:19.265658 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:42:19.265749 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:19.265688 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:42:19.312493 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:19.312472 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sb4qh\"" Nov 03 03:42:19.314128 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:19.314110 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ff431bf-xxlhd"] Nov 03 03:42:19.324663 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:19.324643 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcb23d87-df37-4ad7-bfc7-e376c1e420b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ff431bf-xxlhd\" (UID: \"bcb23d87-df37-4ad7-bfc7-e376c1e420b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff431bf-xxlhd" Nov 03 03:42:19.324743 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:19.324685 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcb23d87-df37-4ad7-bfc7-e376c1e420b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ff431bf-xxlhd\" (UID: \"bcb23d87-df37-4ad7-bfc7-e376c1e420b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff431bf-xxlhd" Nov 03 03:42:19.324743 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:19.324707 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rhwv\" (UniqueName: \"kubernetes.io/projected/bcb23d87-df37-4ad7-bfc7-e376c1e420b1-kube-api-access-9rhwv\") pod \"maas-api-revoke-sub-2ff431bf-xxlhd\" (UID: \"bcb23d87-df37-4ad7-bfc7-e376c1e420b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff431bf-xxlhd" Nov 03 03:42:19.325293 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:19.325271 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcb23d87-df37-4ad7-bfc7-e376c1e420b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ff431bf-xxlhd\" (UID: \"bcb23d87-df37-4ad7-bfc7-e376c1e420b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff431bf-xxlhd" Nov 03 03:42:19.327024 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:19.326999 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcb23d87-df37-4ad7-bfc7-e376c1e420b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ff431bf-xxlhd\" (UID: \"bcb23d87-df37-4ad7-bfc7-e376c1e420b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff431bf-xxlhd" Nov 03 03:42:19.378307 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:19.378279 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rhwv\" (UniqueName: \"kubernetes.io/projected/bcb23d87-df37-4ad7-bfc7-e376c1e420b1-kube-api-access-9rhwv\") pod \"maas-api-revoke-sub-2ff431bf-xxlhd\" (UID: \"bcb23d87-df37-4ad7-bfc7-e376c1e420b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff431bf-xxlhd" Nov 03 03:42:19.529315 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:19.529259 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff431bf-xxlhd" Nov 03 03:42:19.895056 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:19.895019 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ff431bf-xxlhd"] Nov 03 03:42:19.901465 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:42:19.901436 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcb23d87_df37_4ad7_bfc7_e376c1e420b1.slice/crio-640974a453c41cb5ba2107ccfb7b57fbbf8ccac098d412b2b3bb23a355c52f55 WatchSource:0}: Error finding container 640974a453c41cb5ba2107ccfb7b57fbbf8ccac098d412b2b3bb23a355c52f55: Status 404 returned error can't find the container with id 640974a453c41cb5ba2107ccfb7b57fbbf8ccac098d412b2b3bb23a355c52f55 Nov 03 03:42:20.396563 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:20.396530 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff431bf-xxlhd" event={"ID":"bcb23d87-df37-4ad7-bfc7-e376c1e420b1","Type":"ContainerStarted","Data":"9fcb646b0e4640495364be43c59b8d68db717d3b47057fd4a5887845a76e7f97"} Nov 03 03:42:20.396891 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:20.396588 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff431bf-xxlhd" event={"ID":"bcb23d87-df37-4ad7-bfc7-e376c1e420b1","Type":"ContainerStarted","Data":"640974a453c41cb5ba2107ccfb7b57fbbf8ccac098d412b2b3bb23a355c52f55"} Nov 03 03:42:20.549202 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:20.549158 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff431bf-xxlhd" podStartSLOduration=1.549143777 podStartE2EDuration="1.549143777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:42:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:42:20.501946778 +0000 UTC m=+6617.761144932" watchObservedRunningTime="2026-11-03 03:42:20.549143777 +0000 UTC m=+6617.808341965" Nov 03 03:42:21.402802 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:21.402763 5605 generic.go:356] "Generic (PLEG): container finished" podID="bcb23d87-df37-4ad7-bfc7-e376c1e420b1" containerID="9fcb646b0e4640495364be43c59b8d68db717d3b47057fd4a5887845a76e7f97" exitCode=0 Nov 03 03:42:21.403188 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:21.402848 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff431bf-xxlhd" event={"ID":"bcb23d87-df37-4ad7-bfc7-e376c1e420b1","Type":"ContainerDied","Data":"9fcb646b0e4640495364be43c59b8d68db717d3b47057fd4a5887845a76e7f97"} Nov 03 03:42:22.533696 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:22.533669 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff431bf-xxlhd" Nov 03 03:42:22.553219 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:22.553193 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcb23d87-df37-4ad7-bfc7-e376c1e420b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcb23d87-df37-4ad7-bfc7-e376c1e420b1-maas-api-openshift-ca\") pod \"bcb23d87-df37-4ad7-bfc7-e376c1e420b1\" (UID: \"bcb23d87-df37-4ad7-bfc7-e376c1e420b1\") " Nov 03 03:42:22.553351 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:22.553268 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcb23d87-df37-4ad7-bfc7-e376c1e420b1-kube-api-access-9rhwv\" (UniqueName: \"kubernetes.io/projected/bcb23d87-df37-4ad7-bfc7-e376c1e420b1-kube-api-access-9rhwv\") pod \"bcb23d87-df37-4ad7-bfc7-e376c1e420b1\" (UID: \"bcb23d87-df37-4ad7-bfc7-e376c1e420b1\") " Nov 03 03:42:22.553351 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:22.553324 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcb23d87-df37-4ad7-bfc7-e376c1e420b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcb23d87-df37-4ad7-bfc7-e376c1e420b1-maas-api-xks-ca\") pod \"bcb23d87-df37-4ad7-bfc7-e376c1e420b1\" (UID: \"bcb23d87-df37-4ad7-bfc7-e376c1e420b1\") " Nov 03 03:42:22.553532 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:22.553509 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcb23d87-df37-4ad7-bfc7-e376c1e420b1-maas-api-openshift-ca" pod "bcb23d87-df37-4ad7-bfc7-e376c1e420b1" (UID: "bcb23d87-df37-4ad7-bfc7-e376c1e420b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:42:22.553661 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:22.553645 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcb23d87-df37-4ad7-bfc7-e376c1e420b1-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:42:22.555216 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:22.555192 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcb23d87-df37-4ad7-bfc7-e376c1e420b1-kube-api-access-9rhwv" pod "bcb23d87-df37-4ad7-bfc7-e376c1e420b1" (UID: "bcb23d87-df37-4ad7-bfc7-e376c1e420b1"). InnerVolumeSpecName "kube-api-access-9rhwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:42:22.555301 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:22.555276 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcb23d87-df37-4ad7-bfc7-e376c1e420b1-maas-api-xks-ca" pod "bcb23d87-df37-4ad7-bfc7-e376c1e420b1" (UID: "bcb23d87-df37-4ad7-bfc7-e376c1e420b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:42:22.655100 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:22.655076 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rhwv\" (UniqueName: \"kubernetes.io/projected/bcb23d87-df37-4ad7-bfc7-e376c1e420b1-kube-api-access-9rhwv\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:42:22.655100 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:22.655101 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcb23d87-df37-4ad7-bfc7-e376c1e420b1-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:42:23.412264 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:23.412234 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff431bf-xxlhd" Nov 03 03:42:23.412463 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:23.412276 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff431bf-xxlhd" event={"ID":"bcb23d87-df37-4ad7-bfc7-e376c1e420b1","Type":"ContainerDied","Data":"640974a453c41cb5ba2107ccfb7b57fbbf8ccac098d412b2b3bb23a355c52f55"} Nov 03 03:42:23.412463 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:23.412327 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="640974a453c41cb5ba2107ccfb7b57fbbf8ccac098d412b2b3bb23a355c52f55" Nov 03 03:42:23.911692 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:42:23.911653 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:42:30.228397 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:30.228344 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c12e58c-dc6j8"] Nov 03 03:42:30.228836 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:30.228696 5605 cpu_manager.go:414] "containerMap: removing container" podUID="bcb23d87-df37-4ad7-bfc7-e376c1e420b1" containerName="revoke-keys" Nov 03 03:42:30.228836 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:30.228707 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcb23d87-df37-4ad7-bfc7-e376c1e420b1" containerName="revoke-keys" Nov 03 03:42:30.228836 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:30.228785 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcb23d87-df37-4ad7-bfc7-e376c1e420b1" containerName="revoke-keys" Nov 03 03:42:30.231946 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:30.231924 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c12e58c-dc6j8" Nov 03 03:42:30.280923 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:30.280900 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:42:30.316915 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:30.316888 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f894000-0dfd-4326-9b28-d3ca8943e17a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c12e58c-dc6j8\" (UID: \"3f894000-0dfd-4326-9b28-d3ca8943e17a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c12e58c-dc6j8" Nov 03 03:42:30.317042 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:30.316933 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wmdx\" (UniqueName: \"kubernetes.io/projected/3f894000-0dfd-4326-9b28-d3ca8943e17a-kube-api-access-8wmdx\") pod \"maas-api-revoke-sub-1c12e58c-dc6j8\" (UID: \"3f894000-0dfd-4326-9b28-d3ca8943e17a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c12e58c-dc6j8" Nov 03 03:42:30.317042 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:30.317006 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f894000-0dfd-4326-9b28-d3ca8943e17a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c12e58c-dc6j8\" (UID: \"3f894000-0dfd-4326-9b28-d3ca8943e17a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c12e58c-dc6j8" Nov 03 03:42:30.327550 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:30.327523 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:42:30.327674 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:30.327549 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:42:30.327674 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:30.327560 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sb4qh\"" Nov 03 03:42:30.329390 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:30.329373 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c12e58c-dc6j8"] Nov 03 03:42:30.417733 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:30.417710 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wmdx\" (UniqueName: \"kubernetes.io/projected/3f894000-0dfd-4326-9b28-d3ca8943e17a-kube-api-access-8wmdx\") pod \"maas-api-revoke-sub-1c12e58c-dc6j8\" (UID: \"3f894000-0dfd-4326-9b28-d3ca8943e17a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c12e58c-dc6j8" Nov 03 03:42:30.417871 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:30.417737 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f894000-0dfd-4326-9b28-d3ca8943e17a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c12e58c-dc6j8\" (UID: \"3f894000-0dfd-4326-9b28-d3ca8943e17a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c12e58c-dc6j8" Nov 03 03:42:30.417871 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:30.417794 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f894000-0dfd-4326-9b28-d3ca8943e17a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c12e58c-dc6j8\" (UID: \"3f894000-0dfd-4326-9b28-d3ca8943e17a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c12e58c-dc6j8" Nov 03 03:42:30.418468 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:30.418448 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f894000-0dfd-4326-9b28-d3ca8943e17a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c12e58c-dc6j8\" (UID: \"3f894000-0dfd-4326-9b28-d3ca8943e17a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c12e58c-dc6j8" Nov 03 03:42:30.420033 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:30.420015 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f894000-0dfd-4326-9b28-d3ca8943e17a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c12e58c-dc6j8\" (UID: \"3f894000-0dfd-4326-9b28-d3ca8943e17a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c12e58c-dc6j8" Nov 03 03:42:30.471152 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:30.471130 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wmdx\" (UniqueName: \"kubernetes.io/projected/3f894000-0dfd-4326-9b28-d3ca8943e17a-kube-api-access-8wmdx\") pod \"maas-api-revoke-sub-1c12e58c-dc6j8\" (UID: \"3f894000-0dfd-4326-9b28-d3ca8943e17a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c12e58c-dc6j8" Nov 03 03:42:30.544681 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:30.544629 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c12e58c-dc6j8" Nov 03 03:42:30.909127 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:30.909097 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c12e58c-dc6j8"] Nov 03 03:42:30.915181 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:42:30.915153 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f894000_0dfd_4326_9b28_d3ca8943e17a.slice/crio-7d6e0219e861d193653f34caa480154ad6a4a2d85c22fc39691e1a633a33bd1d WatchSource:0}: Error finding container 7d6e0219e861d193653f34caa480154ad6a4a2d85c22fc39691e1a633a33bd1d: Status 404 returned error can't find the container with id 7d6e0219e861d193653f34caa480154ad6a4a2d85c22fc39691e1a633a33bd1d Nov 03 03:42:31.451976 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:31.451943 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c12e58c-dc6j8" event={"ID":"3f894000-0dfd-4326-9b28-d3ca8943e17a","Type":"ContainerStarted","Data":"8ace78d0f750409c42003a06140a51706e501c3043e1f6605992c2fee199edd9"} Nov 03 03:42:31.452336 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:31.452031 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c12e58c-dc6j8" event={"ID":"3f894000-0dfd-4326-9b28-d3ca8943e17a","Type":"ContainerStarted","Data":"7d6e0219e861d193653f34caa480154ad6a4a2d85c22fc39691e1a633a33bd1d"} Nov 03 03:42:31.558759 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:31.558712 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c12e58c-dc6j8" podStartSLOduration=1.558700944 podStartE2EDuration="1.558700944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:42:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:42:31.558087944 +0000 UTC m=+6628.817286099" watchObservedRunningTime="2026-11-03 03:42:31.558700944 +0000 UTC m=+6628.817899098" Nov 03 03:42:32.461514 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:32.461473 5605 generic.go:356] "Generic (PLEG): container finished" podID="3f894000-0dfd-4326-9b28-d3ca8943e17a" containerID="8ace78d0f750409c42003a06140a51706e501c3043e1f6605992c2fee199edd9" exitCode=0 Nov 03 03:42:32.462016 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:32.461564 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c12e58c-dc6j8" event={"ID":"3f894000-0dfd-4326-9b28-d3ca8943e17a","Type":"ContainerDied","Data":"8ace78d0f750409c42003a06140a51706e501c3043e1f6605992c2fee199edd9"} Nov 03 03:42:34.152427 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:34.152090 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c12e58c-dc6j8" Nov 03 03:42:34.180913 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:34.179756 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f894000-0dfd-4326-9b28-d3ca8943e17a-kube-api-access-8wmdx\" (UniqueName: \"kubernetes.io/projected/3f894000-0dfd-4326-9b28-d3ca8943e17a-kube-api-access-8wmdx\") pod \"3f894000-0dfd-4326-9b28-d3ca8943e17a\" (UID: \"3f894000-0dfd-4326-9b28-d3ca8943e17a\") " Nov 03 03:42:34.180913 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:34.179841 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f894000-0dfd-4326-9b28-d3ca8943e17a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f894000-0dfd-4326-9b28-d3ca8943e17a-maas-api-openshift-ca\") pod \"3f894000-0dfd-4326-9b28-d3ca8943e17a\" (UID: \"3f894000-0dfd-4326-9b28-d3ca8943e17a\") " Nov 03 03:42:34.180913 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:34.179935 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f894000-0dfd-4326-9b28-d3ca8943e17a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f894000-0dfd-4326-9b28-d3ca8943e17a-maas-api-xks-ca\") pod \"3f894000-0dfd-4326-9b28-d3ca8943e17a\" (UID: \"3f894000-0dfd-4326-9b28-d3ca8943e17a\") " Nov 03 03:42:34.187682 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:34.186802 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f894000-0dfd-4326-9b28-d3ca8943e17a-maas-api-openshift-ca" pod "3f894000-0dfd-4326-9b28-d3ca8943e17a" (UID: "3f894000-0dfd-4326-9b28-d3ca8943e17a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:42:34.189802 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:34.189766 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f894000-0dfd-4326-9b28-d3ca8943e17a-kube-api-access-8wmdx" pod "3f894000-0dfd-4326-9b28-d3ca8943e17a" (UID: "3f894000-0dfd-4326-9b28-d3ca8943e17a"). InnerVolumeSpecName "kube-api-access-8wmdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:42:34.197716 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:34.197682 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f894000-0dfd-4326-9b28-d3ca8943e17a-maas-api-xks-ca" pod "3f894000-0dfd-4326-9b28-d3ca8943e17a" (UID: "3f894000-0dfd-4326-9b28-d3ca8943e17a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:42:34.281192 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:34.281163 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f894000-0dfd-4326-9b28-d3ca8943e17a-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:42:34.281439 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:34.281418 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f894000-0dfd-4326-9b28-d3ca8943e17a-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:42:34.281582 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:34.281567 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wmdx\" (UniqueName: \"kubernetes.io/projected/3f894000-0dfd-4326-9b28-d3ca8943e17a-kube-api-access-8wmdx\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:42:34.493444 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:34.492823 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c12e58c-dc6j8" event={"ID":"3f894000-0dfd-4326-9b28-d3ca8943e17a","Type":"ContainerDied","Data":"7d6e0219e861d193653f34caa480154ad6a4a2d85c22fc39691e1a633a33bd1d"} Nov 03 03:42:34.493444 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:34.492887 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d6e0219e861d193653f34caa480154ad6a4a2d85c22fc39691e1a633a33bd1d" Nov 03 03:42:34.493444 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:34.492960 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c12e58c-dc6j8" Nov 03 03:42:40.984612 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:40.984577 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b8c07f5-r87q5"] Nov 03 03:42:40.985076 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:40.984918 5605 cpu_manager.go:414] "containerMap: removing container" podUID="3f894000-0dfd-4326-9b28-d3ca8943e17a" containerName="revoke-keys" Nov 03 03:42:40.985076 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:40.984929 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f894000-0dfd-4326-9b28-d3ca8943e17a" containerName="revoke-keys" Nov 03 03:42:40.985076 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:40.984991 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f894000-0dfd-4326-9b28-d3ca8943e17a" containerName="revoke-keys" Nov 03 03:42:40.988265 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:40.988244 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b8c07f5-r87q5" Nov 03 03:42:41.036903 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:41.036883 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:42:41.057323 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:41.057292 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltpfm\" (UniqueName: \"kubernetes.io/projected/6dfa0951-bb9c-4d22-adf2-0fee83200c2e-kube-api-access-ltpfm\") pod \"maas-api-revoke-sub-7b8c07f5-r87q5\" (UID: \"6dfa0951-bb9c-4d22-adf2-0fee83200c2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b8c07f5-r87q5" Nov 03 03:42:41.057455 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:41.057339 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dfa0951-bb9c-4d22-adf2-0fee83200c2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b8c07f5-r87q5\" (UID: \"6dfa0951-bb9c-4d22-adf2-0fee83200c2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b8c07f5-r87q5" Nov 03 03:42:41.057549 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:41.057520 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dfa0951-bb9c-4d22-adf2-0fee83200c2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b8c07f5-r87q5\" (UID: \"6dfa0951-bb9c-4d22-adf2-0fee83200c2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b8c07f5-r87q5" Nov 03 03:42:41.083517 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:41.083497 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:42:41.083630 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:41.083543 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:42:41.083687 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:41.083656 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sb4qh\"" Nov 03 03:42:41.085349 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:41.085332 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b8c07f5-r87q5"] Nov 03 03:42:41.158796 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:41.158775 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltpfm\" (UniqueName: \"kubernetes.io/projected/6dfa0951-bb9c-4d22-adf2-0fee83200c2e-kube-api-access-ltpfm\") pod \"maas-api-revoke-sub-7b8c07f5-r87q5\" (UID: \"6dfa0951-bb9c-4d22-adf2-0fee83200c2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b8c07f5-r87q5" Nov 03 03:42:41.158899 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:41.158811 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dfa0951-bb9c-4d22-adf2-0fee83200c2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b8c07f5-r87q5\" (UID: \"6dfa0951-bb9c-4d22-adf2-0fee83200c2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b8c07f5-r87q5" Nov 03 03:42:41.158899 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:41.158852 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dfa0951-bb9c-4d22-adf2-0fee83200c2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b8c07f5-r87q5\" (UID: \"6dfa0951-bb9c-4d22-adf2-0fee83200c2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b8c07f5-r87q5" Nov 03 03:42:41.159441 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:41.159423 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dfa0951-bb9c-4d22-adf2-0fee83200c2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b8c07f5-r87q5\" (UID: \"6dfa0951-bb9c-4d22-adf2-0fee83200c2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b8c07f5-r87q5" Nov 03 03:42:41.161160 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:41.161140 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dfa0951-bb9c-4d22-adf2-0fee83200c2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b8c07f5-r87q5\" (UID: \"6dfa0951-bb9c-4d22-adf2-0fee83200c2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b8c07f5-r87q5" Nov 03 03:42:41.211722 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:41.211697 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltpfm\" (UniqueName: \"kubernetes.io/projected/6dfa0951-bb9c-4d22-adf2-0fee83200c2e-kube-api-access-ltpfm\") pod \"maas-api-revoke-sub-7b8c07f5-r87q5\" (UID: \"6dfa0951-bb9c-4d22-adf2-0fee83200c2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b8c07f5-r87q5" Nov 03 03:42:41.300454 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:41.300434 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b8c07f5-r87q5" Nov 03 03:42:41.665956 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:41.665883 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b8c07f5-r87q5"] Nov 03 03:42:41.672162 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:42:41.672123 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dfa0951_bb9c_4d22_adf2_0fee83200c2e.slice/crio-479b1fe781580bac9bd1b21d52f663fcb46a536d02eafd0ae936aafe7d8bd236 WatchSource:0}: Error finding container 479b1fe781580bac9bd1b21d52f663fcb46a536d02eafd0ae936aafe7d8bd236: Status 404 returned error can't find the container with id 479b1fe781580bac9bd1b21d52f663fcb46a536d02eafd0ae936aafe7d8bd236 Nov 03 03:42:42.531208 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:42.531175 5605 generic.go:356] "Generic (PLEG): container finished" podID="6dfa0951-bb9c-4d22-adf2-0fee83200c2e" containerID="7c1492e7efc9fe3e28430318b54cca1f64329fd677850ee03974204096c80e0e" exitCode=0 Nov 03 03:42:42.531612 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:42.531251 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b8c07f5-r87q5" event={"ID":"6dfa0951-bb9c-4d22-adf2-0fee83200c2e","Type":"ContainerDied","Data":"7c1492e7efc9fe3e28430318b54cca1f64329fd677850ee03974204096c80e0e"} Nov 03 03:42:42.531612 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:42.531290 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b8c07f5-r87q5" event={"ID":"6dfa0951-bb9c-4d22-adf2-0fee83200c2e","Type":"ContainerStarted","Data":"479b1fe781580bac9bd1b21d52f663fcb46a536d02eafd0ae936aafe7d8bd236"} Nov 03 03:42:43.661410 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:43.661385 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b8c07f5-r87q5" Nov 03 03:42:43.779375 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:43.779339 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6dfa0951-bb9c-4d22-adf2-0fee83200c2e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dfa0951-bb9c-4d22-adf2-0fee83200c2e-maas-api-xks-ca\") pod \"6dfa0951-bb9c-4d22-adf2-0fee83200c2e\" (UID: \"6dfa0951-bb9c-4d22-adf2-0fee83200c2e\") " Nov 03 03:42:43.779533 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:43.779516 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dfa0951-bb9c-4d22-adf2-0fee83200c2e-kube-api-access-ltpfm\" (UniqueName: \"kubernetes.io/projected/6dfa0951-bb9c-4d22-adf2-0fee83200c2e-kube-api-access-ltpfm\") pod \"6dfa0951-bb9c-4d22-adf2-0fee83200c2e\" (UID: \"6dfa0951-bb9c-4d22-adf2-0fee83200c2e\") " Nov 03 03:42:43.779587 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:43.779553 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dfa0951-bb9c-4d22-adf2-0fee83200c2e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dfa0951-bb9c-4d22-adf2-0fee83200c2e-maas-api-openshift-ca\") pod \"6dfa0951-bb9c-4d22-adf2-0fee83200c2e\" (UID: \"6dfa0951-bb9c-4d22-adf2-0fee83200c2e\") " Nov 03 03:42:43.779927 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:43.779903 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dfa0951-bb9c-4d22-adf2-0fee83200c2e-maas-api-openshift-ca" pod "6dfa0951-bb9c-4d22-adf2-0fee83200c2e" (UID: "6dfa0951-bb9c-4d22-adf2-0fee83200c2e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:42:43.781454 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:43.781426 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6dfa0951-bb9c-4d22-adf2-0fee83200c2e-maas-api-xks-ca" pod "6dfa0951-bb9c-4d22-adf2-0fee83200c2e" (UID: "6dfa0951-bb9c-4d22-adf2-0fee83200c2e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:42:43.781650 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:43.781630 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dfa0951-bb9c-4d22-adf2-0fee83200c2e-kube-api-access-ltpfm" pod "6dfa0951-bb9c-4d22-adf2-0fee83200c2e" (UID: "6dfa0951-bb9c-4d22-adf2-0fee83200c2e"). InnerVolumeSpecName "kube-api-access-ltpfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:42:43.880675 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:43.880654 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dfa0951-bb9c-4d22-adf2-0fee83200c2e-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:42:43.880675 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:43.880674 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltpfm\" (UniqueName: \"kubernetes.io/projected/6dfa0951-bb9c-4d22-adf2-0fee83200c2e-kube-api-access-ltpfm\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:42:43.880806 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:43.880685 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dfa0951-bb9c-4d22-adf2-0fee83200c2e-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:42:44.542679 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:44.542647 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b8c07f5-r87q5" Nov 03 03:42:44.542875 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:44.542679 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b8c07f5-r87q5" event={"ID":"6dfa0951-bb9c-4d22-adf2-0fee83200c2e","Type":"ContainerDied","Data":"479b1fe781580bac9bd1b21d52f663fcb46a536d02eafd0ae936aafe7d8bd236"} Nov 03 03:42:44.542875 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:44.542733 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="479b1fe781580bac9bd1b21d52f663fcb46a536d02eafd0ae936aafe7d8bd236" Nov 03 03:42:51.733998 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:51.733963 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fb407af-ldbpf"] Nov 03 03:42:51.734407 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:51.734318 5605 cpu_manager.go:414] "containerMap: removing container" podUID="6dfa0951-bb9c-4d22-adf2-0fee83200c2e" containerName="revoke-keys" Nov 03 03:42:51.734407 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:51.734330 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dfa0951-bb9c-4d22-adf2-0fee83200c2e" containerName="revoke-keys" Nov 03 03:42:51.734499 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:51.734422 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dfa0951-bb9c-4d22-adf2-0fee83200c2e" containerName="revoke-keys" Nov 03 03:42:51.744407 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:51.744378 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb407af-ldbpf" Nov 03 03:42:51.793771 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:51.793745 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:42:51.840324 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:51.840300 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:42:51.840324 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:51.840309 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:42:51.840547 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:51.840465 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sb4qh\"" Nov 03 03:42:51.842240 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:51.842221 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fb407af-ldbpf"] Nov 03 03:42:51.882255 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:51.882229 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62e6ef37-20e0-4e7e-87af-b87799101bf8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fb407af-ldbpf\" (UID: \"62e6ef37-20e0-4e7e-87af-b87799101bf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb407af-ldbpf" Nov 03 03:42:51.882399 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:51.882314 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62e6ef37-20e0-4e7e-87af-b87799101bf8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fb407af-ldbpf\" (UID: \"62e6ef37-20e0-4e7e-87af-b87799101bf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb407af-ldbpf" Nov 03 03:42:51.882453 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:51.882403 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87brm\" (UniqueName: \"kubernetes.io/projected/62e6ef37-20e0-4e7e-87af-b87799101bf8-kube-api-access-87brm\") pod \"maas-api-revoke-sub-8fb407af-ldbpf\" (UID: \"62e6ef37-20e0-4e7e-87af-b87799101bf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb407af-ldbpf" Nov 03 03:42:51.983738 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:51.983713 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87brm\" (UniqueName: \"kubernetes.io/projected/62e6ef37-20e0-4e7e-87af-b87799101bf8-kube-api-access-87brm\") pod \"maas-api-revoke-sub-8fb407af-ldbpf\" (UID: \"62e6ef37-20e0-4e7e-87af-b87799101bf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb407af-ldbpf" Nov 03 03:42:51.983876 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:51.983750 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62e6ef37-20e0-4e7e-87af-b87799101bf8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fb407af-ldbpf\" (UID: \"62e6ef37-20e0-4e7e-87af-b87799101bf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb407af-ldbpf" Nov 03 03:42:51.983876 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:51.983795 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62e6ef37-20e0-4e7e-87af-b87799101bf8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fb407af-ldbpf\" (UID: \"62e6ef37-20e0-4e7e-87af-b87799101bf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb407af-ldbpf" Nov 03 03:42:51.984531 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:51.984482 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62e6ef37-20e0-4e7e-87af-b87799101bf8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fb407af-ldbpf\" (UID: \"62e6ef37-20e0-4e7e-87af-b87799101bf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb407af-ldbpf" Nov 03 03:42:51.986021 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:51.986004 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62e6ef37-20e0-4e7e-87af-b87799101bf8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fb407af-ldbpf\" (UID: \"62e6ef37-20e0-4e7e-87af-b87799101bf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb407af-ldbpf" Nov 03 03:42:52.037289 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:52.037258 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87brm\" (UniqueName: \"kubernetes.io/projected/62e6ef37-20e0-4e7e-87af-b87799101bf8-kube-api-access-87brm\") pod \"maas-api-revoke-sub-8fb407af-ldbpf\" (UID: \"62e6ef37-20e0-4e7e-87af-b87799101bf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb407af-ldbpf" Nov 03 03:42:52.057596 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:52.057576 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb407af-ldbpf" Nov 03 03:42:52.424211 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:52.424171 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fb407af-ldbpf"] Nov 03 03:42:52.430573 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:42:52.430539 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62e6ef37_20e0_4e7e_87af_b87799101bf8.slice/crio-bff68f7baafc461778a990c2dce5aa1a4bb62e0b8fe5461c13c15cccc7a94495 WatchSource:0}: Error finding container bff68f7baafc461778a990c2dce5aa1a4bb62e0b8fe5461c13c15cccc7a94495: Status 404 returned error can't find the container with id bff68f7baafc461778a990c2dce5aa1a4bb62e0b8fe5461c13c15cccc7a94495 Nov 03 03:42:52.601928 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:52.601895 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb407af-ldbpf" event={"ID":"62e6ef37-20e0-4e7e-87af-b87799101bf8","Type":"ContainerStarted","Data":"bff68f7baafc461778a990c2dce5aa1a4bb62e0b8fe5461c13c15cccc7a94495"} Nov 03 03:42:53.608297 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:53.608264 5605 generic.go:356] "Generic (PLEG): container finished" podID="62e6ef37-20e0-4e7e-87af-b87799101bf8" containerID="50ed4876a3519228c19131b0f59bc00e75dbad55ed40667d88fb1ba85c2a5d78" exitCode=0 Nov 03 03:42:53.608719 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:53.608328 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb407af-ldbpf" event={"ID":"62e6ef37-20e0-4e7e-87af-b87799101bf8","Type":"ContainerDied","Data":"50ed4876a3519228c19131b0f59bc00e75dbad55ed40667d88fb1ba85c2a5d78"} Nov 03 03:42:53.911924 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:42:53.911839 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:42:54.736414 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:54.736380 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb407af-ldbpf" Nov 03 03:42:54.809661 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:54.809633 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62e6ef37-20e0-4e7e-87af-b87799101bf8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62e6ef37-20e0-4e7e-87af-b87799101bf8-maas-api-xks-ca\") pod \"62e6ef37-20e0-4e7e-87af-b87799101bf8\" (UID: \"62e6ef37-20e0-4e7e-87af-b87799101bf8\") " Nov 03 03:42:54.809771 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:54.809681 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62e6ef37-20e0-4e7e-87af-b87799101bf8-kube-api-access-87brm\" (UniqueName: \"kubernetes.io/projected/62e6ef37-20e0-4e7e-87af-b87799101bf8-kube-api-access-87brm\") pod \"62e6ef37-20e0-4e7e-87af-b87799101bf8\" (UID: \"62e6ef37-20e0-4e7e-87af-b87799101bf8\") " Nov 03 03:42:54.809771 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:54.809737 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62e6ef37-20e0-4e7e-87af-b87799101bf8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62e6ef37-20e0-4e7e-87af-b87799101bf8-maas-api-openshift-ca\") pod \"62e6ef37-20e0-4e7e-87af-b87799101bf8\" (UID: \"62e6ef37-20e0-4e7e-87af-b87799101bf8\") " Nov 03 03:42:54.810117 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:54.810095 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62e6ef37-20e0-4e7e-87af-b87799101bf8-maas-api-openshift-ca" pod "62e6ef37-20e0-4e7e-87af-b87799101bf8" (UID: "62e6ef37-20e0-4e7e-87af-b87799101bf8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:42:54.811629 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:54.811610 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62e6ef37-20e0-4e7e-87af-b87799101bf8-maas-api-xks-ca" pod "62e6ef37-20e0-4e7e-87af-b87799101bf8" (UID: "62e6ef37-20e0-4e7e-87af-b87799101bf8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:42:54.811793 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:54.811774 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62e6ef37-20e0-4e7e-87af-b87799101bf8-kube-api-access-87brm" pod "62e6ef37-20e0-4e7e-87af-b87799101bf8" (UID: "62e6ef37-20e0-4e7e-87af-b87799101bf8"). InnerVolumeSpecName "kube-api-access-87brm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:42:54.911237 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:54.911183 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62e6ef37-20e0-4e7e-87af-b87799101bf8-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:42:54.911237 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:54.911207 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87brm\" (UniqueName: \"kubernetes.io/projected/62e6ef37-20e0-4e7e-87af-b87799101bf8-kube-api-access-87brm\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:42:54.911237 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:54.911218 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62e6ef37-20e0-4e7e-87af-b87799101bf8-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:42:55.618880 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:55.618848 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb407af-ldbpf" Nov 03 03:42:55.619061 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:55.618848 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb407af-ldbpf" event={"ID":"62e6ef37-20e0-4e7e-87af-b87799101bf8","Type":"ContainerDied","Data":"bff68f7baafc461778a990c2dce5aa1a4bb62e0b8fe5461c13c15cccc7a94495"} Nov 03 03:42:55.619061 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:42:55.618971 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bff68f7baafc461778a990c2dce5aa1a4bb62e0b8fe5461c13c15cccc7a94495" Nov 03 03:43:02.484885 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:02.484850 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-502f081c-2pj8b"] Nov 03 03:43:02.485291 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:02.485185 5605 cpu_manager.go:414] "containerMap: removing container" podUID="62e6ef37-20e0-4e7e-87af-b87799101bf8" containerName="revoke-keys" Nov 03 03:43:02.485291 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:02.485196 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62e6ef37-20e0-4e7e-87af-b87799101bf8" containerName="revoke-keys" Nov 03 03:43:02.485291 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:02.485266 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="62e6ef37-20e0-4e7e-87af-b87799101bf8" containerName="revoke-keys" Nov 03 03:43:02.488322 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:02.488298 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-502f081c-2pj8b" Nov 03 03:43:02.537660 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:02.537638 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:43:02.573485 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:02.573455 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfq5h\" (UniqueName: \"kubernetes.io/projected/03f5364e-f3b5-4591-818d-44a17155e6de-kube-api-access-kfq5h\") pod \"maas-api-revoke-sub-502f081c-2pj8b\" (UID: \"03f5364e-f3b5-4591-818d-44a17155e6de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-502f081c-2pj8b" Nov 03 03:43:02.573644 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:02.573503 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03f5364e-f3b5-4591-818d-44a17155e6de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-502f081c-2pj8b\" (UID: \"03f5364e-f3b5-4591-818d-44a17155e6de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-502f081c-2pj8b" Nov 03 03:43:02.573722 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:02.573663 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03f5364e-f3b5-4591-818d-44a17155e6de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-502f081c-2pj8b\" (UID: \"03f5364e-f3b5-4591-818d-44a17155e6de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-502f081c-2pj8b" Nov 03 03:43:02.584334 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:02.584316 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:43:02.584442 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:02.584417 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:43:02.584527 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:02.584511 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sb4qh\"" Nov 03 03:43:02.586448 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:02.586424 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-502f081c-2pj8b"] Nov 03 03:43:02.674670 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:02.674651 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03f5364e-f3b5-4591-818d-44a17155e6de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-502f081c-2pj8b\" (UID: \"03f5364e-f3b5-4591-818d-44a17155e6de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-502f081c-2pj8b" Nov 03 03:43:02.674779 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:02.674702 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfq5h\" (UniqueName: \"kubernetes.io/projected/03f5364e-f3b5-4591-818d-44a17155e6de-kube-api-access-kfq5h\") pod \"maas-api-revoke-sub-502f081c-2pj8b\" (UID: \"03f5364e-f3b5-4591-818d-44a17155e6de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-502f081c-2pj8b" Nov 03 03:43:02.674779 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:02.674734 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03f5364e-f3b5-4591-818d-44a17155e6de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-502f081c-2pj8b\" (UID: \"03f5364e-f3b5-4591-818d-44a17155e6de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-502f081c-2pj8b" Nov 03 03:43:02.675330 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:02.675309 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03f5364e-f3b5-4591-818d-44a17155e6de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-502f081c-2pj8b\" (UID: \"03f5364e-f3b5-4591-818d-44a17155e6de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-502f081c-2pj8b" Nov 03 03:43:02.677148 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:02.677126 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03f5364e-f3b5-4591-818d-44a17155e6de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-502f081c-2pj8b\" (UID: \"03f5364e-f3b5-4591-818d-44a17155e6de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-502f081c-2pj8b" Nov 03 03:43:02.728129 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:02.728101 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfq5h\" (UniqueName: \"kubernetes.io/projected/03f5364e-f3b5-4591-818d-44a17155e6de-kube-api-access-kfq5h\") pod \"maas-api-revoke-sub-502f081c-2pj8b\" (UID: \"03f5364e-f3b5-4591-818d-44a17155e6de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-502f081c-2pj8b" Nov 03 03:43:02.801104 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:02.801074 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-502f081c-2pj8b" Nov 03 03:43:03.166591 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:03.166519 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-502f081c-2pj8b"] Nov 03 03:43:03.172437 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:43:03.172406 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03f5364e_f3b5_4591_818d_44a17155e6de.slice/crio-9303af9a3c4d64aadeeb6fa473294c14da45fc90adce15d02863a2077496250b WatchSource:0}: Error finding container 9303af9a3c4d64aadeeb6fa473294c14da45fc90adce15d02863a2077496250b: Status 404 returned error can't find the container with id 9303af9a3c4d64aadeeb6fa473294c14da45fc90adce15d02863a2077496250b Nov 03 03:43:03.660947 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:03.660901 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-502f081c-2pj8b" event={"ID":"03f5364e-f3b5-4591-818d-44a17155e6de","Type":"ContainerStarted","Data":"35ec1be3b3fdc3f3e70fba1b18b28fa2083276af5dc8a5188953d78d9363518c"} Nov 03 03:43:03.661296 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:03.661012 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-502f081c-2pj8b" event={"ID":"03f5364e-f3b5-4591-818d-44a17155e6de","Type":"ContainerStarted","Data":"9303af9a3c4d64aadeeb6fa473294c14da45fc90adce15d02863a2077496250b"} Nov 03 03:43:04.668081 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:04.668044 5605 generic.go:356] "Generic (PLEG): container finished" podID="03f5364e-f3b5-4591-818d-44a17155e6de" containerID="35ec1be3b3fdc3f3e70fba1b18b28fa2083276af5dc8a5188953d78d9363518c" exitCode=0 Nov 03 03:43:04.668553 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:04.668138 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-502f081c-2pj8b" event={"ID":"03f5364e-f3b5-4591-818d-44a17155e6de","Type":"ContainerDied","Data":"35ec1be3b3fdc3f3e70fba1b18b28fa2083276af5dc8a5188953d78d9363518c"} Nov 03 03:43:05.900866 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:05.900529 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-502f081c-2pj8b" Nov 03 03:43:06.006435 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:06.005605 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03f5364e-f3b5-4591-818d-44a17155e6de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03f5364e-f3b5-4591-818d-44a17155e6de-maas-api-xks-ca\") pod \"03f5364e-f3b5-4591-818d-44a17155e6de\" (UID: \"03f5364e-f3b5-4591-818d-44a17155e6de\") " Nov 03 03:43:06.006435 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:06.005741 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03f5364e-f3b5-4591-818d-44a17155e6de-kube-api-access-kfq5h\" (UniqueName: \"kubernetes.io/projected/03f5364e-f3b5-4591-818d-44a17155e6de-kube-api-access-kfq5h\") pod \"03f5364e-f3b5-4591-818d-44a17155e6de\" (UID: \"03f5364e-f3b5-4591-818d-44a17155e6de\") " Nov 03 03:43:06.006435 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:06.005806 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03f5364e-f3b5-4591-818d-44a17155e6de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03f5364e-f3b5-4591-818d-44a17155e6de-maas-api-openshift-ca\") pod \"03f5364e-f3b5-4591-818d-44a17155e6de\" (UID: \"03f5364e-f3b5-4591-818d-44a17155e6de\") " Nov 03 03:43:06.009283 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:06.007429 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03f5364e-f3b5-4591-818d-44a17155e6de-maas-api-openshift-ca" pod "03f5364e-f3b5-4591-818d-44a17155e6de" (UID: "03f5364e-f3b5-4591-818d-44a17155e6de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:43:06.009840 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:06.009802 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03f5364e-f3b5-4591-818d-44a17155e6de-maas-api-xks-ca" pod "03f5364e-f3b5-4591-818d-44a17155e6de" (UID: "03f5364e-f3b5-4591-818d-44a17155e6de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:43:06.041617 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:06.024469 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03f5364e-f3b5-4591-818d-44a17155e6de-kube-api-access-kfq5h" pod "03f5364e-f3b5-4591-818d-44a17155e6de" (UID: "03f5364e-f3b5-4591-818d-44a17155e6de"). InnerVolumeSpecName "kube-api-access-kfq5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:43:06.132885 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:06.106801 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03f5364e-f3b5-4591-818d-44a17155e6de-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:43:06.132885 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:06.106839 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfq5h\" (UniqueName: \"kubernetes.io/projected/03f5364e-f3b5-4591-818d-44a17155e6de-kube-api-access-kfq5h\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:43:06.132885 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:06.106857 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03f5364e-f3b5-4591-818d-44a17155e6de-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:43:06.690382 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:06.689847 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-502f081c-2pj8b" event={"ID":"03f5364e-f3b5-4591-818d-44a17155e6de","Type":"ContainerDied","Data":"9303af9a3c4d64aadeeb6fa473294c14da45fc90adce15d02863a2077496250b"} Nov 03 03:43:06.690382 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:06.689913 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9303af9a3c4d64aadeeb6fa473294c14da45fc90adce15d02863a2077496250b" Nov 03 03:43:06.690382 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:06.689989 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-502f081c-2pj8b" Nov 03 03:43:13.292075 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:13.291991 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61635b25-cjf5t"] Nov 03 03:43:13.292683 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:13.292583 5605 cpu_manager.go:414] "containerMap: removing container" podUID="03f5364e-f3b5-4591-818d-44a17155e6de" containerName="revoke-keys" Nov 03 03:43:13.292683 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:13.292605 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03f5364e-f3b5-4591-818d-44a17155e6de" containerName="revoke-keys" Nov 03 03:43:13.292820 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:13.292701 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="03f5364e-f3b5-4591-818d-44a17155e6de" containerName="revoke-keys" Nov 03 03:43:13.296183 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:13.296162 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61635b25-cjf5t" Nov 03 03:43:13.345576 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:13.345546 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:43:13.392263 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:13.392228 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:43:13.392423 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:13.392287 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:43:13.392423 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:13.392407 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sb4qh\"" Nov 03 03:43:13.394263 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:13.394242 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61635b25-cjf5t"] Nov 03 03:43:13.407626 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:13.407599 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsdhs\" (UniqueName: \"kubernetes.io/projected/94ac84ed-63ec-4ce5-ad6c-0e40b5f89c86-kube-api-access-gsdhs\") pod \"maas-api-revoke-sub-61635b25-cjf5t\" (UID: \"94ac84ed-63ec-4ce5-ad6c-0e40b5f89c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61635b25-cjf5t" Nov 03 03:43:13.407721 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:13.407656 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94ac84ed-63ec-4ce5-ad6c-0e40b5f89c86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61635b25-cjf5t\" (UID: \"94ac84ed-63ec-4ce5-ad6c-0e40b5f89c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61635b25-cjf5t" Nov 03 03:43:13.407767 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:13.407757 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94ac84ed-63ec-4ce5-ad6c-0e40b5f89c86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61635b25-cjf5t\" (UID: \"94ac84ed-63ec-4ce5-ad6c-0e40b5f89c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61635b25-cjf5t" Nov 03 03:43:13.508538 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:13.508515 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsdhs\" (UniqueName: \"kubernetes.io/projected/94ac84ed-63ec-4ce5-ad6c-0e40b5f89c86-kube-api-access-gsdhs\") pod \"maas-api-revoke-sub-61635b25-cjf5t\" (UID: \"94ac84ed-63ec-4ce5-ad6c-0e40b5f89c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61635b25-cjf5t" Nov 03 03:43:13.508641 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:13.508562 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94ac84ed-63ec-4ce5-ad6c-0e40b5f89c86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61635b25-cjf5t\" (UID: \"94ac84ed-63ec-4ce5-ad6c-0e40b5f89c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61635b25-cjf5t" Nov 03 03:43:13.508641 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:13.508600 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94ac84ed-63ec-4ce5-ad6c-0e40b5f89c86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61635b25-cjf5t\" (UID: \"94ac84ed-63ec-4ce5-ad6c-0e40b5f89c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61635b25-cjf5t" Nov 03 03:43:13.509333 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:13.509309 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94ac84ed-63ec-4ce5-ad6c-0e40b5f89c86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61635b25-cjf5t\" (UID: \"94ac84ed-63ec-4ce5-ad6c-0e40b5f89c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61635b25-cjf5t" Nov 03 03:43:13.510980 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:13.510962 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94ac84ed-63ec-4ce5-ad6c-0e40b5f89c86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61635b25-cjf5t\" (UID: \"94ac84ed-63ec-4ce5-ad6c-0e40b5f89c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61635b25-cjf5t" Nov 03 03:43:13.561753 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:13.561732 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsdhs\" (UniqueName: \"kubernetes.io/projected/94ac84ed-63ec-4ce5-ad6c-0e40b5f89c86-kube-api-access-gsdhs\") pod \"maas-api-revoke-sub-61635b25-cjf5t\" (UID: \"94ac84ed-63ec-4ce5-ad6c-0e40b5f89c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61635b25-cjf5t" Nov 03 03:43:13.610028 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:13.610006 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61635b25-cjf5t" Nov 03 03:43:13.976109 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:13.976032 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61635b25-cjf5t"] Nov 03 03:43:13.982686 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:43:13.982650 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94ac84ed_63ec_4ce5_ad6c_0e40b5f89c86.slice/crio-eb4be1d33c6b173d9135fe9ff384687087ae2da358b12b3f335d569ba5b17938 WatchSource:0}: Error finding container eb4be1d33c6b173d9135fe9ff384687087ae2da358b12b3f335d569ba5b17938: Status 404 returned error can't find the container with id eb4be1d33c6b173d9135fe9ff384687087ae2da358b12b3f335d569ba5b17938 Nov 03 03:43:14.736720 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:14.736681 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61635b25-cjf5t" event={"ID":"94ac84ed-63ec-4ce5-ad6c-0e40b5f89c86","Type":"ContainerStarted","Data":"c3fb7dfe0568412f2f0a36fc99fd1b64104644a46e43090ecbc0a78749a03087"} Nov 03 03:43:14.737090 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:14.736745 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61635b25-cjf5t" event={"ID":"94ac84ed-63ec-4ce5-ad6c-0e40b5f89c86","Type":"ContainerStarted","Data":"eb4be1d33c6b173d9135fe9ff384687087ae2da358b12b3f335d569ba5b17938"} Nov 03 03:43:14.843412 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:14.843345 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61635b25-cjf5t" podStartSLOduration=1.84333266 podStartE2EDuration="1.84333266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:43:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:43:14.842201411 +0000 UTC m=+6672.101399564" watchObservedRunningTime="2026-11-03 03:43:14.84333266 +0000 UTC m=+6672.102530813" Nov 03 03:43:15.747933 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:15.747890 5605 generic.go:356] "Generic (PLEG): container finished" podID="94ac84ed-63ec-4ce5-ad6c-0e40b5f89c86" containerID="c3fb7dfe0568412f2f0a36fc99fd1b64104644a46e43090ecbc0a78749a03087" exitCode=0 Nov 03 03:43:15.748503 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:15.747960 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61635b25-cjf5t" event={"ID":"94ac84ed-63ec-4ce5-ad6c-0e40b5f89c86","Type":"ContainerDied","Data":"c3fb7dfe0568412f2f0a36fc99fd1b64104644a46e43090ecbc0a78749a03087"} Nov 03 03:43:17.079379 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:17.074153 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c6e4f9a-z2m2v"] Nov 03 03:43:17.185522 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:17.185484 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c6e4f9a-z2m2v"] Nov 03 03:43:17.409737 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:17.409708 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61635b25-cjf5t" Nov 03 03:43:17.494380 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:17.491770 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94ac84ed-63ec-4ce5-ad6c-0e40b5f89c86-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94ac84ed-63ec-4ce5-ad6c-0e40b5f89c86-maas-api-xks-ca\") pod \"94ac84ed-63ec-4ce5-ad6c-0e40b5f89c86\" (UID: \"94ac84ed-63ec-4ce5-ad6c-0e40b5f89c86\") " Nov 03 03:43:17.494380 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:17.491840 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94ac84ed-63ec-4ce5-ad6c-0e40b5f89c86-kube-api-access-gsdhs\" (UniqueName: \"kubernetes.io/projected/94ac84ed-63ec-4ce5-ad6c-0e40b5f89c86-kube-api-access-gsdhs\") pod \"94ac84ed-63ec-4ce5-ad6c-0e40b5f89c86\" (UID: \"94ac84ed-63ec-4ce5-ad6c-0e40b5f89c86\") " Nov 03 03:43:17.494380 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:17.491909 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94ac84ed-63ec-4ce5-ad6c-0e40b5f89c86-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94ac84ed-63ec-4ce5-ad6c-0e40b5f89c86-maas-api-openshift-ca\") pod \"94ac84ed-63ec-4ce5-ad6c-0e40b5f89c86\" (UID: \"94ac84ed-63ec-4ce5-ad6c-0e40b5f89c86\") " Nov 03 03:43:17.494380 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:17.493340 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94ac84ed-63ec-4ce5-ad6c-0e40b5f89c86-maas-api-openshift-ca" pod "94ac84ed-63ec-4ce5-ad6c-0e40b5f89c86" (UID: "94ac84ed-63ec-4ce5-ad6c-0e40b5f89c86"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:43:17.516380 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:17.506530 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94ac84ed-63ec-4ce5-ad6c-0e40b5f89c86-kube-api-access-gsdhs" pod "94ac84ed-63ec-4ce5-ad6c-0e40b5f89c86" (UID: "94ac84ed-63ec-4ce5-ad6c-0e40b5f89c86"). InnerVolumeSpecName "kube-api-access-gsdhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:43:17.516380 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:17.506767 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94ac84ed-63ec-4ce5-ad6c-0e40b5f89c86-maas-api-xks-ca" pod "94ac84ed-63ec-4ce5-ad6c-0e40b5f89c86" (UID: "94ac84ed-63ec-4ce5-ad6c-0e40b5f89c86"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:43:17.685432 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:17.685223 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94ac84ed-63ec-4ce5-ad6c-0e40b5f89c86-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:43:17.685432 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:17.685260 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsdhs\" (UniqueName: \"kubernetes.io/projected/94ac84ed-63ec-4ce5-ad6c-0e40b5f89c86-kube-api-access-gsdhs\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:43:17.685432 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:17.685279 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94ac84ed-63ec-4ce5-ad6c-0e40b5f89c86-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:43:17.775379 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:17.775056 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61635b25-cjf5t" event={"ID":"94ac84ed-63ec-4ce5-ad6c-0e40b5f89c86","Type":"ContainerDied","Data":"eb4be1d33c6b173d9135fe9ff384687087ae2da358b12b3f335d569ba5b17938"} Nov 03 03:43:17.775379 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:17.775120 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb4be1d33c6b173d9135fe9ff384687087ae2da358b12b3f335d569ba5b17938" Nov 03 03:43:17.775379 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:17.775188 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61635b25-cjf5t" Nov 03 03:43:18.847984 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:18.847951 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee6e54e2-3009-41a1-a617-d2d4dc6166d7" path="/var/lib/kubelet/pods/ee6e54e2-3009-41a1-a617-d2d4dc6166d7/volumes" Nov 03 03:43:23.911207 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:43:23.911169 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:43:24.039461 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:24.039428 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4583cec-l4kzb"] Nov 03 03:43:24.039814 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:24.039800 5605 cpu_manager.go:414] "containerMap: removing container" podUID="94ac84ed-63ec-4ce5-ad6c-0e40b5f89c86" containerName="revoke-keys" Nov 03 03:43:24.039862 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:24.039816 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94ac84ed-63ec-4ce5-ad6c-0e40b5f89c86" containerName="revoke-keys" Nov 03 03:43:24.039899 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:24.039887 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="94ac84ed-63ec-4ce5-ad6c-0e40b5f89c86" containerName="revoke-keys" Nov 03 03:43:24.043247 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:24.043223 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4583cec-l4kzb" Nov 03 03:43:24.045105 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:24.045072 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87101208-30f4-4ce8-b2b8-9023537f9aac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4583cec-l4kzb\" (UID: \"87101208-30f4-4ce8-b2b8-9023537f9aac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4583cec-l4kzb" Nov 03 03:43:24.045209 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:24.045124 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jwj8\" (UniqueName: \"kubernetes.io/projected/87101208-30f4-4ce8-b2b8-9023537f9aac-kube-api-access-6jwj8\") pod \"maas-api-revoke-sub-a4583cec-l4kzb\" (UID: \"87101208-30f4-4ce8-b2b8-9023537f9aac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4583cec-l4kzb" Nov 03 03:43:24.045209 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:24.045164 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87101208-30f4-4ce8-b2b8-9023537f9aac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4583cec-l4kzb\" (UID: \"87101208-30f4-4ce8-b2b8-9023537f9aac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4583cec-l4kzb" Nov 03 03:43:24.139130 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:24.139101 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:43:24.139130 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:24.139116 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sb4qh\"" Nov 03 03:43:24.139130 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:24.139101 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:43:24.139349 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:24.139105 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:43:24.141272 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:24.141248 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4583cec-l4kzb"] Nov 03 03:43:24.146750 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:24.146730 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87101208-30f4-4ce8-b2b8-9023537f9aac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4583cec-l4kzb\" (UID: \"87101208-30f4-4ce8-b2b8-9023537f9aac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4583cec-l4kzb" Nov 03 03:43:24.146889 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:24.146759 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jwj8\" (UniqueName: \"kubernetes.io/projected/87101208-30f4-4ce8-b2b8-9023537f9aac-kube-api-access-6jwj8\") pod \"maas-api-revoke-sub-a4583cec-l4kzb\" (UID: \"87101208-30f4-4ce8-b2b8-9023537f9aac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4583cec-l4kzb" Nov 03 03:43:24.146889 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:24.146786 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87101208-30f4-4ce8-b2b8-9023537f9aac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4583cec-l4kzb\" (UID: \"87101208-30f4-4ce8-b2b8-9023537f9aac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4583cec-l4kzb" Nov 03 03:43:24.147427 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:24.147408 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87101208-30f4-4ce8-b2b8-9023537f9aac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4583cec-l4kzb\" (UID: \"87101208-30f4-4ce8-b2b8-9023537f9aac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4583cec-l4kzb" Nov 03 03:43:24.149182 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:24.149156 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87101208-30f4-4ce8-b2b8-9023537f9aac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4583cec-l4kzb\" (UID: \"87101208-30f4-4ce8-b2b8-9023537f9aac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4583cec-l4kzb" Nov 03 03:43:24.201870 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:24.201817 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jwj8\" (UniqueName: \"kubernetes.io/projected/87101208-30f4-4ce8-b2b8-9023537f9aac-kube-api-access-6jwj8\") pod \"maas-api-revoke-sub-a4583cec-l4kzb\" (UID: \"87101208-30f4-4ce8-b2b8-9023537f9aac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4583cec-l4kzb" Nov 03 03:43:24.356234 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:24.356211 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4583cec-l4kzb" Nov 03 03:43:24.723971 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:24.723931 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4583cec-l4kzb"] Nov 03 03:43:24.730799 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:43:24.730770 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87101208_30f4_4ce8_b2b8_9023537f9aac.slice/crio-b3d5f4d530b8de93bd4df8a9c94cf508236911678be933e6a37f1ccb3777f0e7 WatchSource:0}: Error finding container b3d5f4d530b8de93bd4df8a9c94cf508236911678be933e6a37f1ccb3777f0e7: Status 404 returned error can't find the container with id b3d5f4d530b8de93bd4df8a9c94cf508236911678be933e6a37f1ccb3777f0e7 Nov 03 03:43:24.810424 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:24.810391 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4583cec-l4kzb" event={"ID":"87101208-30f4-4ce8-b2b8-9023537f9aac","Type":"ContainerStarted","Data":"b3d5f4d530b8de93bd4df8a9c94cf508236911678be933e6a37f1ccb3777f0e7"} Nov 03 03:43:25.817321 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:25.817289 5605 generic.go:356] "Generic (PLEG): container finished" podID="87101208-30f4-4ce8-b2b8-9023537f9aac" containerID="3173a73f8c0ce4706cf042377fd3caf17af873d74a0447a7007010c53de01848" exitCode=0 Nov 03 03:43:25.817722 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:25.817414 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4583cec-l4kzb" event={"ID":"87101208-30f4-4ce8-b2b8-9023537f9aac","Type":"ContainerDied","Data":"3173a73f8c0ce4706cf042377fd3caf17af873d74a0447a7007010c53de01848"} Nov 03 03:43:27.006534 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:27.006495 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4583cec-l4kzb" Nov 03 03:43:27.054816 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:27.054786 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e7b8ba1-5zr5h"] Nov 03 03:43:27.068718 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:27.068689 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87101208-30f4-4ce8-b2b8-9023537f9aac-kube-api-access-6jwj8\" (UniqueName: \"kubernetes.io/projected/87101208-30f4-4ce8-b2b8-9023537f9aac-kube-api-access-6jwj8\") pod \"87101208-30f4-4ce8-b2b8-9023537f9aac\" (UID: \"87101208-30f4-4ce8-b2b8-9023537f9aac\") " Nov 03 03:43:27.068854 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:27.068787 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87101208-30f4-4ce8-b2b8-9023537f9aac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87101208-30f4-4ce8-b2b8-9023537f9aac-maas-api-openshift-ca\") pod \"87101208-30f4-4ce8-b2b8-9023537f9aac\" (UID: \"87101208-30f4-4ce8-b2b8-9023537f9aac\") " Nov 03 03:43:27.068854 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:27.068835 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87101208-30f4-4ce8-b2b8-9023537f9aac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87101208-30f4-4ce8-b2b8-9023537f9aac-maas-api-xks-ca\") pod \"87101208-30f4-4ce8-b2b8-9023537f9aac\" (UID: \"87101208-30f4-4ce8-b2b8-9023537f9aac\") " Nov 03 03:43:27.069256 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:27.069227 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87101208-30f4-4ce8-b2b8-9023537f9aac-maas-api-openshift-ca" pod "87101208-30f4-4ce8-b2b8-9023537f9aac" (UID: "87101208-30f4-4ce8-b2b8-9023537f9aac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:43:27.071418 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:27.071389 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87101208-30f4-4ce8-b2b8-9023537f9aac-kube-api-access-6jwj8" pod "87101208-30f4-4ce8-b2b8-9023537f9aac" (UID: "87101208-30f4-4ce8-b2b8-9023537f9aac"). InnerVolumeSpecName "kube-api-access-6jwj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:43:27.071616 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:27.071590 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87101208-30f4-4ce8-b2b8-9023537f9aac-maas-api-xks-ca" pod "87101208-30f4-4ce8-b2b8-9023537f9aac" (UID: "87101208-30f4-4ce8-b2b8-9023537f9aac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:43:27.101459 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:27.101427 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e7b8ba1-5zr5h"] Nov 03 03:43:27.169873 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:27.169839 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87101208-30f4-4ce8-b2b8-9023537f9aac-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:43:27.170044 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:27.169877 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jwj8\" (UniqueName: \"kubernetes.io/projected/87101208-30f4-4ce8-b2b8-9023537f9aac-kube-api-access-6jwj8\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:43:27.170044 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:27.169895 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87101208-30f4-4ce8-b2b8-9023537f9aac-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:43:27.844070 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:27.844031 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4583cec-l4kzb" event={"ID":"87101208-30f4-4ce8-b2b8-9023537f9aac","Type":"ContainerDied","Data":"b3d5f4d530b8de93bd4df8a9c94cf508236911678be933e6a37f1ccb3777f0e7"} Nov 03 03:43:27.844340 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:27.844322 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3d5f4d530b8de93bd4df8a9c94cf508236911678be933e6a37f1ccb3777f0e7" Nov 03 03:43:27.844680 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:27.844663 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4583cec-l4kzb" Nov 03 03:43:28.857150 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:28.857115 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eee62b00-65d0-42a7-8a77-2d1a1d8bf7c4" path="/var/lib/kubelet/pods/eee62b00-65d0-42a7-8a77-2d1a1d8bf7c4/volumes" Nov 03 03:43:34.836486 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:34.836445 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa4bbf4e-ct5rh"] Nov 03 03:43:34.836928 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:34.836828 5605 cpu_manager.go:414] "containerMap: removing container" podUID="87101208-30f4-4ce8-b2b8-9023537f9aac" containerName="revoke-keys" Nov 03 03:43:34.836928 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:34.836841 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87101208-30f4-4ce8-b2b8-9023537f9aac" containerName="revoke-keys" Nov 03 03:43:34.837005 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:34.836927 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="87101208-30f4-4ce8-b2b8-9023537f9aac" containerName="revoke-keys" Nov 03 03:43:34.841697 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:34.841669 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa4bbf4e-ct5rh" Nov 03 03:43:34.878139 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:34.878111 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7975dc6d-dffb-4af7-a245-45aa005ee595-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa4bbf4e-ct5rh\" (UID: \"7975dc6d-dffb-4af7-a245-45aa005ee595\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa4bbf4e-ct5rh" Nov 03 03:43:34.878330 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:34.878305 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7975dc6d-dffb-4af7-a245-45aa005ee595-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa4bbf4e-ct5rh\" (UID: \"7975dc6d-dffb-4af7-a245-45aa005ee595\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa4bbf4e-ct5rh" Nov 03 03:43:34.878563 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:34.878542 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcjlb\" (UniqueName: \"kubernetes.io/projected/7975dc6d-dffb-4af7-a245-45aa005ee595-kube-api-access-kcjlb\") pod \"maas-api-revoke-sub-aa4bbf4e-ct5rh\" (UID: \"7975dc6d-dffb-4af7-a245-45aa005ee595\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa4bbf4e-ct5rh" Nov 03 03:43:34.895150 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:34.895129 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:43:34.896498 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:34.896481 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:43:34.896565 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:34.896543 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:43:34.896607 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:34.896590 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sb4qh\"" Nov 03 03:43:34.938969 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:34.938943 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa4bbf4e-ct5rh"] Nov 03 03:43:34.979820 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:34.979800 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7975dc6d-dffb-4af7-a245-45aa005ee595-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa4bbf4e-ct5rh\" (UID: \"7975dc6d-dffb-4af7-a245-45aa005ee595\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa4bbf4e-ct5rh" Nov 03 03:43:34.979927 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:34.979833 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7975dc6d-dffb-4af7-a245-45aa005ee595-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa4bbf4e-ct5rh\" (UID: \"7975dc6d-dffb-4af7-a245-45aa005ee595\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa4bbf4e-ct5rh" Nov 03 03:43:34.979927 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:34.979906 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcjlb\" (UniqueName: \"kubernetes.io/projected/7975dc6d-dffb-4af7-a245-45aa005ee595-kube-api-access-kcjlb\") pod \"maas-api-revoke-sub-aa4bbf4e-ct5rh\" (UID: \"7975dc6d-dffb-4af7-a245-45aa005ee595\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa4bbf4e-ct5rh" Nov 03 03:43:34.980531 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:34.980513 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7975dc6d-dffb-4af7-a245-45aa005ee595-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa4bbf4e-ct5rh\" (UID: \"7975dc6d-dffb-4af7-a245-45aa005ee595\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa4bbf4e-ct5rh" Nov 03 03:43:34.982088 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:34.982072 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7975dc6d-dffb-4af7-a245-45aa005ee595-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa4bbf4e-ct5rh\" (UID: \"7975dc6d-dffb-4af7-a245-45aa005ee595\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa4bbf4e-ct5rh" Nov 03 03:43:35.035095 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:35.035065 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcjlb\" (UniqueName: \"kubernetes.io/projected/7975dc6d-dffb-4af7-a245-45aa005ee595-kube-api-access-kcjlb\") pod \"maas-api-revoke-sub-aa4bbf4e-ct5rh\" (UID: \"7975dc6d-dffb-4af7-a245-45aa005ee595\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa4bbf4e-ct5rh" Nov 03 03:43:35.156024 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:35.155971 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa4bbf4e-ct5rh" Nov 03 03:43:35.522211 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:35.522129 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa4bbf4e-ct5rh"] Nov 03 03:43:35.528442 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:43:35.528401 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7975dc6d_dffb_4af7_a245_45aa005ee595.slice/crio-a83bd24786beb2c93868f690efd6696655e043b47411419a66500edd75dc86f2 WatchSource:0}: Error finding container a83bd24786beb2c93868f690efd6696655e043b47411419a66500edd75dc86f2: Status 404 returned error can't find the container with id a83bd24786beb2c93868f690efd6696655e043b47411419a66500edd75dc86f2 Nov 03 03:43:35.890114 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:35.890082 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa4bbf4e-ct5rh" event={"ID":"7975dc6d-dffb-4af7-a245-45aa005ee595","Type":"ContainerStarted","Data":"e357491257054701698b19f0167582ae50554defcf689f1d347564f74a7f8b1d"} Nov 03 03:43:35.890459 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:35.890133 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa4bbf4e-ct5rh" event={"ID":"7975dc6d-dffb-4af7-a245-45aa005ee595","Type":"ContainerStarted","Data":"a83bd24786beb2c93868f690efd6696655e043b47411419a66500edd75dc86f2"} Nov 03 03:43:35.996822 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:35.996776 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa4bbf4e-ct5rh" podStartSLOduration=1.996764244 podStartE2EDuration="1.996764244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:43:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:43:35.996143035 +0000 UTC m=+6693.255341190" watchObservedRunningTime="2026-11-03 03:43:35.996764244 +0000 UTC m=+6693.255962398" Nov 03 03:43:36.897450 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:36.897415 5605 generic.go:356] "Generic (PLEG): container finished" podID="7975dc6d-dffb-4af7-a245-45aa005ee595" containerID="e357491257054701698b19f0167582ae50554defcf689f1d347564f74a7f8b1d" exitCode=0 Nov 03 03:43:36.897843 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:36.897481 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa4bbf4e-ct5rh" event={"ID":"7975dc6d-dffb-4af7-a245-45aa005ee595","Type":"ContainerDied","Data":"e357491257054701698b19f0167582ae50554defcf689f1d347564f74a7f8b1d"} Nov 03 03:43:38.360139 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:38.359864 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa4bbf4e-ct5rh" Nov 03 03:43:38.484110 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:38.481215 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7975dc6d-dffb-4af7-a245-45aa005ee595-kube-api-access-kcjlb\" (UniqueName: \"kubernetes.io/projected/7975dc6d-dffb-4af7-a245-45aa005ee595-kube-api-access-kcjlb\") pod \"7975dc6d-dffb-4af7-a245-45aa005ee595\" (UID: \"7975dc6d-dffb-4af7-a245-45aa005ee595\") " Nov 03 03:43:38.484110 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:38.481310 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7975dc6d-dffb-4af7-a245-45aa005ee595-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7975dc6d-dffb-4af7-a245-45aa005ee595-maas-api-openshift-ca\") pod \"7975dc6d-dffb-4af7-a245-45aa005ee595\" (UID: \"7975dc6d-dffb-4af7-a245-45aa005ee595\") " Nov 03 03:43:38.484110 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:38.481377 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7975dc6d-dffb-4af7-a245-45aa005ee595-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7975dc6d-dffb-4af7-a245-45aa005ee595-maas-api-xks-ca\") pod \"7975dc6d-dffb-4af7-a245-45aa005ee595\" (UID: \"7975dc6d-dffb-4af7-a245-45aa005ee595\") " Nov 03 03:43:38.637491 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:38.635670 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7975dc6d-dffb-4af7-a245-45aa005ee595-maas-api-openshift-ca" pod "7975dc6d-dffb-4af7-a245-45aa005ee595" (UID: "7975dc6d-dffb-4af7-a245-45aa005ee595"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:43:38.637491 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:38.635812 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7975dc6d-dffb-4af7-a245-45aa005ee595-maas-api-xks-ca" pod "7975dc6d-dffb-4af7-a245-45aa005ee595" (UID: "7975dc6d-dffb-4af7-a245-45aa005ee595"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:43:38.644545 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:38.644505 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7975dc6d-dffb-4af7-a245-45aa005ee595-kube-api-access-kcjlb" pod "7975dc6d-dffb-4af7-a245-45aa005ee595" (UID: "7975dc6d-dffb-4af7-a245-45aa005ee595"). InnerVolumeSpecName "kube-api-access-kcjlb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:43:38.688019 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:38.687990 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7975dc6d-dffb-4af7-a245-45aa005ee595-maas-api-openshift-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:43:38.688174 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:38.688158 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7975dc6d-dffb-4af7-a245-45aa005ee595-maas-api-xks-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:43:38.688301 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:38.688286 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcjlb\" (UniqueName: \"kubernetes.io/projected/7975dc6d-dffb-4af7-a245-45aa005ee595-kube-api-access-kcjlb\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:43:38.939390 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:38.939150 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa4bbf4e-ct5rh" event={"ID":"7975dc6d-dffb-4af7-a245-45aa005ee595","Type":"ContainerDied","Data":"a83bd24786beb2c93868f690efd6696655e043b47411419a66500edd75dc86f2"} Nov 03 03:43:38.939390 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:38.939212 5605 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a83bd24786beb2c93868f690efd6696655e043b47411419a66500edd75dc86f2" Nov 03 03:43:38.939390 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:38.939281 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa4bbf4e-ct5rh" Nov 03 03:43:47.529025 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:43:47.528981 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-skgg9"] Nov 03 03:43:53.912875 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:43:53.912837 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:44:01.043605 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:01.043565 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fc94408-m8qwl"] Nov 03 03:44:01.090174 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:01.090141 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fc94408-m8qwl"] Nov 03 03:44:02.848853 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:02.848818 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fcbfb33-bfd4-401c-9054-bc3abd0ea4e3" path="/var/lib/kubelet/pods/0fcbfb33-bfd4-401c-9054-bc3abd0ea4e3/volumes" Nov 03 03:44:17.630901 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:17.630871 5605 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-rbsfv_fc905c8d-4bfa-49ee-b0b7-8c434727e284/manager/0.log" Nov 03 03:44:18.024659 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:18.024588 5605 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-594d9845fc-7m8kb_949a8686-ae14-4c12-9da2-6bb05250692b/manager/0.log" Nov 03 03:44:20.764805 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:20.764770 5605 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-1c12e58c-dc6j8_3f894000-0dfd-4326-9b28-d3ca8943e17a/revoke-keys/0.log" Nov 03 03:44:20.985233 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:20.985205 5605 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-20d6150f-4ztgk_f270da8f-bf1a-46b7-8d66-3cba86c1f267/revoke-keys/0.log" Nov 03 03:44:21.206099 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:21.206060 5605 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-2ff431bf-xxlhd_bcb23d87-df37-4ad7-bfc7-e376c1e420b1/revoke-keys/0.log" Nov 03 03:44:21.429235 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:21.429203 5605 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-442c374b-lxwqb_6320e544-6957-4312-9a62-6c1338837e2a/revoke-keys/0.log" Nov 03 03:44:21.641233 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:21.641200 5605 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-502f081c-2pj8b_03f5364e-f3b5-4591-818d-44a17155e6de/revoke-keys/0.log" Nov 03 03:44:21.860141 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:21.860112 5605 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-61635b25-cjf5t_94ac84ed-63ec-4ce5-ad6c-0e40b5f89c86/revoke-keys/0.log" Nov 03 03:44:22.086287 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:22.086255 5605 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-7451fda3-q7dx9_ca706570-f5be-404a-8a82-25c8466d74c4/revoke-keys/0.log" Nov 03 03:44:22.307787 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:22.307755 5605 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-7b8c07f5-r87q5_6dfa0951-bb9c-4d22-adf2-0fee83200c2e/revoke-keys/0.log" Nov 03 03:44:22.525868 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:22.525782 5605 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-8fb407af-ldbpf_62e6ef37-20e0-4e7e-87af-b87799101bf8/revoke-keys/0.log" Nov 03 03:44:22.748132 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:22.748101 5605 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-986b3c70-tnprm_21584a67-219d-4646-86b7-90719a04de2b/revoke-keys/0.log" Nov 03 03:44:22.974825 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:22.974794 5605 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-a4583cec-l4kzb_87101208-30f4-4ce8-b2b8-9023537f9aac/revoke-keys/0.log" Nov 03 03:44:23.194396 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:23.194370 5605 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-aa4bbf4e-ct5rh_7975dc6d-dffb-4af7-a245-45aa005ee595/revoke-keys/0.log" Nov 03 03:44:23.410669 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:23.410643 5605 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-b18aaf4c-mgbvb_e6261b7a-3a8d-4da1-90ab-775aa83aa9ad/revoke-keys/0.log" Nov 03 03:44:23.629779 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:23.629744 5605 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-c81be95d-2phsq_53ed8432-b78a-4b9b-ad15-56ff5772752b/revoke-keys/0.log" Nov 03 03:44:23.859076 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:23.859044 5605 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-dfb5a0e2-hm9wv_c0fa70c6-7511-4f78-9775-704c55afca0b/revoke-keys/0.log" Nov 03 03:44:23.912175 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:44:23.912134 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:44:24.077582 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:24.077549 5605 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-e6644eaa-nrq72_4819c8b3-339a-4a9b-9c54-81cda7e94236/revoke-keys/0.log" Nov 03 03:44:27.845829 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:27.845799 5605 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-skgg9_ce015707-8aae-4215-bb4b-6ebb7bef21c5/limitador/0.log" Nov 03 03:44:29.911923 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:29.911887 5605 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6dc8f585b4-sb2dx_8e0e5200-c6f3-4e09-aa80-dfd996cf38d1/kube-auth-proxy/0.log" Nov 03 03:44:30.411373 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:30.411336 5605 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-bpbq7_563c5b47-58ea-4fb8-b9f1-332cf8dae641/istio-proxy/0.log" Nov 03 03:44:30.999491 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:30.999452 5605 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-dbd7q_dc490604-2e00-49dd-8423-47fc70709678/istio-proxy/0.log" Nov 03 03:44:31.593707 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:31.593672 5605 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-k449x_0378fcbc-4027-4275-b274-2c188f3ff442/istio-proxy/0.log" Nov 03 03:44:31.908871 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:31.908791 5605 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-p8fqq_0bedb6e8-5829-4cc2-a912-35131c941e09/istio-proxy/0.log" Nov 03 03:44:33.349661 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:33.349627 5605 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-659c975df4-4wn4f_3344d6d8-48b8-4aa5-afba-b7e9b504b5ee/payload-pre-processing/0.log" Nov 03 03:44:43.042954 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:43.042858 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20d6150f-4ztgk"] Nov 03 03:44:43.089781 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:43.089741 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20d6150f-4ztgk"] Nov 03 03:44:44.849199 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:44.849163 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f270da8f-bf1a-46b7-8d66-3cba86c1f267" path="/var/lib/kubelet/pods/f270da8f-bf1a-46b7-8d66-3cba86c1f267/volumes" Nov 03 03:44:47.624584 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:47.624545 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bjrfm/must-gather-rxbbk"] Nov 03 03:44:47.624965 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:47.624906 5605 cpu_manager.go:414] "containerMap: removing container" podUID="7975dc6d-dffb-4af7-a245-45aa005ee595" containerName="revoke-keys" Nov 03 03:44:47.624965 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:47.624917 5605 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7975dc6d-dffb-4af7-a245-45aa005ee595" containerName="revoke-keys" Nov 03 03:44:47.625048 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:47.624978 5605 memory_manager.go:365] "RemoveStaleState removing state" podUID="7975dc6d-dffb-4af7-a245-45aa005ee595" containerName="revoke-keys" Nov 03 03:44:47.628436 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:47.628409 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bjrfm/must-gather-rxbbk" Nov 03 03:44:47.677094 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:47.677058 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bjrfm\"/\"openshift-service-ca.crt\"" Nov 03 03:44:47.706910 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:47.706877 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4c347109-712f-428d-b430-8defce4f75f7-must-gather-output\") pod \"must-gather-rxbbk\" (UID: \"4c347109-712f-428d-b430-8defce4f75f7\") " pod="openshift-must-gather-bjrfm/must-gather-rxbbk" Nov 03 03:44:47.707112 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:47.706931 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs7bb\" (UniqueName: \"kubernetes.io/projected/4c347109-712f-428d-b430-8defce4f75f7-kube-api-access-bs7bb\") pod \"must-gather-rxbbk\" (UID: \"4c347109-712f-428d-b430-8defce4f75f7\") " pod="openshift-must-gather-bjrfm/must-gather-rxbbk" Nov 03 03:44:47.723870 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:47.723839 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bjrfm\"/\"kube-root-ca.crt\"" Nov 03 03:44:47.723870 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:47.723849 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bjrfm\"/\"default-dockercfg-qw7f8\"" Nov 03 03:44:47.725796 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:47.725776 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bjrfm/must-gather-rxbbk"] Nov 03 03:44:47.732189 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:47.732170 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-l75xk/must-gather-ftj7t"] Nov 03 03:44:47.736460 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:47.736443 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-l75xk/must-gather-ftj7t" Nov 03 03:44:47.785477 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:47.785448 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-l75xk\"/\"default-dockercfg-hq8x6\"" Nov 03 03:44:47.785750 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:47.785734 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-l75xk\"/\"kube-root-ca.crt\"" Nov 03 03:44:47.785801 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:47.785775 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-l75xk\"/\"openshift-service-ca.crt\"" Nov 03 03:44:47.793181 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:47.793147 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-l75xk/must-gather-ftj7t"] Nov 03 03:44:47.807700 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:47.807665 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1474911a-b1b8-47de-9a4c-24920a605022-must-gather-output\") pod \"must-gather-ftj7t\" (UID: \"1474911a-b1b8-47de-9a4c-24920a605022\") " pod="openshift-must-gather-l75xk/must-gather-ftj7t" Nov 03 03:44:47.807834 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:47.807721 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8ctg\" (UniqueName: \"kubernetes.io/projected/1474911a-b1b8-47de-9a4c-24920a605022-kube-api-access-c8ctg\") pod \"must-gather-ftj7t\" (UID: \"1474911a-b1b8-47de-9a4c-24920a605022\") " pod="openshift-must-gather-l75xk/must-gather-ftj7t" Nov 03 03:44:47.807834 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:47.807808 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4c347109-712f-428d-b430-8defce4f75f7-must-gather-output\") pod \"must-gather-rxbbk\" (UID: \"4c347109-712f-428d-b430-8defce4f75f7\") " pod="openshift-must-gather-bjrfm/must-gather-rxbbk" Nov 03 03:44:47.807919 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:47.807854 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs7bb\" (UniqueName: \"kubernetes.io/projected/4c347109-712f-428d-b430-8defce4f75f7-kube-api-access-bs7bb\") pod \"must-gather-rxbbk\" (UID: \"4c347109-712f-428d-b430-8defce4f75f7\") " pod="openshift-must-gather-bjrfm/must-gather-rxbbk" Nov 03 03:44:47.808144 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:47.808128 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4c347109-712f-428d-b430-8defce4f75f7-must-gather-output\") pod \"must-gather-rxbbk\" (UID: \"4c347109-712f-428d-b430-8defce4f75f7\") " pod="openshift-must-gather-bjrfm/must-gather-rxbbk" Nov 03 03:44:47.861386 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:47.861341 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs7bb\" (UniqueName: \"kubernetes.io/projected/4c347109-712f-428d-b430-8defce4f75f7-kube-api-access-bs7bb\") pod \"must-gather-rxbbk\" (UID: \"4c347109-712f-428d-b430-8defce4f75f7\") " pod="openshift-must-gather-bjrfm/must-gather-rxbbk" Nov 03 03:44:47.908400 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:47.908297 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1474911a-b1b8-47de-9a4c-24920a605022-must-gather-output\") pod \"must-gather-ftj7t\" (UID: \"1474911a-b1b8-47de-9a4c-24920a605022\") " pod="openshift-must-gather-l75xk/must-gather-ftj7t" Nov 03 03:44:47.908400 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:47.908333 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8ctg\" (UniqueName: \"kubernetes.io/projected/1474911a-b1b8-47de-9a4c-24920a605022-kube-api-access-c8ctg\") pod \"must-gather-ftj7t\" (UID: \"1474911a-b1b8-47de-9a4c-24920a605022\") " pod="openshift-must-gather-l75xk/must-gather-ftj7t" Nov 03 03:44:47.908692 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:47.908671 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1474911a-b1b8-47de-9a4c-24920a605022-must-gather-output\") pod \"must-gather-ftj7t\" (UID: \"1474911a-b1b8-47de-9a4c-24920a605022\") " pod="openshift-must-gather-l75xk/must-gather-ftj7t" Nov 03 03:44:47.942145 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:47.942114 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bjrfm/must-gather-rxbbk" Nov 03 03:44:47.961646 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:47.961617 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8ctg\" (UniqueName: \"kubernetes.io/projected/1474911a-b1b8-47de-9a4c-24920a605022-kube-api-access-c8ctg\") pod \"must-gather-ftj7t\" (UID: \"1474911a-b1b8-47de-9a4c-24920a605022\") " pod="openshift-must-gather-l75xk/must-gather-ftj7t" Nov 03 03:44:48.051298 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:48.051260 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-l75xk/must-gather-ftj7t" Nov 03 03:44:48.312275 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:48.312245 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bjrfm/must-gather-rxbbk"] Nov 03 03:44:48.318226 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:44:48.318187 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c347109_712f_428d_b430_8defce4f75f7.slice/crio-ee6209b85e025419c6bd7bead3d6b2649942676eb72e022dc593b5f18548dcd1 WatchSource:0}: Error finding container ee6209b85e025419c6bd7bead3d6b2649942676eb72e022dc593b5f18548dcd1: Status 404 returned error can't find the container with id ee6209b85e025419c6bd7bead3d6b2649942676eb72e022dc593b5f18548dcd1 Nov 03 03:44:48.319905 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:48.319887 5605 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 03:44:48.423331 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:48.423293 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-l75xk/must-gather-ftj7t"] Nov 03 03:44:48.429374 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:44:48.429334 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1474911a_b1b8_47de_9a4c_24920a605022.slice/crio-0ee700c373e1e05d03ecf6d5ce411a6c88fb4a2435d3b82e3ed059732544a54c WatchSource:0}: Error finding container 0ee700c373e1e05d03ecf6d5ce411a6c88fb4a2435d3b82e3ed059732544a54c: Status 404 returned error can't find the container with id 0ee700c373e1e05d03ecf6d5ce411a6c88fb4a2435d3b82e3ed059732544a54c Nov 03 03:44:49.306350 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:49.306309 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l75xk/must-gather-ftj7t" event={"ID":"1474911a-b1b8-47de-9a4c-24920a605022","Type":"ContainerStarted","Data":"0ee700c373e1e05d03ecf6d5ce411a6c88fb4a2435d3b82e3ed059732544a54c"} Nov 03 03:44:49.307640 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:49.307610 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bjrfm/must-gather-rxbbk" event={"ID":"4c347109-712f-428d-b430-8defce4f75f7","Type":"ContainerStarted","Data":"ee6209b85e025419c6bd7bead3d6b2649942676eb72e022dc593b5f18548dcd1"} Nov 03 03:44:50.316105 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:50.316066 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l75xk/must-gather-ftj7t" event={"ID":"1474911a-b1b8-47de-9a4c-24920a605022","Type":"ContainerStarted","Data":"90ca17b169cf999ad14de6a1c005a801bbadf618097472739f55d73d3f0d816a"} Nov 03 03:44:50.316105 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:50.316112 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l75xk/must-gather-ftj7t" event={"ID":"1474911a-b1b8-47de-9a4c-24920a605022","Type":"ContainerStarted","Data":"450111a809216925191d761270dfe4ca4fcbe2f74b3904142e104544c0eed931"} Nov 03 03:44:50.423346 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:50.423287 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-l75xk/must-gather-ftj7t" podStartSLOduration=2.078041697 podStartE2EDuration="3.423272049s" totalImagesPullingTime="1.345230352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:44:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:44:50.422288352 +0000 UTC m=+6767.681486519" watchObservedRunningTime="2026-11-03 03:44:50.423272049 +0000 UTC m=+6767.682470202" Nov 03 03:44:53.928152 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:44:53.928115 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:44:54.044951 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:54.044911 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfb5a0e2-hm9wv"] Nov 03 03:44:54.090905 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:54.090836 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfb5a0e2-hm9wv"] Nov 03 03:44:54.854919 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:54.854877 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0fa70c6-7511-4f78-9775-704c55afca0b" path="/var/lib/kubelet/pods/c0fa70c6-7511-4f78-9775-704c55afca0b/volumes" Nov 03 03:44:56.679329 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:56.679288 5605 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-kkngd_28f00eb6-78c2-4d93-bde5-c1ea12601dbd/global-pull-secret-syncer/0.log" Nov 03 03:44:59.462282 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:59.461344 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bjrfm/must-gather-rxbbk" event={"ID":"4c347109-712f-428d-b430-8defce4f75f7","Type":"ContainerStarted","Data":"8247c3c00ebfdef0d55f137834991f3007e49809c03fb078f50372c85a1e5b66"} Nov 03 03:44:59.462282 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:59.461425 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bjrfm/must-gather-rxbbk" event={"ID":"4c347109-712f-428d-b430-8defce4f75f7","Type":"ContainerStarted","Data":"501309ba2b162ef5f82adff12085ef1fdf564dbeea7c8b25533c6afaa6c9e614"} Nov 03 03:44:59.579387 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:44:59.577409 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-bjrfm/must-gather-rxbbk" podStartSLOduration=2.8883824860000002 podStartE2EDuration="12.577388071s" totalImagesPullingTime="9.689005585s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:44:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:44:59.576716901 +0000 UTC m=+6776.835915056" watchObservedRunningTime="2026-11-03 03:44:59.577388071 +0000 UTC m=+6776.836586225" Nov 03 03:45:00.163683 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:00.163645 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894625-qnwt9"] Nov 03 03:45:00.178387 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:00.170622 5605 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-f524q_7d2438ed-bfc2-4639-b87d-84689a82123e/konnectivity-agent/0.log" Nov 03 03:45:00.278377 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:00.276230 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894625-qnwt9"] Nov 03 03:45:00.278377 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:00.276334 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894625-qnwt9" Nov 03 03:45:00.323684 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:00.323352 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4fceab0-d8a1-4df8-be90-bcc21dda6afd-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894625-qnwt9\" (UID: \"e4fceab0-d8a1-4df8-be90-bcc21dda6afd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894625-qnwt9" Nov 03 03:45:00.323684 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:00.323562 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp8q4\" (UniqueName: \"kubernetes.io/projected/e4fceab0-d8a1-4df8-be90-bcc21dda6afd-kube-api-access-gp8q4\") pod \"maas-api-key-cleanup-29894625-qnwt9\" (UID: \"e4fceab0-d8a1-4df8-be90-bcc21dda6afd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894625-qnwt9" Nov 03 03:45:00.327728 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:00.327678 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:45:00.377682 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:00.377646 5605 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:45:00.378067 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:00.378047 5605 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sb4qh\"" Nov 03 03:45:00.424611 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:00.424458 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4fceab0-d8a1-4df8-be90-bcc21dda6afd-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894625-qnwt9\" (UID: \"e4fceab0-d8a1-4df8-be90-bcc21dda6afd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894625-qnwt9" Nov 03 03:45:00.425252 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:00.425226 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4fceab0-d8a1-4df8-be90-bcc21dda6afd-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894625-qnwt9\" (UID: \"e4fceab0-d8a1-4df8-be90-bcc21dda6afd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894625-qnwt9" Nov 03 03:45:00.425757 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:00.425492 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp8q4\" (UniqueName: \"kubernetes.io/projected/e4fceab0-d8a1-4df8-be90-bcc21dda6afd-kube-api-access-gp8q4\") pod \"maas-api-key-cleanup-29894625-qnwt9\" (UID: \"e4fceab0-d8a1-4df8-be90-bcc21dda6afd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894625-qnwt9" Nov 03 03:45:00.482074 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:00.482024 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp8q4\" (UniqueName: \"kubernetes.io/projected/e4fceab0-d8a1-4df8-be90-bcc21dda6afd-kube-api-access-gp8q4\") pod \"maas-api-key-cleanup-29894625-qnwt9\" (UID: \"e4fceab0-d8a1-4df8-be90-bcc21dda6afd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894625-qnwt9" Nov 03 03:45:00.683075 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:00.682973 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894625-qnwt9" Nov 03 03:45:01.247404 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:01.247347 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894625-qnwt9"] Nov 03 03:45:01.260470 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:45:01.260430 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4fceab0_d8a1_4df8_be90_bcc21dda6afd.slice/crio-65985b083ac38833396698bfb0b4536e875dfcd916fa54ed754d65e6b200efd0 WatchSource:0}: Error finding container 65985b083ac38833396698bfb0b4536e875dfcd916fa54ed754d65e6b200efd0: Status 404 returned error can't find the container with id 65985b083ac38833396698bfb0b4536e875dfcd916fa54ed754d65e6b200efd0 Nov 03 03:45:01.482547 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:01.482503 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894625-qnwt9" event={"ID":"e4fceab0-d8a1-4df8-be90-bcc21dda6afd","Type":"ContainerStarted","Data":"65985b083ac38833396698bfb0b4536e875dfcd916fa54ed754d65e6b200efd0"} Nov 03 03:45:01.913072 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:01.912994 5605 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-140.us-west-2.compute.internal_d1cefe6117be002b0e139e82f22d4917/haproxy/0.log" Nov 03 03:45:02.495989 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:02.494451 5605 generic.go:356] "Generic (PLEG): container finished" podID="e4fceab0-d8a1-4df8-be90-bcc21dda6afd" containerID="5a9e0107b4d4625572f80fd29aee42e8fd712bac60b431f18f44999d2c769d9b" exitCode=60 Nov 03 03:45:02.495989 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:02.494613 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894625-qnwt9" event={"ID":"e4fceab0-d8a1-4df8-be90-bcc21dda6afd","Type":"ContainerDied","Data":"5a9e0107b4d4625572f80fd29aee42e8fd712bac60b431f18f44999d2c769d9b"} Nov 03 03:45:02.495989 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:02.495001 5605 scope.go:122] "RemoveContainer" containerID="5a9e0107b4d4625572f80fd29aee42e8fd712bac60b431f18f44999d2c769d9b" Nov 03 03:45:03.515473 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:03.515442 5605 generic.go:356] "Generic (PLEG): container finished" podID="e4fceab0-d8a1-4df8-be90-bcc21dda6afd" containerID="aa9c1636b0df150fe2c2fd3d90c15ae1ca82ad3ac3f749786b1f9d643068b95d" exitCode=60 Nov 03 03:45:03.516213 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:03.516161 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894625-qnwt9" event={"ID":"e4fceab0-d8a1-4df8-be90-bcc21dda6afd","Type":"ContainerDied","Data":"aa9c1636b0df150fe2c2fd3d90c15ae1ca82ad3ac3f749786b1f9d643068b95d"} Nov 03 03:45:03.516386 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:03.516241 5605 scope.go:122] "RemoveContainer" containerID="5a9e0107b4d4625572f80fd29aee42e8fd712bac60b431f18f44999d2c769d9b" Nov 03 03:45:03.516492 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:03.516471 5605 scope.go:122] "RemoveContainer" containerID="aa9c1636b0df150fe2c2fd3d90c15ae1ca82ad3ac3f749786b1f9d643068b95d" Nov 03 03:45:03.516759 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:45:03.516733 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894625-qnwt9_odh-ai-gateway-infra(e4fceab0-d8a1-4df8-be90-bcc21dda6afd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894625-qnwt9" podUID="e4fceab0-d8a1-4df8-be90-bcc21dda6afd" Nov 03 03:45:04.545165 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:04.542828 5605 scope.go:122] "RemoveContainer" containerID="aa9c1636b0df150fe2c2fd3d90c15ae1ca82ad3ac3f749786b1f9d643068b95d" Nov 03 03:45:04.545165 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:45:04.543140 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894625-qnwt9_odh-ai-gateway-infra(e4fceab0-d8a1-4df8-be90-bcc21dda6afd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894625-qnwt9" podUID="e4fceab0-d8a1-4df8-be90-bcc21dda6afd" Nov 03 03:45:05.541378 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:05.541020 5605 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-rbsfv_fc905c8d-4bfa-49ee-b0b7-8c434727e284/manager/0.log" Nov 03 03:45:05.807733 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:05.807691 5605 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-594d9845fc-7m8kb_949a8686-ae14-4c12-9da2-6bb05250692b/manager/0.log" Nov 03 03:45:06.165206 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:06.165095 5605 scope.go:122] "RemoveContainer" containerID="f0c7cf2509a22694cd7adcabcd8fcfc43e0db6a2be73f6a591f17273d0d002f8" Nov 03 03:45:06.176530 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:06.175199 5605 scope.go:122] "RemoveContainer" containerID="9cf3bedf69f7babb9c69647a6ee231d99804f5858f52dd5e3d1d37b49b76d13b" Nov 03 03:45:06.187755 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:06.186524 5605 scope.go:122] "RemoveContainer" containerID="d0e94d49f7b3db5855d02258681a3dc63277f5552ded67c5fa6e387588e6d379" Nov 03 03:45:07.734454 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:07.734422 5605 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-l75xk/perf-node-gather-daemonset-4q7vv"] Nov 03 03:45:07.742427 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:07.742394 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-l75xk/perf-node-gather-daemonset-4q7vv" Nov 03 03:45:07.833138 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:07.833094 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-l75xk/perf-node-gather-daemonset-4q7vv"] Nov 03 03:45:07.865070 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:07.865039 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bb5c4843-7442-4c84-8484-51e87b0638f6-podres\") pod \"perf-node-gather-daemonset-4q7vv\" (UID: \"bb5c4843-7442-4c84-8484-51e87b0638f6\") " pod="openshift-must-gather-l75xk/perf-node-gather-daemonset-4q7vv" Nov 03 03:45:07.865266 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:07.865086 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bb5c4843-7442-4c84-8484-51e87b0638f6-sys\") pod \"perf-node-gather-daemonset-4q7vv\" (UID: \"bb5c4843-7442-4c84-8484-51e87b0638f6\") " pod="openshift-must-gather-l75xk/perf-node-gather-daemonset-4q7vv" Nov 03 03:45:07.865266 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:07.865154 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bb5c4843-7442-4c84-8484-51e87b0638f6-proc\") pod \"perf-node-gather-daemonset-4q7vv\" (UID: \"bb5c4843-7442-4c84-8484-51e87b0638f6\") " pod="openshift-must-gather-l75xk/perf-node-gather-daemonset-4q7vv" Nov 03 03:45:07.865266 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:07.865198 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwwx2\" (UniqueName: \"kubernetes.io/projected/bb5c4843-7442-4c84-8484-51e87b0638f6-kube-api-access-qwwx2\") pod \"perf-node-gather-daemonset-4q7vv\" (UID: \"bb5c4843-7442-4c84-8484-51e87b0638f6\") " pod="openshift-must-gather-l75xk/perf-node-gather-daemonset-4q7vv" Nov 03 03:45:07.865266 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:07.865261 5605 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bb5c4843-7442-4c84-8484-51e87b0638f6-lib-modules\") pod \"perf-node-gather-daemonset-4q7vv\" (UID: \"bb5c4843-7442-4c84-8484-51e87b0638f6\") " pod="openshift-must-gather-l75xk/perf-node-gather-daemonset-4q7vv" Nov 03 03:45:07.966026 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:07.965989 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bb5c4843-7442-4c84-8484-51e87b0638f6-proc\") pod \"perf-node-gather-daemonset-4q7vv\" (UID: \"bb5c4843-7442-4c84-8484-51e87b0638f6\") " pod="openshift-must-gather-l75xk/perf-node-gather-daemonset-4q7vv" Nov 03 03:45:07.966202 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:07.966050 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwwx2\" (UniqueName: \"kubernetes.io/projected/bb5c4843-7442-4c84-8484-51e87b0638f6-kube-api-access-qwwx2\") pod \"perf-node-gather-daemonset-4q7vv\" (UID: \"bb5c4843-7442-4c84-8484-51e87b0638f6\") " pod="openshift-must-gather-l75xk/perf-node-gather-daemonset-4q7vv" Nov 03 03:45:07.966202 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:07.966148 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bb5c4843-7442-4c84-8484-51e87b0638f6-proc\") pod \"perf-node-gather-daemonset-4q7vv\" (UID: \"bb5c4843-7442-4c84-8484-51e87b0638f6\") " pod="openshift-must-gather-l75xk/perf-node-gather-daemonset-4q7vv" Nov 03 03:45:07.966202 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:07.966165 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bb5c4843-7442-4c84-8484-51e87b0638f6-lib-modules\") pod \"perf-node-gather-daemonset-4q7vv\" (UID: \"bb5c4843-7442-4c84-8484-51e87b0638f6\") " pod="openshift-must-gather-l75xk/perf-node-gather-daemonset-4q7vv" Nov 03 03:45:07.966368 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:07.966246 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bb5c4843-7442-4c84-8484-51e87b0638f6-lib-modules\") pod \"perf-node-gather-daemonset-4q7vv\" (UID: \"bb5c4843-7442-4c84-8484-51e87b0638f6\") " pod="openshift-must-gather-l75xk/perf-node-gather-daemonset-4q7vv" Nov 03 03:45:07.966368 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:07.966279 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bb5c4843-7442-4c84-8484-51e87b0638f6-podres\") pod \"perf-node-gather-daemonset-4q7vv\" (UID: \"bb5c4843-7442-4c84-8484-51e87b0638f6\") " pod="openshift-must-gather-l75xk/perf-node-gather-daemonset-4q7vv" Nov 03 03:45:07.966368 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:07.966339 5605 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bb5c4843-7442-4c84-8484-51e87b0638f6-sys\") pod \"perf-node-gather-daemonset-4q7vv\" (UID: \"bb5c4843-7442-4c84-8484-51e87b0638f6\") " pod="openshift-must-gather-l75xk/perf-node-gather-daemonset-4q7vv" Nov 03 03:45:07.966526 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:07.966397 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bb5c4843-7442-4c84-8484-51e87b0638f6-podres\") pod \"perf-node-gather-daemonset-4q7vv\" (UID: \"bb5c4843-7442-4c84-8484-51e87b0638f6\") " pod="openshift-must-gather-l75xk/perf-node-gather-daemonset-4q7vv" Nov 03 03:45:07.966526 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:07.966469 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bb5c4843-7442-4c84-8484-51e87b0638f6-sys\") pod \"perf-node-gather-daemonset-4q7vv\" (UID: \"bb5c4843-7442-4c84-8484-51e87b0638f6\") " pod="openshift-must-gather-l75xk/perf-node-gather-daemonset-4q7vv" Nov 03 03:45:08.020194 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:08.020108 5605 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwwx2\" (UniqueName: \"kubernetes.io/projected/bb5c4843-7442-4c84-8484-51e87b0638f6-kube-api-access-qwwx2\") pod \"perf-node-gather-daemonset-4q7vv\" (UID: \"bb5c4843-7442-4c84-8484-51e87b0638f6\") " pod="openshift-must-gather-l75xk/perf-node-gather-daemonset-4q7vv" Nov 03 03:45:08.069709 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:08.069677 5605 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-l75xk/perf-node-gather-daemonset-4q7vv" Nov 03 03:45:08.457736 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:08.457703 5605 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-l75xk/perf-node-gather-daemonset-4q7vv"] Nov 03 03:45:08.463733 ip-10-0-140-140 kubenswrapper[5605]: W1103 03:45:08.463701 5605 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbb5c4843_7442_4c84_8484_51e87b0638f6.slice/crio-8e902980241f2eb645516bb4ced5300a97d227687884f9cf25fda208730eaf7a WatchSource:0}: Error finding container 8e902980241f2eb645516bb4ced5300a97d227687884f9cf25fda208730eaf7a: Status 404 returned error can't find the container with id 8e902980241f2eb645516bb4ced5300a97d227687884f9cf25fda208730eaf7a Nov 03 03:45:08.578913 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:08.578875 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l75xk/perf-node-gather-daemonset-4q7vv" event={"ID":"bb5c4843-7442-4c84-8484-51e87b0638f6","Type":"ContainerStarted","Data":"3e82b4a7a9f5ddbced36e077ca04d8f969cfe85554275603ea4d1b450d7e773a"} Nov 03 03:45:08.579037 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:08.578933 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l75xk/perf-node-gather-daemonset-4q7vv" event={"ID":"bb5c4843-7442-4c84-8484-51e87b0638f6","Type":"ContainerStarted","Data":"8e902980241f2eb645516bb4ced5300a97d227687884f9cf25fda208730eaf7a"} Nov 03 03:45:08.579037 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:08.579017 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-l75xk/perf-node-gather-daemonset-4q7vv" Nov 03 03:45:08.685240 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:08.685176 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-l75xk/perf-node-gather-daemonset-4q7vv" podStartSLOduration=1.6851590920000001 podStartE2EDuration="1.685159092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 03:45:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:45:08.685016537 +0000 UTC m=+6785.944214692" watchObservedRunningTime="2026-11-03 03:45:08.685159092 +0000 UTC m=+6785.944357248" Nov 03 03:45:09.614324 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:09.614289 5605 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-skgg9_ce015707-8aae-4215-bb4b-6ebb7bef21c5/limitador/0.log" Nov 03 03:45:11.621028 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:11.620973 5605 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-rbsfv_fc905c8d-4bfa-49ee-b0b7-8c434727e284/manager/0.log" Nov 03 03:45:11.883432 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:11.883252 5605 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-594d9845fc-7m8kb_949a8686-ae14-4c12-9da2-6bb05250692b/manager/0.log" Nov 03 03:45:12.766580 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:12.766545 5605 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_83c2e7de-c853-4518-a01a-80e3a29107ed/alertmanager/0.log" Nov 03 03:45:13.174732 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:13.174700 5605 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_83c2e7de-c853-4518-a01a-80e3a29107ed/config-reloader/0.log" Nov 03 03:45:13.579194 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:13.579162 5605 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_83c2e7de-c853-4518-a01a-80e3a29107ed/kube-rbac-proxy-web/0.log" Nov 03 03:45:13.985597 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:13.985511 5605 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_83c2e7de-c853-4518-a01a-80e3a29107ed/kube-rbac-proxy/0.log" Nov 03 03:45:14.394168 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:14.394126 5605 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_83c2e7de-c853-4518-a01a-80e3a29107ed/kube-rbac-proxy-metric/0.log" Nov 03 03:45:14.596581 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:14.596547 5605 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-l75xk/perf-node-gather-daemonset-4q7vv" Nov 03 03:45:14.801904 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:14.801875 5605 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_83c2e7de-c853-4518-a01a-80e3a29107ed/prom-label-proxy/0.log" Nov 03 03:45:14.843594 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:14.843566 5605 scope.go:122] "RemoveContainer" containerID="aa9c1636b0df150fe2c2fd3d90c15ae1ca82ad3ac3f749786b1f9d643068b95d" Nov 03 03:45:15.207808 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:15.207769 5605 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_83c2e7de-c853-4518-a01a-80e3a29107ed/init-config-reloader/0.log" Nov 03 03:45:15.622730 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:15.622688 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894625-qnwt9" event={"ID":"e4fceab0-d8a1-4df8-be90-bcc21dda6afd","Type":"ContainerStarted","Data":"34e7579713cec41aaca36de061e4c6a161463e7dbccd21147feddf208ec8edda"} Nov 03 03:45:15.729573 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:15.729516 5605 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894625-qnwt9" podStartSLOduration=15.729501083 podStartE2EDuration="15.729501083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:45:15.72873025 +0000 UTC m=+6792.987928404" watchObservedRunningTime="2026-11-03 03:45:15.729501083 +0000 UTC m=+6792.988699237" Nov 03 03:45:16.630931 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:16.630899 5605 generic.go:356] "Generic (PLEG): container finished" podID="e4fceab0-d8a1-4df8-be90-bcc21dda6afd" containerID="34e7579713cec41aaca36de061e4c6a161463e7dbccd21147feddf208ec8edda" exitCode=60 Nov 03 03:45:16.631472 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:16.630959 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894625-qnwt9" event={"ID":"e4fceab0-d8a1-4df8-be90-bcc21dda6afd","Type":"ContainerDied","Data":"34e7579713cec41aaca36de061e4c6a161463e7dbccd21147feddf208ec8edda"} Nov 03 03:45:16.631472 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:16.631026 5605 scope.go:122] "RemoveContainer" containerID="aa9c1636b0df150fe2c2fd3d90c15ae1ca82ad3ac3f749786b1f9d643068b95d" Nov 03 03:45:16.631472 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:16.631423 5605 scope.go:122] "RemoveContainer" containerID="34e7579713cec41aaca36de061e4c6a161463e7dbccd21147feddf208ec8edda" Nov 03 03:45:16.631697 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:45:16.631673 5605 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894625-qnwt9_odh-ai-gateway-infra(e4fceab0-d8a1-4df8-be90-bcc21dda6afd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894625-qnwt9" podUID="e4fceab0-d8a1-4df8-be90-bcc21dda6afd" Nov 03 03:45:16.738846 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:16.738804 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894625-qnwt9"] Nov 03 03:45:17.389389 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:17.389342 5605 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-b984f6b6b-g6f2h_e32389d5-705f-4202-98ea-6fc6a5ad5d54/metrics-server/0.log" Nov 03 03:45:17.819330 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:17.819288 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894625-qnwt9" Nov 03 03:45:17.822770 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:17.822743 5605 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-rbsfv_fc905c8d-4bfa-49ee-b0b7-8c434727e284/manager/0.log" Nov 03 03:45:17.962800 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:17.962705 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4fceab0-d8a1-4df8-be90-bcc21dda6afd-kube-api-access-gp8q4\" (UniqueName: \"kubernetes.io/projected/e4fceab0-d8a1-4df8-be90-bcc21dda6afd-kube-api-access-gp8q4\") pod \"e4fceab0-d8a1-4df8-be90-bcc21dda6afd\" (UID: \"e4fceab0-d8a1-4df8-be90-bcc21dda6afd\") " Nov 03 03:45:17.963104 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:17.963081 5605 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4fceab0-d8a1-4df8-be90-bcc21dda6afd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4fceab0-d8a1-4df8-be90-bcc21dda6afd-maas-api-service-ca\") pod \"e4fceab0-d8a1-4df8-be90-bcc21dda6afd\" (UID: \"e4fceab0-d8a1-4df8-be90-bcc21dda6afd\") " Nov 03 03:45:17.964558 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:17.964523 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4fceab0-d8a1-4df8-be90-bcc21dda6afd-maas-api-service-ca" pod "e4fceab0-d8a1-4df8-be90-bcc21dda6afd" (UID: "e4fceab0-d8a1-4df8-be90-bcc21dda6afd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:45:17.966992 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:17.966967 5605 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4fceab0-d8a1-4df8-be90-bcc21dda6afd-kube-api-access-gp8q4" pod "e4fceab0-d8a1-4df8-be90-bcc21dda6afd" (UID: "e4fceab0-d8a1-4df8-be90-bcc21dda6afd"). InnerVolumeSpecName "kube-api-access-gp8q4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:45:18.064352 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:18.064312 5605 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4fceab0-d8a1-4df8-be90-bcc21dda6afd-maas-api-service-ca\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:45:18.064352 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:18.064352 5605 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gp8q4\" (UniqueName: \"kubernetes.io/projected/e4fceab0-d8a1-4df8-be90-bcc21dda6afd-kube-api-access-gp8q4\") on node \"ip-10-0-140-140.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:45:18.081857 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:18.081820 5605 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-594d9845fc-7m8kb_949a8686-ae14-4c12-9da2-6bb05250692b/manager/0.log" Nov 03 03:45:18.219480 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:18.219397 5605 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2q7kh_4480cca6-b2e9-4840-ac8c-a88a40ed90f2/node-exporter/0.log" Nov 03 03:45:18.623238 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:18.623201 5605 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2q7kh_4480cca6-b2e9-4840-ac8c-a88a40ed90f2/kube-rbac-proxy/0.log" Nov 03 03:45:18.646802 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:18.646777 5605 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894625-qnwt9" Nov 03 03:45:18.646978 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:18.646780 5605 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894625-qnwt9" event={"ID":"e4fceab0-d8a1-4df8-be90-bcc21dda6afd","Type":"ContainerDied","Data":"65985b083ac38833396698bfb0b4536e875dfcd916fa54ed754d65e6b200efd0"} Nov 03 03:45:18.646978 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:18.646928 5605 scope.go:122] "RemoveContainer" containerID="34e7579713cec41aaca36de061e4c6a161463e7dbccd21147feddf208ec8edda" Nov 03 03:45:18.800630 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:18.800598 5605 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894625-qnwt9"] Nov 03 03:45:18.852581 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:18.852448 5605 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894625-qnwt9"] Nov 03 03:45:19.029321 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:19.029239 5605 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2q7kh_4480cca6-b2e9-4840-ac8c-a88a40ed90f2/init-textfile/0.log" Nov 03 03:45:20.848526 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:20.848488 5605 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4fceab0-d8a1-4df8-be90-bcc21dda6afd" path="/var/lib/kubelet/pods/e4fceab0-d8a1-4df8-be90-bcc21dda6afd/volumes" Nov 03 03:45:23.925746 ip-10-0-140-140 kubenswrapper[5605]: E1103 03:45:23.925705 5605 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmrb8" containerName="extractor" Nov 03 03:45:25.968674 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:25.968636 5605 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-jmj4g_5d514c70-f1a7-462a-bd3e-6f5bdc016c39/prometheus-operator/0.log" Nov 03 03:45:26.198134 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:26.198103 5605 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6dc8f585b4-sb2dx_8e0e5200-c6f3-4e09-aa80-dfd996cf38d1/kube-auth-proxy/0.log" Nov 03 03:45:26.404318 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:26.404267 5605 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-jmj4g_5d514c70-f1a7-462a-bd3e-6f5bdc016c39/kube-rbac-proxy/0.log" Nov 03 03:45:26.619405 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:26.619373 5605 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-bpbq7_563c5b47-58ea-4fb8-b9f1-332cf8dae641/istio-proxy/0.log" Nov 03 03:45:27.038195 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:27.038162 5605 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-dbd7q_dc490604-2e00-49dd-8423-47fc70709678/istio-proxy/0.log" Nov 03 03:45:27.221600 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:27.221571 5605 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-699d68669f-9jg2q_923b90de-5e45-4add-b092-91af16b7dbad/telemeter-client/0.log" Nov 03 03:45:27.457398 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:27.457368 5605 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-k449x_0378fcbc-4027-4275-b274-2c188f3ff442/istio-proxy/0.log" Nov 03 03:45:27.626856 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:27.626829 5605 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-699d68669f-9jg2q_923b90de-5e45-4add-b092-91af16b7dbad/reload/0.log" Nov 03 03:45:27.664757 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:27.664728 5605 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-p8fqq_0bedb6e8-5829-4cc2-a912-35131c941e09/istio-proxy/0.log" Nov 03 03:45:28.030847 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:28.030813 5605 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-699d68669f-9jg2q_923b90de-5e45-4add-b092-91af16b7dbad/kube-rbac-proxy/0.log" Nov 03 03:45:28.704443 ip-10-0-140-140 kubenswrapper[5605]: I1103 03:45:28.704405 5605 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-659c975df4-4wn4f_3344d6d8-48b8-4aa5-afba-b7e9b504b5ee/payload-pre-processing/0.log"